Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,18 @@
# Changelog

## 0.13.2 - 2026-11-20

### Changed

- Say on stderr when read or extract uses the default locale because the file has no copy in the one asked for

### Fixed

- Extract each name once when extracting every file, under the locale it is stored in, and name the left-out locales on stderr
- Extract files stored only under a non-default locale when extracting every file, instead of failing
- List every locale of a name stored under more than 32, instead of reading past a buffer
- Show an empty time in list for a file time that cannot be converted, instead of reading past a buffer

## 0.13.1 - 2026-11-14

### Changed
Expand Down
2 changes: 1 addition & 1 deletion CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
cmake_minimum_required(VERSION 3.25)

project(mpqcli VERSION 0.13.1)
project(mpqcli VERSION 0.13.2)

# An option() name is global to the CMake cache, including a consumer's, so these
# carry the project prefix; a bare BUILD_STATIC claims a name anyone might want.
Expand Down
13 changes: 13 additions & 0 deletions docs/changelog.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,18 @@
# Changelog

## 0.13.2 - 2026-11-20

### Changed

- Say on stderr when read or extract uses the default locale because the file has no copy in the one asked for

### Fixed

- Extract each name once when extracting every file, under the locale it is stored in, and name the left-out locales on stderr
- Extract files stored only under a non-default locale when extracting every file, instead of failing
- List every locale of a name stored under more than 32, instead of reading past a buffer
- Show an empty time in list for a file time that cannot be converted, instead of reading past a buffer

## 0.13.1 - 2026-11-14

### Changed
Expand Down
13 changes: 12 additions & 1 deletion docs/commands/extract.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,17 @@ $ mpqcli extract wow-patch.mpq
[*] Extracted: realmlist.wtf
```

## Extract all files from an archive with several locales

An archive can store the same file name more than once, under different locales. Extracting every file writes each name once: the variant under the `--locale` given, else the default (neutral) one, else the one with the lowest locale ID. The variants left out are named on stderr.

```bash
$ mpqcli extract --locale deDE patch.mpq
[*] Extracted: (listfile)
[*] Extracted: cats.txt
[*] Left out other locales of cats.txt: enUS, esES
```

## Extract all files to an output directory

Extract files to a specific output directory, which will be created if it doesn't already exist. In this example, `patch-1.10` is the user-specified output directory.
Expand Down Expand Up @@ -65,7 +76,7 @@ $ mpqcli list wow-patch.mpq | grep -i '\.exe$' | mpqcli extract wow-patch.mpq -

## Extract one specific file with locale

Use the `-e` or `--locale` argument to specify the locale of the file to extract. If there is no file with the requested name and locale, the default locale will be used instead.
Use the `-e` or `--locale` argument to specify the locale of the file to extract. If there is no file with the requested name and locale, the default locale is used instead, and a line on stderr says so.

```bash
$ mpqcli extract Patch_rt.mpq "rez\gluBNRes.res" --locale deDE
Expand Down
2 changes: 1 addition & 1 deletion docs/commands/read.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ $ mpqcli read Patch_rt.mpq "rez/stat_txt.tbl"

## Read one specific file with locale

Use the `-e` or `--locale` argument to specify the locale of the file to read. If there is no file with the requested name and locale, the default locale will be used instead.
Use the `-e` or `--locale` argument to specify the locale of the file to read. If there is no file with the requested name and locale, the default locale is used instead, and a line on stderr says so.

```bash
$ mpqcli read Patch_rt.mpq "rez\stat_txt.tbl" --locale ptPT
Expand Down
27 changes: 19 additions & 8 deletions src/commands/extract.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -54,17 +54,28 @@ bool Extract(const ExtractOptions &options, std::ostream &err, const std::atomic
LCID lcid = options.locale.has_value() ? LangToLocale(options.locale.value()) : default_locale;

// Every name is settled before the output directory is created, so a
// rejected --flat run leaves nothing on disk
const std::vector<std::string> file_names =
options.files.has_value()
? options.files.value()
: ListArchivedFiles(archive.Handle(), UsableListfile(options.listfile, err), cancelled);
if (file_names.empty()) {
// rejected --flat run leaves nothing on disk. A named file is asked for under
// the given locale; extracting everything writes one variant of each name.
std::vector<FileToExtract> files;
if (options.files.has_value()) {
for (const auto &name : options.files.value()) {
files.push_back({name, lcid, {}});
}
} else {
files = ChooseVariants(
ListArchivedFiles(archive.Handle(), UsableListfile(options.listfile, err), cancelled),
lcid);
}
if (files.empty()) {
err << "[*] No files to extract" << std::endl;
archive.Close();
return true;
}
if (options.flat) {
std::vector<std::string> file_names;
for (const auto &file : files) {
file_names.push_back(file.name);
}
const auto collision = FindFlatCollision(file_names);
if (collision.has_value()) {
err << "[!] --flat would extract " << collision->first << " and " << collision->second
Expand Down Expand Up @@ -98,8 +109,8 @@ bool Extract(const ExtractOptions &options, std::ostream &err, const std::atomic
<< effective_output.string() << std::endl;
}

const int result = ExtractFiles(archive.Handle(), effective_output, file_names, options.flat,
lcid, err, cancelled);
const int result =
ExtractFiles(archive.Handle(), effective_output, files, options.flat, err, cancelled);
archive.Close();

if (result != 0) {
Expand Down
78 changes: 65 additions & 13 deletions src/mpq/extract.cpp
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
#include "mpq/extract.h"

#include <algorithm>
#include <atomic>
#include <cstddef>
#include <filesystem>
#include <map>
#include <optional>
Expand Down Expand Up @@ -36,9 +38,9 @@ std::string ExtractedName(const std::string &file_name, bool flat) {

} // namespace

std::vector<std::string> ListArchivedFiles(HANDLE archive,
const std::optional<fs::path> &listfile_name,
const std::atomic<bool> &cancelled) {
std::vector<ArchivedFile> ListArchivedFiles(HANDLE archive,
const std::optional<fs::path> &listfile_name,
const std::atomic<bool> &cancelled) {
const std::string listfile_string =
listfile_name.has_value() ? listfile_name->string() : std::string();
const char *listfile = listfile_name.has_value() ? listfile_string.c_str() : nullptr;
Expand All @@ -54,19 +56,58 @@ std::vector<std::string> ListArchivedFiles(HANDLE archive,
throw StormError("Failed to find first file in MPQ archive", error);
}

std::vector<std::string> file_names;
std::vector<ArchivedFile> files;
do {
// Closed by hand rather than through ThrowIfCancelled, so the find handle
// does not leak on the way out
if (cancelled.load(std::memory_order_relaxed)) {
SFileFindClose(find_handle);
throw Interrupted{};
}
file_names.emplace_back(find_data.cFileName);
files.push_back({find_data.cFileName, find_data.lcLocale});
} while (SFileFindNextFile(find_handle, &find_data));

SFileFindClose(find_handle);
return file_names;
return files;
}

std::vector<FileToExtract> ChooseVariants(const std::vector<ArchivedFile> &files,
LCID preferred_locale) {
std::vector<std::string> names;
std::map<std::string, std::vector<LCID>> locales_by_name;
for (const auto &file : files) {
auto &locales = locales_by_name[file.name];
if (locales.empty()) {
names.push_back(file.name);
}
locales.push_back(file.locale);
}

// Compared without the platform StormLib packs into bits 16 to 23, as list
// prints them; the chosen locale keeps it, so the lookup finds that exact entry
const auto by_locale = [](LCID a, LCID b) { return SFILE_LOCALE(a) < SFILE_LOCALE(b); };
const auto stored_under = [](const std::vector<LCID> &locales, LCID wanted) {
return std::find_if(locales.begin(), locales.end(), [wanted](LCID locale) {
return SFILE_LOCALE(locale) == SFILE_LOCALE(wanted);
});
};

std::vector<FileToExtract> chosen;
for (const auto &name : names) {
auto locales = locales_by_name[name];
std::stable_sort(locales.begin(), locales.end(), by_locale);
auto pick = stored_under(locales, preferred_locale);
if (pick == locales.end()) {
pick = stored_under(locales, default_locale);
}
if (pick == locales.end()) {
pick = locales.begin();
}
const LCID locale = *pick;
locales.erase(pick);
chosen.push_back({name, locale, locales});
}
return chosen;
}

std::optional<std::pair<std::string, std::string>>
Expand All @@ -86,13 +127,19 @@ FindFlatCollision(const std::vector<std::string> &file_names) {
return std::nullopt;
}

int ExtractFiles(HANDLE archive, const fs::path &output, const std::vector<std::string> &file_names,
bool flat, LCID preferred_locale, std::ostream &err,
const std::atomic<bool> &cancelled) {
int ExtractFiles(HANDLE archive, const fs::path &output, const std::vector<FileToExtract> &files,
bool flat, std::ostream &err, const std::atomic<bool> &cancelled) {
int result = 0;
for (const auto &name : file_names) {
for (const auto &file : files) {
ThrowIfCancelled(cancelled);
result |= ExtractFile(archive, output, name, flat, preferred_locale, err);
result |= ExtractFile(archive, output, file.name, flat, file.locale, err);
if (!file.left_out.empty()) {
err << "[*] Left out other locales of " << file.name << ": ";
for (std::size_t i = 0; i < file.left_out.size(); i++) {
err << (i == 0 ? "" : ", ") << LocaleToLang(SFILE_LOCALE(file.left_out[i]));
}
err << std::endl;
}
}
return result;
}
Expand All @@ -101,13 +148,18 @@ int ExtractFile(HANDLE archive, const fs::path &output, const std::string &file_
LCID preferred_locale, std::ostream &err) {
SFileSetLocale(preferred_locale);
const std::string stored_name = StoredFileName(archive, file_name, preferred_locale);
if (!FileExistsInArchiveForLocale(archive, stored_name, preferred_locale) &&
!FileExistsInArchiveForLocale(archive, stored_name, default_locale)) {
const bool has_preferred = FileExistsInArchiveForLocale(archive, stored_name, preferred_locale);
if (!has_preferred && !FileExistsInArchiveForLocale(archive, stored_name, default_locale)) {
err << "[!] Failed: File doesn't exist"
<< PrettyPrintLocale(preferred_locale, " for locale ", true) << ": " << stored_name
<< std::endl;
return 1;
}
// Only a named file can fall back: extracting everything asks for each
// variant under the locale it is stored in
if (!has_preferred) {
ReportLocaleFallback(stored_name, preferred_locale, err);
}

const std::string file_name_string = ExtractedName(stored_name, flat);

Expand Down
33 changes: 29 additions & 4 deletions src/mpq/extract.h
Original file line number Diff line number Diff line change
Expand Up @@ -13,23 +13,48 @@

namespace mpqcli {

/// One stored variant of a file: its name, and the locale it is stored under
///
/// The locale is as StormLib reports it, with the platform in bits 16 to 23.
struct ArchivedFile {
std::string name;
LCID locale = 0;
};

/// A file to extract, under the locale to ask for it in
struct FileToExtract {
std::string name;
LCID locale = 0;
/// The other locales the name is stored under, which are not extracted
std::vector<LCID> left_out;
};

/// Lists every file the archive can name, once per locale it is stored under
///
/// An archive with no entries gives an empty list.
///
/// @throws StormError if StormLib cannot enumerate the archive's files
std::vector<std::string>
std::vector<ArchivedFile>
ListArchivedFiles(HANDLE archive, const std::optional<std::filesystem::path> &listfile_name,
const std::atomic<bool> &cancelled);

/// Picks one variant of each name, in the order the names are first listed
///
/// The variant under preferred_locale, else the neutral one, else the one with the
/// lowest locale. The others are named in left_out, lowest locale first.
std::vector<FileToExtract> ChooseVariants(const std::vector<ArchivedFile> &files,
LCID preferred_locale);

/// Returns two different files that a flat extraction would write to the same path
std::optional<std::pair<std::string, std::string>>
FindFlatCollision(const std::vector<std::string> &file_names);

/// Extracts each named file into output, returning non-zero if any failed
/// Extracts each file into output under its locale, returning non-zero if any failed
///
/// A file with other variants left out is followed by a line naming them.
int ExtractFiles(HANDLE archive, const std::filesystem::path &output,
const std::vector<std::string> &file_names, bool flat, LCID preferred_locale,
std::ostream &err, const std::atomic<bool> &cancelled);
const std::vector<FileToExtract> &files, bool flat, std::ostream &err,
const std::atomic<bool> &cancelled);
int ExtractFile(HANDLE archive, const std::filesystem::path &output, const std::string &file_name,
bool flat, LCID preferred_locale, std::ostream &err);

Expand Down
9 changes: 9 additions & 0 deletions src/mpq/query.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
#include <StormLib.h>

#include "util/hash.h"
#include "util/locales.h"
#include "util/path.h"

namespace fs = std::filesystem;
Expand All @@ -33,6 +34,14 @@ bool FileExistsInArchiveForLocale(const HANDLE archive, const std::string &file_
return file_exists;
}

void ReportLocaleFallback(const std::string &file_path, LCID locale, std::ostream &err) {
if (locale == LangToLocale("enUS")) {
return;
}
err << "[*] " << file_path << " has no " << LocaleToLang(locale) << " variant, using "
<< LocaleToLang(default_locale) << std::endl;
}

std::string StoredFileName(HANDLE archive, const std::string &file_path, LCID locale) {
SFileSetLocale(locale);
if (SFileHasFile(archive, file_path.c_str())) {
Expand Down
6 changes: 6 additions & 0 deletions src/mpq/query.h
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,12 @@ inline const std::vector<std::string> special_mpq_files = {"(listfile)", "(signa

bool FileExistsInArchiveForLocale(HANDLE archive, const std::string &file_path, LCID locale);

/// Reports on err that file_path has no variant under locale, so the default one is used
///
/// Says nothing when locale is enUS: the default locale prints as enUS too, so the line
/// would say enUS fell back to enUS.
void ReportLocaleFallback(const std::string &file_path, LCID locale, std::ostream &err);

/// Returns the name the archive stores file_path under, looked up under locale
///
/// An MPQ normally stores paths with backslashes, and StormLib hashes / and \ as different
Expand Down
7 changes: 5 additions & 2 deletions src/mpq/read.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -16,13 +16,16 @@ std::unique_ptr<char[]> ReadArchivedFile(HANDLE archive, const char *file_name,
std::ostream &err) {
SFileSetLocale(preferred_locale);
const std::string stored_name = StoredFileName(archive, file_name, preferred_locale);
if (!FileExistsInArchiveForLocale(archive, stored_name, preferred_locale) &&
!FileExistsInArchiveForLocale(archive, stored_name, default_locale)) {
const bool has_preferred = FileExistsInArchiveForLocale(archive, stored_name, preferred_locale);
if (!has_preferred && !FileExistsInArchiveForLocale(archive, stored_name, default_locale)) {
err << "[!] Failed: File doesn't exist"
<< PrettyPrintLocale(preferred_locale, " for locale ", true) << ": " << stored_name
<< std::endl;
return nullptr;
}
if (!has_preferred) {
ReportLocaleFallback(stored_name, preferred_locale, err);
}

HANDLE file;
if (!SFileOpenFileEx(archive, stored_name.c_str(), SFILE_OPEN_FROM_MPQ, &file)) {
Expand Down
Loading