Repository navigation
Conversation
Zip entry names that do not carry the UTF-8 flag were always decoded as CP437 (the ZIP spec default), which produces mojibake for archives created by tools that store names in UTF-8 without the flag, or in a legacy code page such as GBK (CP936), Big5 or Shift_JIS (issue ouch-org#691). - Add `--encoding <ENCODING>` to `decompress` and `list`. It accepts WHATWG encoding labels (gbk, big5, shift_jis, windows-1251, ...) and the familiar code page names that WHATWG does not label (cp936, windows-1252, ...), resolved with the new encoding_rs dependency. - Entry names flagged as UTF-8 are unaffected, per the ZIP spec. - Without `--encoding`, unflagged names that are valid UTF-8 are now decoded as UTF-8 instead of CP437, recovering archives written by tools that store UTF-8 names but forget to set the flag. - Otherwise the spec-mandated CP437 decode is kept as the fallback. Path safety checks (enclosed_name/mangled_name) now run on the decoded name; they are reimplemented on top of typed_path, the same crate the zip crate uses, so the semantics are unchanged. Decoding before sanitizing also fixes multi-byte names whose trail bytes happen to be a backslash byte, which used to be treated as path separators.
Collaborator
|
Author
|
Addressed in cebf2e8. |
Collaborator
|
Files with a name whose last byte is 0x5C is now extracted as an empty directory and the file content is lost |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #691.
Problem
When a zip's entry names aren't UTF-8 and the archive's UTF-8 flag (bit 11) is unset, the
zipcrate falls back to CP437 — so archives written on systems using GBK/Big5/Shift_JIS/windows-125x, or UTF-8 names written without the flag, come out as mojibake that can't even be fixed afterwards withconvmv/iconv(the decoded text is already wrong).unzip/7zzrecover such archives fine.Changes
--encoding <ENCODING>flag onouch decompressandouch list, followingunzip -Oprecedent. Accepts WHATWG encoding labels (gbk,big5,shift_jis,windows-1251, ...) plus common code-page aliases (cp936,windows-1252,cp65001). Unknown labels get a cleanUnknown encodingerror. Zip-only; documented in--help.enclosed_name/mangled_namereimplemented viatyped-path(the same cratezipuses internally) so sanitization runs on the decoded name — this also closes a subtle bug where a multibyte trail byte0x5C(e.g. in GBK说) could be misread as a path separator.file.is_dir()instead of checking for a trailing/.Testing
zip -rarchive with raw GBK (CP936) filename bytes → before:test-name-╒Γ╩╟╥╗╕÷▓Γ╩╘╬─╝■.txt; after--encoding gbk:test-name-这是一个测试文件.txt✓Schwarz-weiß.txt) now recovers with no flag (issue's German-umlaut case) ✓--encoding not-a-charset→ clean error ✓cargo test --profile fast: 59 unit / 57 integration / 13 ui / 3 mime);cargo clippy --all-targetsclean incl.--no-default-features; nightlycargo fmtapplied.Discussion point
Auto-detection (e.g.
chardetng) was considered instead of a flag but rejected: it misdecodes genuine CP437 names (verified: CP437Grüße→ Shift_JISGr≪e) and would silently change existing behavior. Happy to add detection behind a flag value (--encoding auto) if you'd like.New deps:
encoding_rs(+6 small transitive crates),typed-path(already in tree viazip),crc32fast(dev-dep for test fixtures).