You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
zstd fails to process some filenames on Windows [we need a hero] #3882
$ dir /bbirds.txtpájaros.txtvögel.txtптицы.txt
$ zstd *.*zstd: can't stat pajaros.txt : No such file or directory -- ignoredzstd: can't stat vogel.txt : No such file or directory -- ignored 2 files compressed : 85.71% ( 3.00 KiB => 2.57 KiB)
$ dir /b *.zstbirds.txt.zstптицы.txt.zst
@Cyan4973said
zstd doesn't do anything special, it uses the C standard fopen() function.
This should be naturally compatible with any utf-8 character set.
But I suspect Windows does something different with the character set,
so there might be a need to employ other non-portable Windows variants of > fopen for these scenarios.
In my example, the file vögel.txt.zst was created correctly.
But the text output to show the progress is not correct.
Therefore, I figure that the fopen() call is ok.
Zstd 1.5.6, the problem is still present, alas. As a result, there is a constant fear that Zstd will miss several files in the folder, so you don't want to use it on an ongoing basis.
However, some other modern CLI compression apps work fine.
$ 7z.exe a hello vögel.txt7-Zip 24.09 (x64) : Copyright (c) 1999-2024 Igor Pavlov : 2024-11-29Scanning the drive:1 file, 5 bytes (1 KiB)Creating archive: hello.7zAdd new data to archive: 1 file, 5 bytes (1 KiB)Files read from disk: 1Archive size: 131 bytes (1 KiB)Everything is Ok
$ zpaqfranz.exe a hello vögel.txtzpaqfranz v60.10w-JIT,-GUI,-L,HW BLAKE3,SHA1/2,4,SFX64 v55.1,(2024-12-15)franz:-noconsoleCreating hello.zpaq at offset 0 + 0Add 2025-02-10 12:03:51 1 5 ( 5.00 B) 2T (0 dirs)1 +added, 0 -removed.0 + (5 -> 5 -> 829) = 829 @ 64.00 B/sFiles added +10.093s (00:00:00,4.03MB) (all OK)
Dear @fcorbelli (the author of ZpaqFranz), could you be so kind to enlighten the audience how did you manage to recognize Unicode filenames under legacy Windows OS?
Certainly, I can give my suggestions. The issue is much, much, MUCH more complicated than it may seem. Nothing is easy with Microsoft.
There are three types of problems:
Internal handling of non-Latin strings within the program.
Output of non-Latin strings to the Windows console (which is no trivial matter).
The incredible complexity and fragility of NTFS.
The incredible complexity and fragility of Microsoft
As a key point, I would like to clarify that zpaqfranz is a derivative of zpaq, and therefore, all primary credit and recognition should go to Matt Mahoney. I believe it is important to acknowledge this.
I really do not know zstd, I can give some advice for a C++ program
As is known, or perhaps not, there are at least half a dozen Microsoft APIs that do the same thing. So I’ll cut to the chase to show ONE way, not necessarily the only one, nor the best.
Let’s assume we have a program that is used to handling C++ strings in the "normal" way, that is, std::string, while we want it to become "smart" and also understand wide strings. If we want to maintain some minimal difference between the sources (compiled on Linux and Windows), then compromises must be made.
Translation: Strings on Linux are perfectly capable of handling Unicode characters (in modern versions of Linux, of course), without requiring any special handling. Windows programs that use Windows-specific Unicode data structures can manage files. The problem arises (speaking from my experience) when you want a source code that can be compiled on both Windows and Linux with some #ifdef directives.
I hope it is clear what the goal is: to have "small" changes to the source code, even at the cost of a bit of spaghetti code (after all, I’m Italian; it’s not a big problem for me).
| This should be naturally compatible with any utf-8 character set.
😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄
Windows rant mode OFF
Windows rant mode ON
You will early learn that just about everything is boh on NTFS
There are particular types of files with convoluted management of all kinds, starting from sparse volumes to video games (!, yes... the video games from the Microsoft app store) to the fake OneDrive files (i.e., the placeholders. Files that exist, but also don’t).
Windows rant mode OFF
Just an example
Is a file real, or not? This is the question...
This is usually a OneDrive placeholder, but no always...
#ifdef _WIN32
#ifndef FILE_ATTRIBUTE_RECALL_ON_DATA_ACCESS
#define FILE_ATTRIBUTE_RECALL_ON_DATA_ACCESS 0x00400000
#endif
bool Jidac::isrealfile(const string& i_filename)
{
// Usa GetFileAttributesEx per ottenere informazioni sul file
WIN32_FILE_ATTRIBUTE_DATA fileAttributes;
if (GetFileAttributesEx((utow(i_filename.c_str()).c_str()),GetFileExInfoStandard, &fileAttributes))
{
// Controlla se il file è un "placeholder" (FILE_ATTRIBUTE_RECALL_ON_DATA_ACCESS o FILE_ATTRIBUTE_PINNED sono indicatori comuni)
if (fileAttributes.dwFileAttributes & FILE_ATTRIBUTE_RECALL_ON_DATA_ACCESS)
return false; // File non scaricato completamente
else
return true; // File scaricato completamente
}
else
{
if (flagverbose)
myprintf("10836$ Cannot get attribute for <<%Z>>\n",i_filename.c_str());
return false; // Assume che non sia disponibile in caso di errore
}
}
#endif
Please, consider copying that code or using some of the suggestions of @fcorbelli. I happily use xxhash every day, along with b3sum, gpg, croc, rg, 7-zip, but not zstd as it skips (does not process) Unicode filenames under Windows 7.
@easyaspi314 is the hero who invented universal Unicode fix for xxhash.
I hope that she is doing well, survived COVID-19 and other shocks of recent years.
Obviously, it's easier for her to tell us how to properly import that solution into zstd.
I'm ready to test a compiled zstd binary with a fix. 🙏 🙏 🙏
Activity
a1880 commentedon Jan 30, 2024
This is what I get with v1.5.5 64-bit under Windows 10:
flagarde commentedon Jan 31, 2024
It seems Windows doesn't deal with UTF-8 by default https://learn.microsoft.com/en-us/cpp/c-runtime-library/reference/fopen-wfopen?view=msvc-170. Seems the safest solution is to use _wfopen and convert narrow string to wide with MultiByteToWideChar function.
a1880 commentedon Jan 31, 2024
In my example, the file
vögel.txt.zstwas created correctly.But the text output to show the progress is not correct.
Therefore, I figure that the
fopen()call is ok.flawrist commentedon Feb 10, 2025
Zstd 1.5.6, the problem is still present, alas. As a result, there is a constant fear that Zstd will miss several files in the folder, so you don't want to use it on an ongoing basis.
However, some other modern CLI compression apps work fine.
Let's do something about that, please.
flawrist commentedon Feb 22, 2025
Dear @fcorbelli (the author of ZpaqFranz), could you be so kind to enlighten the audience how did you manage to recognize Unicode filenames under legacy Windows OS?
fcorbelli commentedon Feb 22, 2025
Certainly, I can give my suggestions. The issue is much, much, MUCH more complicated than it may seem. Nothing is easy with Microsoft.
There are three types of problems:
As a key point, I would like to clarify that zpaqfranz is a derivative of zpaq, and therefore, all primary credit and recognition should go to Matt Mahoney. I believe it is important to acknowledge this.
fcorbelli commentedon Feb 22, 2025
I really do not know zstd, I can give some advice for a C++ program
As is known, or perhaps not, there are at least half a dozen Microsoft APIs that do the same thing. So I’ll cut to the chase to show ONE way, not necessarily the only one, nor the best.
Let’s assume we have a program that is used to handling C++ strings in the "normal" way, that is, std::string, while we want it to become "smart" and also understand wide strings. If we want to maintain some minimal difference between the sources (compiled on Linux and Windows), then compromises must be made.
Translation: Strings on Linux are perfectly capable of handling Unicode characters (in modern versions of Linux, of course), without requiring any special handling. Windows programs that use Windows-specific Unicode data structures can manage files. The problem arises (speaking from my experience) when you want a source code that can be compiled on both Windows and Linux with some
#ifdefdirectives.I hope it is clear what the goal is: to have "small" changes to the source code, even at the cost of a bit of spaghetti code (after all, I’m Italian; it’s not a big problem for me).
fcorbelli commentedon Feb 22, 2025
Then you should make some different functions. The first is something like that
This will "brutally" turn a "standard" C++ string into a wstring (a wide-Unicode-aware string)
fcorbelli commentedon Feb 22, 2025
BTW there are a gazillion different way to do this stuff, whatever
Then you need (2) a print-to-console (just a snippet, to be adapted to whatever)
Then, on Windows, you need to take the console then put back (!)
...
When exiting from the program
fcorbelli commentedon Feb 22, 2025
BTW I rewrited printf() to become myprintf(), with a new %Z to write out non-latin strings on Windows, aka calling something like printUTF8()
And now the magic
You should write a different fopen function, say myfopen that to something like that
This will call the overloaded CreateFileW (because the first parameter is a wstring, not a string)
That's all (of course you must handle myfopen for READ instead of WRITE or whatever)
fcorbelli commentedon Feb 22, 2025
The myfclose() something like that (this is just the idea)
BTW on *nix
on Windows
fcorbelli commentedon Feb 22, 2025
Shall I continue, or am I perhaps writing trivial things that are already well known?
fcorbelli commentedon Feb 22, 2025
Windows rant mode ON
| This should be naturally compatible with any utf-8 character set.
😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄
Windows rant mode OFF
Windows rant mode ON
You will early learn that just about everything is boh on NTFS
There are particular types of files with convoluted management of all kinds, starting from sparse volumes to video games (!, yes... the video games from the Microsoft app store) to the fake OneDrive files (i.e., the placeholders. Files that exist, but also don’t).
Windows rant mode OFF
fcorbelli commentedon Feb 22, 2025
Just an example
Is a file real, or not? This is the question...
This is usually a OneDrive placeholder, but no always...
😄
flawrist commentedon Feb 24, 2025
Dear @Cyan4973, we encountered a similar Unicode issue when developing xxhash, and found a solution in the following thread:
Please, consider copying that code or using some of the suggestions of @fcorbelli. I happily use xxhash every day, along with b3sum, gpg, croc, rg, 7-zip, but not zstd as it skips (does not process) Unicode filenames under Windows 7.
fcorbelli commentedon Feb 24, 2025
BTW Windows 7 and Windows XP are different, either
Windows 7 have a very strange console output support.
For XP you must dynamically load the functions from kernel32.dll
And, finally, you'll need some "smarter" function to enumerate files, like
"add myfile foldersomething"
And it is just the VERY beginning 😅
And something like that
flawrist commentedon Feb 24, 2025
@easyaspi314 is the hero who invented universal Unicode fix for xxhash.
I hope that she is doing well, survived COVID-19 and other shocks of recent years.
Obviously, it's easier for her to tell us how to properly import that solution into zstd.
I'm ready to test a compiled zstd binary with a fix. 🙏 🙏 🙏
easyaspi314 commentedon Feb 26, 2025
I might be able to whip up a solution, yes
flawrist commentedon Mar 19, 2025
Dear @easyaspi314, how are you doing? Unicode dragon is hard to defeat?
Fix zstd delta creation for non-ASCII paths on Windows