Skip to content

zstd fails to process some filenames on Windows [we need a hero] #3882

Open

Description

@flawrist

Windows 7 x64, zstd 1.5.5

$ dir /b
birds.txt
pájaros.txt
vögel.txt
птицы.txt

$ zstd *.*
zstd: can't stat pajaros.txt : No such file or directory -- ignored
zstd: can't stat vogel.txt : No such file or directory -- ignored
  2 files compressed : 85.71% (  3.00 KiB =>   2.57 KiB)
 
$ dir /b *.zst
birds.txt.zst
птицы.txt.zst

@Cyan4973 said
zstd doesn't do anything special, it uses the C standard fopen() function.
This should be naturally compatible with any utf-8 character set.
But I suspect Windows does something different with the character set,
so there might be a need to employ other non-portable Windows variants of > fopen for these scenarios.

Activity

  1. a1880 commented on Jan 30, 2024

    @a1880

    This is what I get with v1.5.5 64-bit under Windows 10:

    c:\xxx\zzz>dir /b
    vögel.txt
    
    c:\xxx\zzz>zstd *.*
    v÷gel.txt            :262.50%   (     8 B =>     21 B, v÷gel.txt.zst)
    
    c:\xxx\zzz>dir /b
    vögel.txt
    vögel.txt.zst
    
    
  2. flagarde commented on Jan 31, 2024

    @flagarde

    It seems Windows doesn't deal with UTF-8 by default https://learn.microsoft.com/en-us/cpp/c-runtime-library/reference/fopen-wfopen?view=msvc-170. Seems the safest solution is to use _wfopen and convert narrow string to wide with MultiByteToWideChar function.

  3. a1880 commented on Jan 31, 2024

    @a1880

    In my example, the file vögel.txt.zst was created correctly.
    But the text output to show the progress is not correct.
    Therefore, I figure that the fopen() call is ok.

  4. flawrist commented on Feb 10, 2025

    @flawrist
    Author

    Zstd 1.5.6, the problem is still present, alas. As a result, there is a constant fear that Zstd will miss several files in the folder, so you don't want to use it on an ongoing basis.

    However, some other modern CLI compression apps work fine.

    $ 7z.exe a hello vögel.txt
    7-Zip 24.09 (x64) : Copyright (c) 1999-2024 Igor Pavlov : 2024-11-29
    Scanning the drive:
    1 file, 5 bytes (1 KiB)
    Creating archive: hello.7z
    Add new data to archive: 1 file, 5 bytes (1 KiB)
    Files read from disk: 1
    Archive size: 131 bytes (1 KiB)
    Everything is Ok
    
    $ zpaqfranz.exe a hello vögel.txt
    zpaqfranz v60.10w-JIT,-GUI,-L,HW BLAKE3,SHA1/2,4,SFX64 v55.1,(2024-12-15)
    franz:-noconsole
    Creating hello.zpaq at offset 0 + 0
    Add 2025-02-10 12:03:51         1                  5 (   5.00  B) 2T (0 dirs)
    1 +added, 0 -removed.
    0 + (5 -> 5 ->  829) = 829  @ 64.00  B/s
    Files added +1
    0.093s (00:00:00,4.03MB) (all OK)

    Let's do something about that, please.

  5. flawrist commented on Feb 22, 2025

    @flawrist
    Author

    Dear @fcorbelli (the author of ZpaqFranz), could you be so kind to enlighten the audience how did you manage to recognize Unicode filenames under legacy Windows OS?

  6. fcorbelli commented on Feb 22, 2025

    @fcorbelli

    Certainly, I can give my suggestions. The issue is much, much, MUCH more complicated than it may seem. Nothing is easy with Microsoft.

    There are three types of problems:

    1. Internal handling of non-Latin strings within the program.
    2. Output of non-Latin strings to the Windows console (which is no trivial matter).
    3. The incredible complexity and fragility of NTFS.
    4. The incredible complexity and fragility of Microsoft

    As a key point, I would like to clarify that zpaqfranz is a derivative of zpaq, and therefore, all primary credit and recognition should go to Matt Mahoney. I believe it is important to acknowledge this.

  7. fcorbelli commented on Feb 22, 2025

    @fcorbelli
    1. Non-latin aware functions

    I really do not know zstd, I can give some advice for a C++ program

    As is known, or perhaps not, there are at least half a dozen Microsoft APIs that do the same thing. So I’ll cut to the chase to show ONE way, not necessarily the only one, nor the best.

    Let’s assume we have a program that is used to handling C++ strings in the "normal" way, that is, std::string, while we want it to become "smart" and also understand wide strings. If we want to maintain some minimal difference between the sources (compiled on Linux and Windows), then compromises must be made.

    Translation: Strings on Linux are perfectly capable of handling Unicode characters (in modern versions of Linux, of course), without requiring any special handling. Windows programs that use Windows-specific Unicode data structures can manage files. The problem arises (speaking from my experience) when you want a source code that can be compiled on both Windows and Linux with some #ifdef directives.

    I hope it is clear what the goal is: to have "small" changes to the source code, even at the cost of a bit of spaghetti code (after all, I’m Italian; it’s not a big problem for me).

  8. fcorbelli commented on Feb 22, 2025

    @fcorbelli

    Then you should make some different functions. The first is something like that

    #ifdef _WIN32
    // In Windows, convert UTF-8 string to wide string ignoring
    // invalid UTF-8 or >64K. Convert "/" to slash (default "\").
    std::wstring utow(const char* ss, char slash='\\') {
      assert(sizeof(wchar_t)==2);
      assert((wchar_t)(-1)==65535);
      std::wstring r;
      if (!ss) return r;
      const unsigned char* s=(const unsigned char*)ss;
      for (; s && *s; ++s) {
        if (s[0]=='/') r+=slash;
        else if (s[0]<128) r+=s[0];
        else if (s[0]>=192 && s[0]<224 && s[1]>=128 && s[1]<192)
          r+=(s[0]-192)*64+s[1]-128, ++s;
        else if (s[0]>=224 && s[0]<240 && s[1]>=128 && s[1]<192
                 && s[2]>=128 && s[2]<192)
          r+=(s[0]-224)*4096+(s[1]-128)*64+s[2]-128, s+=2;
      }
      return r;
    }
    #endif // corresponds to #ifdef (#ifdef _WIN32)
    

    This will "brutally" turn a "standard" C++ string into a wstring (a wide-Unicode-aware string)

  9. fcorbelli commented on Feb 22, 2025

    @fcorbelli

    BTW there are a gazillion different way to do this stuff, whatever

    Then you need (2) a print-to-console (just a snippet, to be adapted to whatever)

    void printUTF8(const char* s, FILE* f=stdout)
    {
    	assert(f);
    	assert(s);
    	if (g_output_handle!=0)
    		fprintf(g_output_handle,"%s",s);
    	if (flagsilent)
    		return;
    #ifdef unix
      fprintf(f, "%s", s);
    #else
      const HANDLE h=(HANDLE)_get_osfhandle(_fileno(f));
      DWORD ft=GetFileType(h);
      if (ft==FILE_TYPE_CHAR) {
        fflush(f);
        std::wstring w=utow(s, '/');
        DWORD n=0;
        WriteConsole(h, w.c_str(), w.size(), &n, 0);
      }
      else
        fprintf(f, "%s", s);
    #endif // corresponds to #ifdef (#ifdef unix)
    }
    

    Then, on Windows, you need to take the console then put back (!)

    int	 g_ConsoleCP;
    int  g_ConsoleOutputCP;
    
    

    ...

    #ifdef _WIN32
    //// set UTF-8 for console
    		g_ConsoleCP			=GetConsoleCP();
    		g_ConsoleOutputCP	=GetConsoleOutputCP();
    
    		SetConsoleCP		(65001);
    		SetConsoleOutputCP	(65001);
    ..,
    

    When exiting from the program

    #ifdef _WIN32
    	SetConsoleCP		(g_ConsoleCP);
    	SetConsoleOutputCP	(g_ConsoleOutputCP);
    
    
  10. fcorbelli commented on Feb 22, 2025

    @fcorbelli

    BTW I rewrited printf() to become myprintf(), with a new %Z to write out non-latin strings on Windows, aka calling something like printUTF8()

    And now the magic
    You should write a different fopen function, say myfopen that to something like that

    ...
    FP risultato=CreateFile(utow(filename).c_str(), access, share,NULL, disp, FILE_ATTRIBUTE_NORMAL , NULL);
    ...
    

    This will call the overloaded CreateFileW (because the first parameter is a wstring, not a string)

    That's all (of course you must handle myfopen for READ instead of WRITE or whatever)

  11. fcorbelli commented on Feb 22, 2025

    @fcorbelli

    The myfclose() something like that (this is just the idea)
    BTW on *nix

    typedef FILE* FP;
    const FP FPNULL=NULL;
    

    on Windows

    typedef HANDLE FP;
    const FP FPNULL=INVALID_HANDLE_VALUE;
    
    #ifdef _WIN32
    int myfclose(FP* fp)
    {
    	if ((*fp)==FPNULL)
    	{	
    		if (flagdebug)
    			myprintf("00092: win fclose on FPNULL ignored\n");
    		return EOF;
    	}
    	int risultato=CloseHandle(*fp) ? 0 : EOF;
    	*fp=FPNULL;
    	return risultato;
    }
    #else
    int myfclose(FP* fp)
    {
    	if ((*fp)==FPNULL)
    	{	
    		if (flagdebug)
    			myprintf("00083: fclose on unix FPNULL ignored\n");
    		return EOF;
    	}
    
    	int risultato=fclose(*fp);
    	*fp=FPNULL;
    	return risultato;
    }
    #endif
    
  12. fcorbelli commented on Feb 22, 2025

    @fcorbelli

    Shall I continue, or am I perhaps writing trivial things that are already well known?

  13. fcorbelli commented on Feb 22, 2025

    @fcorbelli

    Windows rant mode ON

    | This should be naturally compatible with any utf-8 character set.
    😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄
    Windows rant mode OFF

    Windows rant mode ON
    You will early learn that just about everything is boh on NTFS
    There are particular types of files with convoluted management of all kinds, starting from sparse volumes to video games (!, yes... the video games from the Microsoft app store) to the fake OneDrive files (i.e., the placeholders. Files that exist, but also don’t).
    Windows rant mode OFF

    while (1) Windows rant mode ON
    
  14. fcorbelli commented on Feb 22, 2025

    @fcorbelli

    Just an example
    Is a file real, or not? This is the question...

    This is usually a OneDrive placeholder, but no always...

    #ifdef _WIN32
    
    #ifndef FILE_ATTRIBUTE_RECALL_ON_DATA_ACCESS
    #define FILE_ATTRIBUTE_RECALL_ON_DATA_ACCESS 0x00400000
    #endif
    
    
    bool Jidac::isrealfile(const string& i_filename) 
    {
        // Usa GetFileAttributesEx per ottenere informazioni sul file
        WIN32_FILE_ATTRIBUTE_DATA fileAttributes;
        if (GetFileAttributesEx((utow(i_filename.c_str()).c_str()),GetFileExInfoStandard, &fileAttributes)) 
    	{
            // Controlla se il file è un "placeholder" (FILE_ATTRIBUTE_RECALL_ON_DATA_ACCESS o FILE_ATTRIBUTE_PINNED sono indicatori comuni)
            if (fileAttributes.dwFileAttributes & FILE_ATTRIBUTE_RECALL_ON_DATA_ACCESS) 
                return false; // File non scaricato completamente
    		else
    			return true; // File scaricato completamente
        } 
    	else 
    	{
    		if (flagverbose)
    			myprintf("10836$ Cannot get attribute for <<%Z>>\n",i_filename.c_str());
    		return false; // Assume che non sia disponibile in caso di errore
        }
    }
    #endif
    
    

    😄

  15. flawrist commented on Feb 24, 2025

    @flawrist
    Author

    Dear @Cyan4973, we encountered a similar Unicode issue when developing xxhash, and found a solution in the following thread:

    Please, consider copying that code or using some of the suggestions of @fcorbelli. I happily use xxhash every day, along with b3sum, gpg, croc, rg, 7-zip, but not zstd as it skips (does not process) Unicode filenames under Windows 7.

  16. fcorbelli commented on Feb 24, 2025

    @fcorbelli

    BTW Windows 7 and Windows XP are different, either

    Windows 7 have a very strange console output support.

    For XP you must dynamically load the functions from kernel32.dll

    And, finally, you'll need some "smarter" function to enumerate files, like
    "add myfile foldersomething"

    And it is just the VERY beginning 😅

    #ifndef unix
    typedef HANDLE (WINAPI* FindFirstStreamW_t)
                       (LPCWSTR, STREAM_INFO_LEVELS, LPVOID, DWORD);
    FindFirstStreamW_t findFirstStreamW=0;
    typedef BOOL (WINAPI* FindNextStreamW_t)(HANDLE, LPVOID);
    FindNextStreamW_t findNextStreamW=0;
    typedef BOOL (WINAPI* GetFinalPathNameByHandleW_t)(HANDLE, LPWSTR,DWORD,DWORD);
    GetFinalPathNameByHandleW_t getFinalPathNameByHandleW=0;
    #endif // corresponds to #ifndef (#ifndef unix)
    

    And something like that

    #ifndef unix
    	HMODULE h=GetModuleHandle(TEXT("kernel32.dll"));
    	if (h==NULL)
    		printerr("14717","GetModuleHandle",0);
    	else
    	{
    		findFirstStreamW=(FindFirstStreamW_t)GetProcAddress(h, "FindFirstStreamW");
    		findNextStreamW=(FindNextStreamW_t)GetProcAddress(h, "FindNextStreamW");
    		///longpath
    		getFinalPathNameByHandleW=(GetFinalPathNameByHandleW_t)GetProcAddress(h, "GetFinalPathNameByHandleW");
    	}
    	/*
    	if (getFinalPathNameByHandleW==NULL)
    	{
    		myprintf("00576: GURU: getFinalPathNameByHandleW is NULL (very old windows?)\n");
    		seppuku();
    	}
    	*/
    	if ((!findFirstStreamW) || (!findNextStreamW))
    		myprintf("00577$ Alternate streams not supported in Windows XP.\n");
    #endif // corresponds to #ifndef (#ifndef unix)
    
    
  17. flawrist commented on Feb 24, 2025

    @flawrist
    Author

    @easyaspi314 is the hero who invented universal Unicode fix for xxhash.
    I hope that she is doing well, survived COVID-19 and other shocks of recent years.
    Obviously, it's easier for her to tell us how to properly import that solution into zstd.

    I'm ready to test a compiled zstd binary with a fix. 🙏 🙏 🙏

  18. easyaspi314 commented on Feb 26, 2025

    @easyaspi314

    I might be able to whip up a solution, yes

  19. flawrist commented on Mar 19, 2025

    @flawrist
    Author

    Dear @easyaspi314, how are you doing? Unicode dragon is hard to defeat?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions