Sync reserved characters proposal

As a user, I want to sync filenames containing reserved/unsupported/special characters on any filesystem. Specifically, I want to sync filenames containing "*:<>?| characters on NTFS/exFAT/FAT32 filesystems, which disallows these characters in filenames. These filesystems are most often found on Windows and Android systems, but any system may have attached media using these formats.

Sync reserved characters proposal v2.3 - 2026-Apr-27

1. Preamble

This proposal is authored by @rasa and @JanKanis, and was inspired by JanKanis' comments here. It was last updated on 03-Jun-24. Feedback appreciated. An editable copy is here.

2. Abstract

Syncthing will report "Out of Sync" errors on peers where the underlying filesystem does not allow certain filenames that are allowed on other peers. This proposal addresses this issue. On https://roadmap.syncthing.net/, the issue is tied for 32nd, but was locked over five years ago, so it can't be voted on any more. If you value this proposal, click the thumbs up icon on this issue instead.

3. Motivation

As a user, I want to sync filenames containing reserved/unsupported/special characters on any filesystem. Specifically, I want to sync filenames containing "*:<>?| characters on NTFS/exFAT/FAT32 filesystems, which disallows these characters in filenames. For more information, see https://en.wikipedia.org/wiki/Filename#Comparison_of_filename_limitations

Use cases

@JanKanis' comment below documents several use cases.

4. Rationale

We chose encoding filenames using either Rclone's full-width Unicode characters, or Unicode's Private Use Area characters (explained below), as this encoding method is effectively an industry standard, as it is how GitBash, Windows Subsystem for Linux (WSL), Cygwin^1, MSYS, Linux's CIFS driver, and other platforms, and was first implemented in 1996.

5. Specification

Each folder will be configured to use an encoder. Initially, there will be three encoders: the "None encoder", the "Rclone encoder", and the "WSL encoder".

All existing folders will start out "using" the None encoder. The None "encoder" isn't really an encoder. It's the way Syncthing works right now. In fact, no new code will be executed when a folder is configured to "use" the None encoder.

Newly created folders will default to using the None encoder (expect on Windows, where the Rclone encoder will be the default).

The user can change a folder's encoder setting via the GUI, but only via Actions > Advanced > Folders > Folder. If possible, when the user clicks "Save", a dialog box will pop up that explains the potential pitfalls, and asks for a further confirmation.

The None encoder

The None encoder, as described above, is not really an encoder, as it reads and writes filenames on disk "as is," without any encoding. It does not reject or ignore any filenames it receives. It is designed to be used on filesystems that allow all characters except / and NUL, but it can be used on any filesystem. If it's used on a FAT-based filesystem, filenames a peer receives containing reserved characters won't be able to be written to disk, leading to out-of-sync errors.

The Rclone encoder

The Rclone encoder replaces reserved characters with their full-width Unicode equivalents. Here's the relevant portion of the encoding:

ASCII Char	ASCII Code	Full-width Unicode Char	Unicode Code
"	0x22	＂	0xFF02
*	0x2A	＊	0xFF0A
/	0x2F	／	0xFF0F
:	0x3A	：	0xFF1A
<	0x3C	＜	0xFF1C
>	0x3E	＞	0xFF1E
?	0x3F	？	0xFF1F
\	0x5C	＼	0xFF3C
|	0x7C	｜	0xFF5C

This encoding is fully compatible with all Windows programs, including Windows Explorer, Windows Command Prompt (CMD), etc.

See https://rclone.org/local/#restricted-characters to see the complete encoding.

The WSL encoder

The WSL encoder is designed to be used on filesystems that disallow the characters \"*:<>?| in filenames. When filenames with these characters are written to disk, the WSL encoder encodes the filename in a format that the filesystem will accept. When read from disk, the filename is decoded to its original filename, before being sent to the other peers. The WSL encoder can be used on any filesystem. For example, you might want to run it on an ext4 filesystem, if you intend to copy (or backup) the files to a FAT-type filesystem.

To clarify, encoding is something that happens purely locally. File names sent over the wire in the Syncthing protocol always use the original pre-encoded names, and peers don’t know if another peer is using any sort of encoder when storing their files.

Unicode Private Use Area (PUA) characters

The WSL encoder will replace reserved characters with Unicode Private Use characters (\xf000 - \xf0ff). A character will be replaced by adding \xf000 to its code point, so for example a ? (code point \x003f) is replaced by \xf03f. It requires that the underlying filesystem allows Unicode characters, such as NTFS, exFAT, and VFAT.

ASCII Char	ASCII Code	PUA Code
"	0x22	0xF022
*	0x2A	0xF02A
/	0x2F	0xFO2F
:	0x3A	0xF03A
<	0x3C	0xF03C
>	0x3E	0xF03E
?	0x3F	0xF03F
\	0x5C	0xF05C
|	0x7C	0xF07C

6. Backwards Compatibility

Since all folders, both existing and newly created ones, will default to using the None encoder, there are no backward compatibility issues. From the user's perspective nothing changes, and encoding-aware peers can communicate with non-encoding-aware peers without any issues.

A user can even downgrade a peer from a encoding-aware build, to a non-encoding-aware build without issue.

The only issue that can manifest, is if all of the following occurs:

1. A folder is set to use an Rclone or WSL encoder
2. The peer using an Rclone or WSL encoder received filenames that required encoding, and it saves these encoded filenames to disk.
3. The user switched the folder back to using the None encoder (or Syncthing was downgraded to a non-encoding-aware build, effectively "using" the None encoder).

First, we'll describe the problem in detail, and then the proposed solution.

The problem

We have two peers: N and F. Both use the None encoder. Peer F's filesystem is FAT, and so it had an out-of-sync error when it received a file named acolon:. Peer F switched its folder's encoder to WSL, which now can save acolon: as acolon\xf03a, and the out-of-sync error goes away.

Now, peer F switched the folder's encoder from WSL, back to None. The None encoder on peer F will find the file acolon\xf03a on disk and sync this file to peer N, which will see it as a new file, and save it. Peer N now has two files named acolon: and acolon\xf03a, which are effectively the same file.

Peer N will then sync these files with peer F. Peer F will still accept acolon\xf03a, but will reject acolon: as it has a reserved character, leading to an out-of-sync issue.

Proposed solution

A separate CLI program is run on any peer where the folder is not on a FAT filesystem. Using the example above, the program is run on peer N. It searches for files where encoded files (acolon\xf03a) coexist with their pre-encoded equivalents (acolon:). If a pair is found, it will see if the files are the same. If they are, it will delete the encoded version (acolon\xf03a).

If the two files differ, it will display the two filenames, timestamps, sizes, and attributes to the user, and ask them to choose:

1. Keep acolon: only (by deleting acolon\xf03a)
2. Keep acolon\xf03a only (by renaming acolon\xf03a to acolon:)
3. Keep both files (so they can research, and rerun, or possibly correct manually)

Option 1 - Keep acolon:

Peer N syncs the delete of acolon\xf03a with the other peers. None peers will process the delete. WSL-encoder peers will silently ignore the delete, as they ignore all encoded filenames they receive on the wire.

Option 2 - Keep acolon\xf03a

Syncthing sees this rename of acolon\xf03a to acolon: as deleting acolon\xf03a and updating acolon:. None peers will process both the delete and update. WSL-encoder peers will ignore the delete, and update acolon:, by encoding the filename as acolon\xf03a.

Automating the process

The following startup options would automate the above selection process:

1. --decoded - always select the pre-encoded filename (choice 1. above)
2. --newer - always select the newer of the two files
3. --encoded - always select the encoded filename (choice 2. above)
4. --older - always select the older of the two files

The program will not back up files before deleting them. If a user wants backups, they should turn on versioning on a None peer, before running the program.

Which option is most likely to be the right one?

Option 1, "Keep acolon:", will almost always be the best choice. Why? Because pre-encoded filenames almost always originated on non-FAT peers, as users cannot generally create these filenames on encoding peers. The most likely way a user on an encoding peer created an encoded filename themselves, is if they created the file via a CLI environment, such as GitBash, Cygwin, MSys2, WSL, etc. So, since they most likely didn't author the file, it's less likely that a FAT peer will be the last one updating it.

7. How to Teach This

The documentation will explain the benefits and drawbacks of changing a folder's encoder.

8. Reference Implementation

@rasa has implemented this proposal in a draft PR with full unit tests. Integration tests have been provided using the new framework based on @calmh's work in https://github.com/syncthing/syncthing/pull/9266. @rasa will also draft a PR for the documentation needed.

9. Alternatives

Other encoding methods that could be implemented

URL encoding

This encoding replaces reserved characters with their URL-encoded equivalent. See https://en.m.wikipedia.org/wiki/Percent-encoding. This would be a good choice on filesystems that don't support UTF-8 characters. Proposed by @AudriusButkevicius.

Samba's Catia encoding

This encoding replaces reserved characters using the mapping "→¨ *→¤ /→ø :→÷ <→« >→» ?→¿ \→ÿ |→¦. This would be a good choice if the user wants to encode to more visually related characters. See https://www.samba.org/samba/docs/current/man-html/vfs_catia.8.html. Proposed by @JanKanis.

10. Open Issues

None that we are aware of, but here's a good place to list a potential future enhancement:

Warning the user the encoder was changed

Due to the duplicate file issue noted above, we may want to alert the user whenever a folder's encoder is changed. To do this, we can update .stfolder/syncthing-folder-xxxxxx.txt (See https://github.com/syncthing/syncthing/pull/9525), with either Encoder: None, 'Encoder: Rclone', or Encoder: WSL, if the entry is missing.

Then whenever Syncthing starts up, if the encoder in the .stfolder file listed Rclone or WSL, but config.xml lists None, a warning is shown in the GUI. The user can select "Revert", "Accept" or "Ignore". If they select "Revert", the encoder setting is changed back to FAT in config.xml. If they select "Accept", the .stfolder file is updated to contain Encoder: None. If they select "Ignore", the message goes away, until Syncthing restarts.

We could also provide CLI users with these options: --report-on-encoder-changes: if the encoder was switched, scan the filesystem, and if there are duplicate files, log the duplicates, and continue --abort-on-encoder-changes: do the above, but quit instead --accept-encoder-changes: Update the .stfolder file with Encoder: None --revert-encoder-changes: Switch the encoder back to Rclone/WSL in the config.xml file

If no option is provided, a warning about the encoder change is logged.

11. Footnotes

For reference, see: https://cygwin.com/cygwin-ug-net/using-specialnames.html http://msdn.microsoft.com/en-us/library/aa365247%28VS.85%29.aspx https://docs.microsoft.com/en-us/windows/win32/fileio/naming-a-file https://en.wikipedia.org/wiki/Filename#Comparison_of_filename_limitations . https://amigotechnotes.wordpress.com/2015/04/02/invalid-characters-in-file-names/

For implementations, see https://github.com/mirror/newlib-cygwin/blob/1b7c72fdcc4bde7520407d2d3364146f04fb8312/winsup/cygwin/path.cc#L466 https://github.com/winfsp/winfsp/blob/4fdec4d37fb4e56b6d810714f5a201e275211aaf/src/shared/ku/posix.c#L1318 https://github.com/torvalds/linux/blob/68f090f09b7241a2f019191c5c737741f13a5c17/fs/smb/client/cifs_unicode.h#L27