SyntaxWarning: invalid escape sequence '\w'
If you run a python interpreter of a certain recency (I think 3.11, maybe 3.12, or later) on bindings\python\py_src\tokenizers\pre_tokenizers\__init__.pyi, you will get the error: 525: SyntaxWarning: invalid escape sequence '\w'. Same if you run a typechecker that interprets this file. This is because \w is not a valid escape in Python, and it occurs in a non-raw string. So, the escape should be changed to \\w or the string should be made raw. I did this in https://github.com/huggingface/tokenizers/pull/1764 before it was pointed out to me that this file was autogenerated and shouldn't be edited. So, here's the real form of the problem, which I don't have the knowledge to solve directly myself:
/// This pre-tokenizer simply splits using the following regex: `\w+|[^\w\s]+`This is a valid rust documentation comment, but whatever turns it into a python docstring needs to be updated to respect the python escape rules. (Or, maybe, it wants to let you use python escapes, and so we must correct this rust documentation comment to escape the escapes? Hmm...)
The impact and urgency of this issue is low, but it would get rid of an annoying warning I see all the time, and make the code technically more correct.
Source: huggingface/tokenizers