UnicodeDecodeError (hard crash) in find_referencing_symbols when a referencing file's encoding differs from the project encoding

Summary

find_referencing_symbols raises an unhandled UnicodeDecodeError when the file containing a reference is encoded differently from the project's configured encoding. Since serena (by design, see #121) assumes one encoding per project, the tool should either degrade gracefully (skip the file with a warning, like search_for_pattern already does) or fall back to encoding detection — but currently it crashes hard and the agent loses the whole tool result.

This was originally found while working with MQL4 sources (MetaEditor saves UTF-16-LE by default), where a backup folder containing UTF-16 copies sat inside the project while the project encoding was utf-8. The repro below, however, is pure Python and requires nothing MQL-specific.

Note: this is related to #121 (encoding config by project, merged in #441) and touches the same assumption discussed in #993 ("Serena assumes a consistent file encoding within a project"). Users cannot always control this assumption: backups, third-party drops, or tool-generated files can introduce files with a different encoding (UTF-16-BOM is common on Windows: PowerShell Out-File, Notepad "Unicode", MetaEditor).

Reproduction

1. Create a project with mixed encodings:

mkdir mixed-encoding-project && cd mixed-encoding-project
cat > main_utf8.py << 'EOF'
from utf16_module import helper_function

def main():
    helper_function(42)

if __name__ == "__main__":
    main()
EOF
cat > utf16_module.py << 'EOF'
# UTF-16 file with BOM (as saved by PowerShell / some Windows editors)
def helper_function(value):
    print("helper called with", value)

def caller_function():
    helper_function(42)

def main_entry():
    caller_function()
EOF
# Convert utf16_module.py to UTF-16 LE with BOM:
python3 -c "
data = open('utf16_module.py', 'rb').read()
open('utf16_module.py', 'wb').write(b'\xff\xfe' + data.decode('utf-8').encode('utf-16-le'))
"

1. Activate the project with Serena (Python LSP backend, default encoding: utf-8 in .serena/project.yml).

2. Call find_referencing_symbols for caller_function in utf16_module.py. The reference (from main_entry) is located in the UTF-16 file itself, so the tool must read it to build content_around_reference.

Observed behavior

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

The exception propagates to the MCP client and the tool result is lost.

Root cause

The solidlsp layer is robust: FileUtils.read_file (src/solidlsp/ls_utils.py) catches UnicodeDecodeError and falls back to charset_normalizer — that's why document symbols and raw references work fine on the mixed project (verified).

The crash comes from the serena-core read path, which has no fallback:

• serena/util/file_proxy.py → LocalProjectFileProxy.get_contents(): open(abs_path, encoding=self._project.project_config.encoding) — strict, no fallback.
• serena/project.py → Project.read_file() → retrieve_content_around_line() — used by FindReferencingSymbolsTool.apply() (serena/tools/symbol_tools.py) for each referencing symbol.

The same strict proxy is used by Project.read_file generally (also backing read_file MCP tool and the code editor's JetBrainsCodeEditor path).

Also note the write side has a worse failure mode in mixed projects: CodeEditor._save_edited_file and create_text_file write with the project encoding, so editing a symbol in a UTF-16 file would silently rewrite it as UTF-8.

Expected behavior

Options (in increasing scope):

1. Graceful degradation (minimal fix): in FindReferencingSymbolsTool.apply, wrap the retrieve_content_around_line call; on UnicodeDecodeError set content_around_reference to a placeholder (e.g. "<unreadable: file encoding differs from project encoding>") and keep the reference metadata. search_files in serena/util/text_utils.py already uses this per-file exception pattern.
2. BOM-based per-file fallback (nice to have): in LocalProjectFileProxy.get_contents, check the file's first bytes for a BOM (FF FE → utf-16, FE FF → utf-16-be, EF BB BF → utf-8-sig) before using the project encoding; without a BOM, fall back to the existing charset_normalizer detection used by FileUtils.read_file. MetaEditor, Notepad and PowerShell always write BOMs, so this covers the practical mixed-encoding cases deterministically.
3. Write-path symmetry: when saving an edited file, preserve its existing on-disk encoding (detect before write, BOM included) so edits don't silently re-encode files.

Option 1 alone prevents the hard crash; 2–3 make mixed-encoding projects work end-to-end. Happy to provide more details or test a patch.