[Bug] Files with UTF-16 LE BOM are displayed as raw binary/null-separated bytes

Operating System and Version

Windows 10 IoT Enterprise LTSC 21H2

Distribution

.exe installer downloaded from the latest release webpage section

Description

When opening a text file encoded in UTF-16 LE with BOM (for example, output generated by Windows PowerShell redirection >), the status bar correctly recognizes the encoding as UTF-16LE BOM.

However, the editor content is not decoded. Instead, the raw bytes are appended directly into the buffer:

• The byte order mark \xFF\xFE is shown at the beginning.
• Each ASCII character is interspersed with null bytes (\x00).
• Carriage returns (\r / 0x0D) are misrecognized as old Macintosh CR line breaks, causing the line count to roughly double (e.g. 22 lines become 43 lines, and character count doubles from 878 to 1758, see the screenshot image for details).

Comparison Screenshot

In image:

• Windows PowerShell redirection > generated file content: the Zen of Python.
• Comparison with in: Windows 10 built-in Notepad, Notepad++, Visual Studio Code, and (not an editor) Google Chrome. (All render the file correctly.)

Likely Cause in Code

Looking at src/ScintillaNext.cpp around readFromDisk():

if (bomType == BomType::Utf16BE || bomType == BomType::Utf16LE) {
    // Um...ignore this for now?
}
appendText(chunk.size(), chunk.constData());

It appears the UTF-16 BOM is detected for status display, but decoding the UTF-16 bytes into UTF-8 before passing them to appendText() is currently commented/unimplemented.

Steps to Reproduce

On modern Windows, after having Python installed, open the Windows PowerShell, then command:

powershell -c "python -m this > python-m-this_powershell.txt"

python-m-this_powershell.txt