#2484·doccano

Annotation of texts with non-BMP characters (e.g., emoji)

Author: TopCoder2KCreated Mar 31, 2026Updated Mar 31, 2026

Hi all!

Thank you for this open-source tool that saved me a lot of time (I needed an annotation tool for testing NER-models on our internal data). One thing caused me a headache: annotations created using Doccano were sometimes wrong when used in Python. I spent a lot of time in order to figure out the reason — Unicode-based indexing in Python didn't match the indexing used in Doccano (JavaScript uses utf-16, right?). The fact that Doccano used utf-16-based indexing didn't worry me when I was uploading the data since the potential inconsistency completely escaped my notice.

Feature description

Related to: Notifications? Specific solution: I propose creating a warning that informs a user about a potential inconsistency between Python and Doccano when a non-BMP character is detected in an annotated text. Moreover, a remark could be added to the Tutorial.