bhojpuri (bho): POS model support
I was working on my Bhojpuri project and looked for Stanza support and found
there isn't any. bho_bhtb is registered in short_name_to_treebank.py, but
bho appears nowhere in default_packages.py — no treebank, pretrain, charlm,
or transformer.
UD_Bhojpuri-BHTB ships only bho_bhtb-ud-test.conllu: 357 sentences, 6,665
tokens, no train/dev. It is the same shape mixed_odia_dataset.py handles. I
tried it, and it runs on Bhojpuri unmodified, exit 0, both modes:
python stanza/utils/datasets/indic/mixed_odia_dataset.py --mode pos \
--odia_file $UDBASE/UD_Bhojpuri-BHTB/bho_bhtb-ud-test.conllu \
--dataset_name bho_bhtb --use_hindi --use_urduOdia split: 278 train / 45 dev / 34 test
hindi: 13306 → 1000 urdu: 4043 → 1000Everything Odia-specific looks cosmetic: the --odia_file arg name, the
"Odia split:" print, the or_odtb default. The docstring raises repurposing it
for more datasets as future work, but the script already does it — what's left is
a rename, not a rewrite.
Happy to do these if useful:
- Rename to target-neutral args (
--target_file,--target_shortname), donors configurable, Odia defaults unchanged — same file, not a per-language copy. - Bhojpuri charlm:
default_charlmshassdbut notbho, andcommunity_oscar_dedup.pydocuments the Sindhi recipe. I'd check for abhoCommunity-OSCAR slice and run it.
Questions first:
- Would you take the rename, or prefer separate scripts per language?
- 34 test sentences is too few for a trustworthy number. What evaluation would make Bhojpuri models shippable, or is BHTB too small to bother?
- Odia and Sindhi lean on MuRIL, which I don't think covers Bhojpuri. Does the recipe need the target in the transformer's pretraining data?
Source: stanfordnlp/stanza