Switch from oscar 2023 to oscar community
Author: AngledLuffaCreated Jun 21, 2026Updated Jul 1, 2026
Labelsenhancementfixed on dev
Oscar 2023 is currently
- not working for datasets >= 4.0
- not accepting new users
so the current charlm build instructions only work for people who already have access and pinned their datasets version < 4
oscar community would require a separate processing script, not terribly onerous, but it does need to be done
Advantage: there are multiple slices of oscar this way. Although we need to deduplicate it. According to Claude:
[we can dedup] using the tlsh field, with a distance cutoff — the OSCAR docs note a cutoff of <40 yields a 0.07% false positive rate at 49.6% detection, while <100 yields 6.43% FP at 94.5% detection, so there's a real precision/recall knob to set
(dedup described here for oscar 2023: https://oscar-project.github.io/documentation/versions/oscar-2301/)
Source: stanfordnlp/stanza