#1622·stanza

Switch from oscar 2023 to oscar community

Author: AngledLuffaCreated Jun 21, 2026Updated Jul 1, 2026
Labelsenhancementfixed on dev

Oscar 2023 is currently

  • not working for datasets >= 4.0
  • not accepting new users

so the current charlm build instructions only work for people who already have access and pinned their datasets version < 4

oscar community would require a separate processing script, not terribly onerous, but it does need to be done

Advantage: there are multiple slices of oscar this way. Although we need to deduplicate it. According to Claude:

[we can dedup] using the tlsh field, with a distance cutoff — the OSCAR docs note a cutoff of <40 yields a 0.07% false positive rate at 49.6% detection, while <100 yields 6.43% FP at 94.5% detection, so there's a real precision/recall knob to set

(dedup described here for oscar 2023: https://oscar-project.github.io/documentation/versions/oscar-2301/)