Audit the trl-internal-testing repos that have no generation script
Follow-up to #7186, where we agreed on the principle: every repo under trl-internal-testing should have its generation script in this repository, datasets included. A repo nobody can regenerate is a repo nobody can reason about.
Deleting a Hub repo is irreversible, and the set is not uniform, so this issue goes through them one by one. Nothing gets deleted until it is agreed here.
What was checked
Every repo in the org cross-referenced against the generation scripts in scripts/generate_tiny_models/ and scripts/, then every deletion candidate cross-referenced against every repo in the huggingface org that mentions trl-internal-testing: this repository, trl-internal, transformers, peft, diffusers, sentence-transformers, blog, h4, community-evals, optimum-habana, open-r1 and the new-model-addition-* repos.
75 model repos, 61 generation scripts. The grouping below follows @qgallouedec's review.
Transfer to the consumer's test org
Agreed approach: rather than writing scripts for repos whose consumers are elsewhere, move the repo to the consumer's own test org. A Hub transfer redirects the old id and preserves download counts, so their CI keeps passing and they can update ids at their own pace. Each needs the receiving team's agreement.
-
tiny-random-LlamaForCausalLM:peft, 27 references, topeft-internal-testing -
tiny-Llama4ForCausalLM:peft, 12 references, topeft-internal-testing -
tiny-DbrxForCausalLM:transformers, alsosentence-transformers, tohf-internal-testing -
tiny-BloomForCausalLM: proposed addition to this group.transformersuses it attests/models/auto/test_tokenization_auto.py:544as a fixture for a repo whosetokenizer_classno longer exists, so it cannot be regenerated without destroying what it tests. Our own 9 references are all for "a tokenizer with no chat template", whichtiny-GPT2LMHeadModelalso provides and does have a script. -
example-images: proposed addition. A documentation image host with nothing to generate, used bytransformersdocs. -
tiny-Qwen3_5ForConditionalGeneration: the case this approach does not cover. Its only consumer istrl-internal, 17 references, which has no separate test org.
Explicit exception to the rule
-
dummy-GPT2-correct-vocab: agreed to leave the generation script on the model card. It predates thetiny-convention, andpush_to_hubin_common.pybuilds ids as{prefix}-{ClassName}[-{suffix}], which can never produce this name.
No consumer anywhere, clear to delete
-
tmp-tiny-DbrxForCausalLM: the name suggests a leftover from a debugging session -
tiny-Qwen2ForCausalLM-2.5-Coder: generator removed in #7186 -
tiny-PaliGemmaForConditionalGeneration: generator removed in #7186 -
tiny-Qwen3MoeForSequenceClassification: generator removed in #7186 -
tiny-VoxtralForConditionalGeneration: from closed #5830. Unused, andvoxtral_for_conditional_generation.pyis still on theaudiobranch if the feature returns. -
zen-audio: from closed #5830. Unused, andgenerate_zen_audio_dataset.pyis still on theaudiobranch if the feature returns. -
tldr-preference-trl-style,hh-rlhf-trl-style,Anthropic-hh-rlhf-processed,tiny-ultrafeedback-binarized,sentiment-trl-style,mlabonne-chatml-dpo-pairs-copy,descriptiveness-trl-style: legacy datasets predating the current test data, referenced nowhere
Deletion blocked by a consumer outside this repository
The sweep found these after the grouping above was first written, so treat any earlier agreement on them as superseded.
-
tiny-DeepseekV4ForCausalLM: ticked as safe before the sweep.sentence-transformersnames it intests/base/modules/transformer/transformers_tiny_models.json, though itsdeepseek_v4architecture is xfailed there. -
tiny-Phi3ForCausalLM: #7239 cleared the last reference here, butsentence-transformersnames it in the same file, not xfailed, andtrl-internalstill has one intests/test_sft_trainer.py. -
tiny-Qwen2AudioForConditionalGeneration: from closed #5830, butsentence-transformersnames it in the same file, not xfailed. -
dolly-chatml-sft: used here attests/test_sft_trainer.py:2515and once intrl-internal. Could be freed by switching that QLoRA test tozen. -
descriptiveness-sentiment-trl-style:trl-internaldocumentation and example command lines -
hh-rlhf-helpful-base-trl-style:trl-internalexample command lines -
tldr-preference-sft-trl-style:h4,experiments/template/config_ppo.yaml
sentence-transformers is a systematic consumer rather than incidental: its update_transformers_tiny_models.py scans trl-internal-testing as one of its source orgs, and the resulting map drives parametrized tests through a conftest.
A caveat on download counts
They are tempting as a usage signal and they are unreliable in both directions. tiny-PeftModel is loaded by three tests on every run and reports zero downloads, because the counter keys on config.json and that repo only has adapter_config.json. In the other direction, several repos with no identifiable consumer anywhere carry six-figure monthly counts. So the grouping above is based on reading code, not on traffic.
Proposed rule
A repo stays if it has a generation script here, a named consumer elsewhere, or a recorded reason it can have neither. Everything else is a deletion candidate. Ticking a box means we agreed on that repo.
Source: huggingface/trl