[Data]: Systemic duplicate wikiDataId across contributions/cities/MX.json (~3,276 records, ~35% of file)
Summary
While fixing #1623 (Epitacio Huerta misclassified state), a 4-record run (ids 70310-70313) was found to share one incorrect wikiDataId — the result of what looks like a copy-forward bug during the original 2019 bulk import (see #1630, .github/fixes-docs/FIX_1623_SUMMARY.md). Pullfrog's automated review of that PR flagged that this pattern is not isolated to those 4 records — it's systemic across the whole file. I independently re-verified the scan; numbers below are mine, cross-checked against pullfrog's.
Scope
contributions/cities/MX.json: 9,321 total records.- 1,939 groups of records share an identical
wikiDataIdwith at least one other record. - 5,215 records fall inside one of these groups; assuming (generously) one "rightful" owner per group, that leaves ~3,276 records (≈35% of the file) carrying a
wikiDataIdthat very likely belongs to a different place. - 1,924 of the 1,939 groups (99%) are fully consecutive
idruns — the same shape as the70310-70313case, strongly suggesting the same root cause (an import-time bug that failed to look up/assign a fresh Wikidata match per row and instead carried the previous row's value forward). - Of the duplicate groups: 1,784 contain records with different
namevalues, and 1,665 span more than onestate_code— both are strong signals these are genuinely different places wrongly sharing one ID, not legitimate multi-locality Wikidata entities.
Example groups
| wikiDataId | # records | Sample |
|---|---|---|
Q1144317 |
11 | 69819 Ejutla de Crespo (OAX), 69820 Ekmul (YUC), 69821 Ekpedz (YUC), 69822 El Abra (TAM)... |
Q20219114 |
10 | 70256 El Vergel (DUR), 70257 El Vicarlo (GUA), 70258 El Volador (VER), 70259 El Walamo (SIN)... |
Q20237772 |
9 | 69682 Detiña (MEX), 69683 Diamante de Echeverría (CHP), 69684 Dieciocho de Marzo (MIC)... |
Q5654722 |
3 | 68017 Abasolo del Valle (VER), 68018 Abdenago C. García (CHH), 68019 Abelardo L. Rodríguez (CAM) |
(Full group listing available on request — didn't want to dump 1,939 groups into this issue body.)
What's already fixed
PR #1630 corrects the 4-record case that surfaced this pattern (70310-70313 — Epazoyucan, Epigmenio González, Epitacio Huerta, Ermita de Guadalupe), each re-verified against Wikidata's raw entity JSON individually.
What this issue tracks
The other ~1,935 groups (~3,272 records) are not yet fixed. This is deliberately not being attempted in one pass — each record needs an individual Wikidata lookup/verification (by name, admin division, and coordinate match, the same method used in #1630) before its wikiDataId can be corrected; there's no safe bulk-automatable fix since the "correct" ID for each row has to be looked up per place, not derived from the existing (wrong) data.
Suggested approach for whoever picks this up
- Regenerate the full list of duplicate-
wikiDataIdgroups programmatically (script above; happy to share it). - For each group, verify each member's real-world identity independently (name + state + coordinates) via Wikidata, exactly as done in #1630/
FIX_1623_SUMMARY.md. - Batch fixes by state or by group size to keep individual PRs reviewable — 3,276 records in one PR would be unreviewable.
- Worth checking whether other country files have the same import-era bug — this scan only covered
MX.json.
Also worth checking
Whether this same 2019-era import bug affected other large country files (US.json, IN.json, BR.json, etc.) — not yet scanned.
Source: dr5hn/countries-states-cities-database