Is a blank cell signal, or just missing?

Is a blank cell signal, or just missing? Sometimes an empty cell is the most informative thing in the row. The trouble is that you usually only know which case you're in by reading the data dictionary — and that doesn't scale to 800 columns named . So we measure it instead, then check the answer against the literature. Ames housing · 1,460 sales · 79 columns · 19 of them contain blanks "Drop any column that's more than 70% missing." I've written that line into more pipelines than I can count. On Ames it deletes four columns — and three of them have real price signal sitting in the gap. The blanks in this dataset are structural. A blank doesn't mean the value was lost; it means the house has no garage. A blank means no alley access. The emptiness is the measurement. That's easy to see here because the columns have English names and a published data dictionary. It is not easy to see on a vendor feed of anonymised features, which is what most real projects look like. So the question worth answering isn't "does missingness carry signal" — it's can you tell, without knowing what the column means? 0.41 R² from the blank/not-blank pattern alone — every value discarded 1.00 AUC recovering the garage blanks from other columns' values ±0.9% Total spread across five strategies — inside a ±1.5% CV noise band 1 · A blank cell has a price tag Start with the crude check: does sale price differ between rows where a column is blank and rows where it isn't? Columns that go blank on the same rows describe one fact, so the five garage columns collapse into one. Fig 1. Median sale price, blank rows vs. valued rows. No garage is a $68k median discount on a $163k median house. Note the sign flip: houses that have an alley or fence are the cheaper ones — those features mark older, denser blocks. "Blank = worse" is not a rule you can assume. Then the harder test. Throw away every value in the table and keep only a 19-column matrix of / — was this cell empty? Fit on that and nothing else. Fig 2. Five-fold CV. A model that has never seen a floor area, a neighbourhood or a year built reaches R² = 0.41 purely from which cells are empty. The blanks are loud. But we only understood them by reading the data dictionary — and that's the part that doesn't scale. 2 · Two questions, and most people only ask the first For every column that contains a blank, ask two things — neither of which requires knowing what the column means. Axis 1 — does the blank move the target? Standardised difference in log price between blank and non-blank rows, with a permutation test and a Benjamini–Hochberg correction across the eight column groups. This is the axis everybody already runs. Axis 2 — is the blank already recoverable from other columns' values? Predict "is this cell blank?" from every other column, with every other column's own missingness destroyed first, so the classifier can only use values you can actually observe. AUC near 1.0 means some other column already tells you. AUC near 0.5–0.8 means nothing else in the table knows. Axis 1 alone is a trap. A blank can be powerfully associated with the target and still be worth nothing, because a neighbouring column carries the same fact. Column group % blank effect adj. p recover AUC verdict Fence 80.8 +0.45 0.0007 0.765 flag it Alley 93.8 +0.55 0.0007 0.942 flag it Bsmt ×5 2.6 −1.23 0.0007 0.982 flag it MiscFeature 96.3 +0.41 0.0051 0.983 flag it LotFrontage 17.7 +0.12 0.0930 0.824 no signal MasVnrType 59.7 −0.79 0.0007 0.999 redundant FireplaceQu 47.3 −1.02 0.0007 1.000 redundant Garage ×5 5.5 −1.41 0.0007 1.000 redundant Fig 3. The two axes together. Bubble size is the share of rows blank. Everything in the shaded band is already encoded somewhere else in the table. The interesting cases sit on the left. at AUC 0.77 and at 0.94: nothing else in the table encodes them. Their blanks are the sole carrier of their fact — and they are precisely the columns a "drop >70% missing" rule deletes. , the classic "just median-impute