#811·cleanlab

[Datalab new issue type] detect incorrect usage of placeholder-values in features

Author: jwmuellerCreated Aug 12, 2023Updated May 17, 2026
Labelsenhancementgood first issuehelp-wanted

New Datalab issue type called something like placeholder that checks features for potentially invalid use of placeholder values.

Motivation: placeholder values are sometimes used in place of missing/null observations in numeric data, eg. -99 in a column of values that are otherwise non-negative. These can negatively affect modeling if somebody doesn't realize these aren't real values but rather just placeholders.

The best implementation of this must be compute-efficient and will require some exploration to figure out the best method for detecting placeholders when they exist, but not flagging false positives in datasets where there are real values that just happen to look like placeholders.

One option is a two-stage algorithm:

First see whether there are any candidate placeholders (i.e. is there any column of features with continuous numbers across a broad range, but with one suspicious constant repeatedly occurring throughout?). This should be done in a very efficient manner (eg just based on subset of the data) and if not, the check should terminate immediately.

Second consider each candidate placeholder and estimate whether it actually seems problematic or not (is the value of the candidate-placeholder a huge outlier in the other values of this column? is the candidate placeholder a negative number while rest of column is non-negative, etc). Sophisticated variant of this could replace the candidate with missing values, try to impute them based on the other features, and see if the imputed values are significantly different than the placeholder values. Want to ensure very few false positives here