Experiment search ranks a text value above the highest number when sorting descending
Bug
FIX:- #1818
ExperimentStore.search(sort_by=..., descending=True) ranks a run whose sort value is text above the run holding the highest number.
A field can hold a number on some runs and text on others — a crashed run logged as "failed", a pending one as "n/a". _order_key handles that by tagging numbers (0, ...) and everything else (1, ...) so numbers group first. But _rank_key returns that tag inside the sort key and _rank applies reverse=descending to the whole key, so the tag reverses too and the text group jumps ahead of every number.
Reproduce
store.log_experiment("baseline", params={"accuracy": 0.72})
store.log_experiment("tuned", params={"accuracy": 0.91})
store.log_experiment("crashed", params={"accuracy": "failed"})
store.search(sort_by="accuracy", descending=True)Actual:
['crashed', 'tuned', 'baseline']Expected:
['tuned', 'baseline', 'crashed']The best model is listed second, behind a run that never produced a number.
Worse with paging
_rank's heap path (heapq.nlargest) selects on the same key, so bounded queries drop real results:
store.search_page(sort_by="accuracy", descending=True, limit=2)
# → ['crashed', 'tuned'] — 'baseline' evicted for a run with no scoreNotes
Ascending order is correct; only descending is affected. No error or warning is raised, which is likely why it has gone unnoticed.
Reproduces on dev @ 690da11. It reduces to plain Python tuple comparison with no visdom involved, so it is not environment-specific:
def order_key(v):
return (0, float(v), '') if isinstance(v, (int, float)) and not isinstance(v, bool) else (1, 0.0, str(v))
rows = [(5, 'run_five'), (10, 'run_ten'), ('pending', 'run_str')]
sorted(rows, key=lambda r: order_key(r[0]), reverse=True)
# → [('pending', 'run_str'), (10, 'run_ten'), (5, 'run_five')]Source: py/visdom/experiments/store.py, _rank_key (~line 119) and _rank (~line 132).
Source: fossasia/visdom