bucket_headings(): np.sort breaks the height/label pairing, and n_clusters is not clamped to distinct sizes
Two bugs in bucket_headings() (marker/processors/sectionheader.py), verified against current master.
1. np.sort destroys the height/label pairing
Line 93:
data_labels = np.sort(data_labels, axis=0)np.sort(..., axis=0) sorts each column independently, so each height is no longer paired with its own cluster label, and cluster_means and the heading_ranges loop below consume mismatched rows, permuting heading levels.
Rows [[10, 0], [24, 1], [14, 2]] become [[10, 0], [14, 1], [24, 2]]: height 24 moves from label 1 to 2, height 14 from 2 to 1. Sorting rows by height gives [[10, 0], [14, 2], [24, 1]].
- data_labels = np.sort(data_labels, axis=0)
+ data_labels = data_labels[data_labels[:, 0].argsort()]2. n_clusters is not clamped to the number of distinct sizes
Lines 85-86 and 89-91:
if len(line_heights) <= self.level_count:
return []
...
labels = KMeans(
n_clusters=self.level_count, random_state=0, n_init="auto"
).fit_predict(data)The guard tests sample count rather than distinct sizes, so near-identical heights get split across clusters and a heading level is invented. Real PDF line heights arrive with rendering jitter: heights [11.9, 12.0, 12.1, 17.9, 18.0, 18.1] with n_clusters=4 yield labels [1, 1, 1, 0, 2, 3] — four labels for two heading sizes.
Counting unique raw values does not help, since all six are distinct. Rounding before counting does: len(np.unique(np.round(line_heights))) is 2, and k=2 recovers [1, 1, 1, 0, 0, 0]. The rounding tolerance is a judgment call, and a gap-based merge may suit better.
data = np.asarray(line_heights).reshape(-1, 1)
+ distinct = len(np.unique(np.round(data))) # tolerance is a judgment call
+ k = min(self.level_count, distinct)
+ if k < 2:
+ return []
labels = KMeans(
- n_clusters=self.level_count, random_state=0, n_init="auto"
+ n_clusters=k, random_state=0, n_init="auto"
).fit_predict(data)Both were found while building a derivative converter (markerlite). Happy to open a PR for either or both.
Source: datalab-to/marker