consider optimizing MultiReader segment lookup in bitio
Reading through pkg/bitio i noticed that MultiReader.ReadBitsAt linearly scans readerEnds, also the existing TODO: smarter, track index? comment. So i tried a local proof of concept using bin search , and benchMarked ReadBitsAt method. For random 64-bit reads on my machine , the median results were :
16 readers: current scan ~38.5 ns/op binary search ~51.0 ns/op
1024 readers: current scan ~299 ns/op binary search ~92 ns/op
65536 readers: current scan ~15.95 us/op binary search ~258 ns/op
So bin search clearly helps with many segments. With that said. I don't want to assume that large segment counts are common enough to justify the trade-off. So ig the question is , will this a path you would be interested in optimizing? If so, I'd be happy to prepare a PR.
Source: wader/fq