Plugins

This is an idea that's been percolating in the back of my head for a while, and I want to braindump it here to instigate discussion well in advance.

Pre-history

In the very first versions of librosa, we had functionality for things like chord recognition, that really are better accomplished as supervised (trained) models. This functionality was eventually deprecated and removed, though we have retained certain estimation algorithms (beat tracking, onset detection) due to their utility in feature extraction pipelines.

The core principle that I/we landed on is that if librosa is meant to be a widely usable library, we should minimize statistical dependencies to reduce the chances of accidental overfitting by users. I still think this was largely correct and the right move to make, though it does reduce our scope.

This is all fine, but it does leave us in a bit of a historical niche when it comes to library utility, as many of the tasks that we initially set out to accomplish are better solved by standalone packages (with maybe some light dependency on librosa). This in turn leads to some instability, at least at the level of API and environment management, as pre-trained models sometimes do nasty things like version pinning that make interoperability a real pain.

Can we solve this with plugins?

The idea I have in mind is to expose entry points in librosa for common MIR tasks. A third party could then register a model for a given task (say, beat tracking), which could be accessed by something like:

beats = librosa.analyze.beat(y=y, sr=sr, ..., plugin='my_plugin_name') 

The idea here is that plugins can depend on librosa functionality, and librosa can surface a consistent API for using different estimators for common MIR tasks. Plugins can also be published and updated out of sync with our release schedule, which seems like an obvious win to me.

For the sake of consistency, we can also host plugin versions of our own classic estimators (beats, tempo, onset, etc). E.g., librosa.analyze.beat(..., plugin='ellis') or plugin='plp'.

When and why?

I'm tagging this for a 2.1 release, under the assumption that 2.0 implements array-api. This will reduce some friction between plugins and core functionality for us - plugins should be able to specify which array backend they support (eg numpy, torch, whatever.. or even array-api entirely) and we could manage whatever translations are necessary at run-time for plugin input. I do not think it will make much sense to try implementing a plugin system before we have array api support.

In terms of plugin output, after kicking the idea around multiple chatbots (gemini, copilot), what I think will make sense is to tie this to jams 2.0 annotation objects. This puts a natural roadblock in place for getting plugins off the ground, but the idea here would be to entirely offload the data interchange and validation problems to a dedicated library, and jams already exists for exactly this purpose.

Another key benefit of going through an annotation format is that we can really decouple some of the analysis parameters that are hard to make internally consistent between core library code and specialized estimators (eg hop lengths or frame lengths). It's easy enough to convert back to flat arrays like we've always used, but we could now also leverage things like automatic display (jams.display) and evaluation format conversion (jams.eval).

JAMS obviously needs a lot of maintenance and overhauling, but I'm planning to take that on anyway. I still think there's a ton of good functionality in jams that almost nobody knows about or uses, and it wouldn't take too much work to polish it up.

Fallout

If we do end up going this route, we end up with jams (and hence mir_eval) as dependencies in our stack. I don't see this as a bad thing necessarily - these packages are both lightweight. It does imply stepping up the maintenance on them individually though, which I'd be happy to do.

If we do end up with mir_eval as a core dependency, this in turn adds some potential for aligning APIs and reducing redundant functionality (particularly for some things planned in #1929 around sonification, annotation processing, etc). As such, we might want to take that seriously and do some advanced planning around it going into the 1.1 milestone.

At this point, it's all just vaporware, though the bots have mocked up some of the scaffolding for me already. Right now I'm mainly soliciting feedback and ideas. Please chime in if you have thoughts about how this could work.