A simple yet effective Audio-to-Midi Automatic Piano Transcription system
A simple yet effective Audio-to-Midi Automatic Piano Transcription system
This repo contains code for following papers for transcribing expressive piano performance into MIDI.
Yujia Yan and Zhiyao Duan, Scoring intervals using non-hierarchical transformer for automatic piano transcription, in Proc. International Society for Music Information Retrieval Conference (ISMIR), 2024, Paper
Yujia Yan, Frank Cwitkowitz, Zhiyao Duan, Skipping the Frame-Level: Event-Based Piano Transcription With Neural Semi-CRFs, Advances in Neural Information Processing Systems, 2021, OpenReview, paper, appendix
pip3 install transkunThe pip package provides a quick command for transcribing piano performance audio into midi:
$ transkun input.mp3 output.midwith cuda:
$ transkun input.mp3 output.mid --device cudaNote: The code/pip package shipped checkpoint is trained without pedal extension of notes, and with data augmentation,which I believe is closer to a real performance. Be cautious that the convention for the piano transcription task in previous works is extending all notes by sustain pedal durations. For more checkpoints, e.g, those reported in the paper, see Model Cards
Colab Notebook Colab
This system works as follows:
In V2, as demonstrated in the ISMIR 2024 paper, the changes from V1 are:
This code includes an neural semi-CRF module that is optimized for the problem domain.
Here is a minimal example for using this module:
……This script can also be used directly as the command line command 'transkun' if the pip package is installed, e.g.,
$ transkun input.mp3 output.mid| Dataset | Activation | Note Onset | Note Onset+Offset | Note Onset+Offset+ vel. | pedal activation | pedal onset | pedal onset+offset | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Checkpoint | prec | recall | F1 | prec | recall | F1 | prec | recall | F1 | prec | recall | F1 | prec | recall | F1 | prec | recall | F1 | prec | recall | F1 | |
| Transkun V2 | Maestro V3 | 0.9576 | 0.9489 | 0.953 | 0.9956 | 0.9714 | 0.9832 | 0.9465 | 0.9238 | 0.9349 | 0.9411 | 0.9186 | 0.9296 | 0.9671 | 0.9453 | 0.9541 | 0.8909 | 0.8421 | 0.8642 | 0.8632 | 0.8165 | 0.8377 |
| MAPS (ad hoc align) | 0.887 | 0.8252 | 0.8535 | 0.8671 | 0.9048 | 0.8849 | 0.6325 | 0.6613 | 0.6461 | 0.4351 | 0.4551 | 0.4446 | 0.8498 | 0.8547 | 0.8449 | 0.6521 | 0.7182 | 0.6732 | 0.4903 | 0.5427 | 0.5088 | |
| SMD | 0.9203 | 0.9491 | 0.934 | 0.9816 | 0.9766 | 0.979 | 0.9013 | 0.8968 | 0.899 | 0.8255 | 0.8211 | 0.8232 | 0.9364 | 0.9507 | 0.942 | 0.8722 | 0.8101 | 0.8388 | 0.803 | 0.7471 | 0.773 | |
| Transkun V2 Aug | Maestro V3 | 0.9495 | 0.9522 | 0.9505 | 0.9971 | 0.9715 | 0.984 | 0.9437 | 0.9197 | 0.9314 | 0.9386 | 0.9149 | 0.9264 | 0.9546 | 0.9416 | 0.9454 | 0.8883 | 0.8116 | 0.8453 | 0.8497 | 0.7790 | 0.8102 |
| MAPS (ad hoc align) | 0.9446 | 0.8334 | 0.8843 | 0.9396 | 0.9056 | 0.9219 | 0.7105 | 0.6854 | 0.6975 | 0.5596 | 0.5401 | 0.5495 | 0.8893 | 0.8389 | 0.8583 | 0.7313 | 0.7529 | 0.7343 | 0.5499 | 0.5650 | 0.5532 | |
| SMD | 0.9389 | 0.9518 | 0.9448 | 0.997 | 0.9801 | 0.9884 | 0.9284 | 0.9128 | 0.9205 | 0.8974 | 0.8823 | 0.8897 | 0.9491 | 0.9428 | 0.9447 | 0.8788 | 0.8043 | 0.8383 | 0.8208 | 0.7526 | 0.7837 | |
| Transkun V2 No Ext | Maestro V3 No Ext | 0.8671 | 0.825 | 0.8441 | 0.9984 | 0.9691 | 0.9833 | 0.8271 | 0.8034 | 0.8149 | 0.823 | 0.7995 | 0.8109 | 0.9498 | 0.9518 | 0.9487 | 0.8872 | 0.8105 | 0.8444 | 0.8413 | 0.7723 | 0.8031 |
| MAPS (ad hoc align) No Ext | 0.9093 | 0.6383 | 0.7465 | 0.941 | 0.9044 | 0.922 | 0.5577 | 0.5369 | 0.5469 | 0.443 | 0.4266 | 0.4345 | 0.8753 | 0.8471 | 0.8543 | 0.7107 | 0.7525 | 0.721 | 0.5331 | 0.5612 | 0.5421 | |
| SMD No Ext | 0.8539 | 0.8533 | 0.8524 | 0.9982 | 0.9774 | 0.9876 | 0.7936 | 0.7778 | 0.7855 | 0.7666 | 0.7513 | 0.7588 | 0.9483 | 0.9453 | 0.9455 | 0.8812 | 0.8067 | 0.8408 | 0.8215 | 0.7539 | 0.7848 |
We assume the data contains only the same sampling rate 44100hz. Therefore for the maestro dataset it is necessary to perform sampling rate conversion to 44100hz for the last two years (2017 and 2018) .
Assuming all audio files have already been converted to the same sampling rate, we iterate the entire dataset to combine the groundtruth midi and metadata into a single file.
The following script will generate train.pt, val.pt and test.pt
python3 -m transkun.createDatasetMaestro -h
usage: createDatasetMaestro.py [-h] [--noPedalExtension] datasetPath metadataCSVPath outputPath
positional arguments:
datasetPath folder path to the maestro dataset
metadataCSVPath path to the metadata file of the maestro dataset (csv)
outputPath path to the output folder
optional arguments:
-h, --help show this help message and exit
--noPedalExtension Do not perform pedal extension according to the sustain pedalThis command will generate train.pt, dev.pt, test.pt in the outputPath.
After generating the metadata files, we can perform training using the dataset. During training, the audio waveforms will be fetched directly from the original .wav files.
Firstly, generate a config template file for the model.:
mkdir checkpoint
python3 -m moduleconf.generate Model:transkun.ModelTransformer > checkpoint/conf.jsonThen we call the training script.
python3 -m transkun.train -hWe also provide an out-of-box tool for computing metrics directly from output midi files.
…Currently, we do not support evaluation of multitrack MIDIs.
This command can also be used directly as the command line script 'transkunEval' if the pip package is installed.
Use the following script to plot the ECDF curve for onset/offset deviations:
…If you find this repository helpful, please consider citing:
Bibtex:
@inproceedings{
yan2021skipping,
title={Skipping the Frame-Level: Event-Based Piano Transcription With Neural Semi-{CRF}s},
author={Yujia Yan and Frank Cwitkowitz and Zhiyao Duan},
booktitle={Advances in Neural Information Processing Systems},
editor={A. Beygelzimer and Y. Dauphin and P. Liang and J. Wortman Vaughan},
year={2021},
url={https://openreview.net/forum?id=AzB2Pq7UFsA}
}@inproceedings{yan2024scoring,
author = {Yujia Yan and Zhiyao Duan},
title = {Scoring Time Intervals Using Non-Hierarchical Transformer for Automatic Piano Transcription},
booktitle = {Proc. International Society for Music Information Retrieval Conference (ISMIR)},
year = {2024},
}No open issues yet, or sync has not completed.