Development plan for ESPnet-3

We want to develop the ESPnet-3, more pythonic/scalable framework with pytorch lightning and espnet recipes. We also aim to utilize multiple datasources, such as Huggingface Dataset, Lhotse, and pytorch dataset.

Based on ESPnet-EZ, we plan to shift from EZ to 3. Welcome your requests for these particular directions and contributions! The followings are our development plans.

New Features

• Pytorch-lightning trainer and model (based on yifan's implementation) (@Masao-Someki, #6172)
• Utility to load configuration (@Masao-Someki, #6171)
• Support multiple optimizer/scheduler (@Masao-Someki, #6173 )
• Parallel processing with Dask (@Masao-Someki, #6175, #6242 )
• Data organizer (easily scale to multiple dataset) (@Masao-Someki, #6167)
• Inference runner/score runner (@Masao-Someki, #6178)
• collect stats (@Masao-Someki, #6175, #6242 )
• Callback support (separated from trainer/model PR) (@Masao-Someki, #6180, #6249 )
• Logging utils (@Masao-Someki, #6329)
• Publication-related stages (@Masao-Someki, #6338)
• Demo-related stages (@Masao-Someki, #6342, #6434 )

System related

• Base and ASR system (@Masao-Someki, #6328)
• Integration test for ASR (@Masao-Someki, #6392)
• Integration test meta PR (@Masao-Someki, #6331)

Recipes

Task	Dataset	Owner	PR
asr1	Aishell	@Masao-Someki	-
asr1	LibriSpeech	@Masao-Someki	-
asr1	LibriSpeech-100h	@Masao-Someki	#6271
asr1	FalAR (Portugal ASR)	@Masao-Someki	-
asr2	InterSpeech 2024 challenge	@Masao-Someki	-
tts1	LJSpeech (VITs)	@Masao-Someki	-
tts1	Emilia (VITs/Valle)	-	-
diar1	-	@popcornell	-
enh1	-	-	-
s2st1	CVSS-C	-	-
s2t1	OWSM	@Masao-Someki	-
spk1	VoxCeleb1	-	-
st1	Europarl	@Masao-Someki	-
tse1	-	-	-
tts1	LibriTTS	-	-
codec1	LibriTTS	-	-
ssl1	-	-	-

Documentation

I will create a PR for the document after all the PRs on systems/components has merged. You can check the current documentations here: https://masao-someki.github.io/espnet_draft_home_page/espnet3/

• Get Started (@Masao-Someki )
• Installation (@Fhrozen, @Masao-Someki )
• callbacks (@Masao-Someki )
• collect_stats (@Masao-Someki )
• config (@Masao-Someki )
• contribution (@Masao-Someki )
• data preparation (@Masao-Someki )
• batching (@Masao-Someki )
• multi-gpu/multi-node training (@Masao-Someki )
• tensorboard/wandb (@Masao-Someki )

Others

• Fix installation (#6135 -> Closed due to its size)

• Cleanup CI (#6135 -> Closed due to its size)

• PR only for flake8 (including formatting) (@Fhrozen, #6140)

• PR for installation (@Masao-Someki, #6139)

  • formatting (@Masao-Someki, #6164 )
  • include all files as a format target (@Masao-Someki #6165)
  • Add documents on pyproject.toml and setup.py (@Masao-Someki, #6139)

• PR for Pytorch>2.6 support (@Masao-Someki, Done by merging master )

• PR for homepage (@juice500ml #6143)

• DevContainer for ESPent-3 (@Masao-Someki, #6141)

• Activate flake8 for espnet-3

• Fix feature for skipping optimization when gradient is invalid (https://github.com/espnet/espnet/pull/6172#discussion_r2260000590)

• Leaderboard

• CI for notebooks

• Speechlm PR (#6050)

  • reviewer: @Masao-Someki, @ftshijt, @Fhrozen

• Update attention.py, using SDPA by default (@popcornell #6149)

• Add installation script for ESPnet-3 (I believe it would be just installing python packages) (@Masao-Someki )

• Refactor dependencies (@Masao-Someki )

Bugfixes

• Merge espnet3 branch into master (@Masao-Someki, #6304)
• Directory update to espnet-3 (@Masao-Someki, #6327)
• Bugfix for recipe (@Masao-Someki, #6270)
• Bugfix on inference and multiple optimizer (@Masao-Someki, #6391)
• Add TEMPLATE and bugfix for integration test (@Masao-Someki, #6390)
• Enhanced unit tests (@Masao-Someki, #6428)
• Use config file names as default experiment/inference dirs (@Masao-Someki, #6416)
• Bug fix for data organizer and dataloader (@Masao-Someki, #6350)
• Fix pip version (pip version>25.3 cause CI issue only in espnet3 branch.. maybe this is related to the shift from setup.py to pyproject.toml)