Audio encoder development plan

Evaluation Recipes

• Machine sounds anomaly detection @shimhz [to be done on top of espnet/espnet/pull/5998]
• Sound event detection @Shikhar-S from BEAN [26th Jan]
  • DCASE dataset @Shikhar-S [26th Jan]
    • Rfcx @popcornell
    • Gibbons @popcornell
    • Hiceas @popcornell
    • Enabirds @popcornell
• Audio Captioning
  • AudioCaps [similar to this recipe] @shimhz
• Audio Entailment @Shikhar-S [30th Jan]
  • Audio Question Answering
• FSD-50K Multi-label classification @juice500ml
• Music @fukafilet
  • Instrument classification
  • Genre classisifcation

BEATs pre-training

• Code for training @Shikhar-S [24th Jan]
  • AudioSet @Shikhar-S
  • AudioSet + inat @Shikhar-S

Pull Requests

• https://github.com/espnet/espnet/pull/5998 Reviewers: @ftshijt @shimhz
  • https://github.com/espnet/espnet/pull/6028 Reviewer: @popcornell
  • https://github.com/espnet/espnet/pull/6004 Reviewer: @popcornell
  • https://github.com/espnet/espnet/pull/6006 [WIP]

Timeline

• 24th Jan: Provide pre-training code to @juice500ml
• 26th Jan: Provide sed task recipe to @popcornell

Any suggestions are welcome.