[IDEA] Improve sentence segmentation with yasbd-lib
这个功能请求是用来解决什么问题的? / Is your feature request related to a problem? Please describe.
Open-LLM-VTuber uses PySBD for sentence boundary detection in sentence_divider.py.
PySBD is stable and works well, but its language support is limited to 23 languages. For a multilingual real-time voice interaction system, broader language coverage and more accurate sentence boundaries can improve TTS chunking and subtitle generation.
您期望的解决方案是什么? / Describe the solution you'd like
Consider using yasbd-lib as the sentence boundary detector.
Example:
from yasbd import BoundaryDetector, get_supported_langs
SUPPORTED_LANGUAGES = get_supported_langs()
segmenter = BoundaryDetector(lang="en")
sentences = list(segmenter.segment(text))yasbd-lib provides:
- Support for 39 languages
- Improved sentence boundary detection accuracy
- Faster segmentation performance
- Streaming support
- Native spaCy integration
A PySBD-compatible adapter is also available if keeping the current API style is preferred.
此功能为何对 Open-LLM-VTuber 很重要? / Why is this important for Open-LLM-VTuber?
Sentence segmentation directly affects real-time voice interaction, subtitle timing, and TTS chunk generation.
Improving the segmentation engine can provide better multilingual support and more reliable sentence splitting for users across different languages.
您考虑过哪些替代方案? / Describe alternatives you've considered
- Continue using PySBD with the current implementation.
- Maintain the current sentence segmentation logic.
您是否愿意参与开发此功能? / Would you like to work on this issue?
Yes
补充信息 / Additional context
Comparison: https://dev.to/speed_k_7e1b449706e59e433/yasbd-lib-vs-pysbd-two-philosophies-of-sentence-boundary-detection-i88 Benchmarks: https://github.com/speedyk-005/yasbd-lib/tree/main/benchmarks
I would be happy to open a PR implementing this change if the maintainers are interested.
Source: Open-LLM-VTuber/Open-LLM-VTuber