wtpsplit

wtpsplit

Segment any Text - Robustly, Efficiently, Adaptably⚡

This repository allows you to segment text into sentences or other semantic units. It implements the models from:

• SaT — Segment Any Text: A Universal Approach for Robust, Efficient and Adaptable Sentence Segmentation by Markus Frohmann, Igor Sterner, Benjamin Minixhofer, Ivan Vulić and Markus Schedl (state-of-the-art, encouraged).
• WtP — Where’s the Point? Self-Supervised Multilingual Punctuation-Agnostic Sentence Segmentation by Benjamin Minixhofer, Jonas Pfeiffer and Ivan Vulić (previous version, maintained for reproducibility).

The namesake WtP is maintained for consistency. Our new followup SaT provides robust, efficient and adaptable sentence segmentation across 85 languages at higher performance and less compute cost. Check out the state-of-the-art results in 8 distinct corpora and 85 languages demonstrated in our Segment any Text paper.

Installation

pip install wtpsplit

Or one of the following for ONNX support:

pip install wtpsplit[onnx-gpu]
pip install wtpsplit[onnx-cpu]

Usage

…

ONNX Support

You can now enable even faster ONNX inference for sat and sat-sm models!

sat = SaT("sat-3l-sm", ort_providers=["CUDAExecutionProvider", "CPUExecutionProvider"])

…

If you wish to use LoRA in combination with an ONNX model:

• Run scripts/export_to_onnx_sat.py with use_lora: True and an appropriate output_dir: .
  • If you have a local LoRA module, use lora_path.
  • If you wish to load a LoRA module from the HuggingFace hub, use style_or_domain and language.
• Load the ONNX model with merged LoRA weights: sat = SaT(, onnx_providers=["CUDAExecutionProvider", "CPUExecutionProvider"])

Available Models

If you need a general sentence segmentation model, use -sm models (e.g., sat-3l-sm) For speed-sensitive applications, we recommend 3-layer models (sat-3l and sat-3l-sm). They provide a great tradeoff between speed and performance. The best models are our 12-layer models: sat-12l and sat-12l-sm.

The scores are macro-average F1 score across all available datasets for "English", and macro-average F1 score across all datasets and languages for "Multilingual". "adapted" means adapation via LoRA; check out the paper for details.

For comparison, here the English scores of some other tools:

Note that this library also supports previous WtP models. You can use them in essentially the same way as SaTmodels:

from wtpsplit import WtP

wtp = WtP("wtp-bert-mini")
# similar functionality as for SaT models
wtp.split("This is a test This is another test.")

For more details on WtP and reproduction details, see the WtP doc.

Paragraph Segmentation

Since SaT are trained to predict newline probablity, they can segment text into paragraphs in addition to sentences.

# returns a list of paragraphs, each containing a list of sentences
# adjust the paragraph threshold via the `paragraph_threshold` argument.
sat.split(text, do_paragraph_segmentation=True)

(NEW! v2.2+) Length-Constrained Segmentation

Control segment lengths with min_length and max_length parameters. This is useful when you need segments within specific size limits (e.g., for embedding models, storage, or downstream processing).

Basic Usage

…

Priors for Length Preference

Use priors to influence segment length distribution. Available priors:

# Gaussian prior (recommended): prefer segments around target_length
sat.split(text, max_length=100, prior_type="gaussian", 
          prior_kwargs={"target_length": 50, "spread": 10})

# Log-normal prior: right-skewed (more tolerant of longer segments)
sat.split(text, max_length=100, prior_type="lognormal", 
          prior_kwargs={"target_length": 70, "spread": 25})

# Clipped polynomial: hard cutoff at ±spread from target
sat.split(text, max_length=100, prior_type="clipped_polynomial", 
          prior_kwargs={"target_length": 60, "spread": 25})

Language-Aware Defaults

Pass lang_code to use language-specific defaults for target_length and spread (based on language-specific corpus statistics):

# German has longer average sentences → auto-uses target_length=90, spread=35
sat.split(text, max_length=150, prior_type="gaussian", 
          prior_kwargs={"lang_code": "de"})

# Chinese has shorter sentences → auto-uses target_length=45, spread=15
sat.split(text, max_length=100, prior_type="gaussian", 
          prior_kwargs={"lang_code": "zh"})

When using LoRA with a language, this happens automatically:

sat = SaT("sat-3l", style_or_domain="ud", language="de")
sat.split(text, max_length=150, prior_type="gaussian")  # auto-uses German defaults

How It Works

The Viterbi algorithm finds globally optimal segmentation points that balance:

• The model's sentence boundary predictions (where natural splits occur)
• Your length preferences (via the prior; if provided)

Text Reconstruction:

# With constraints (max_length or min_length):
original_text = "".join(segments)  # segments may contain newlines

# Without constraints (SaT default with split_on_input_newlines=True):
original_text = "\n".join(segments)

Note: When using length constraints, segments may contain newlines. If you want to remove them, you can just post-process the output.

Note: When max_length is set, the threshold parameter is ignored. The Viterbi/greedy algorithms use raw model probabilities directly instead of threshold-based filtering.

For more details, see the Length Constraints Documentation.

Adaptation

SaT can be domain- and style-adapted via LoRA. We provide trained LoRA modules for Universal Dependencies, OPUS100, Ersatz, and TED (i.e., ASR-style transcribed speeches) sentence styles in 81 languages for sat-3land sat-12l. Additionally, we provide LoRA modules for legal documents (laws and judgements) in 6 languages, code-switching in 4 language pairs, and tweets in 3 languages. For details, we refer to our paper.

We also provided verse segmentation modules for 16 genres for sat-12-no-limited-lookahead.

Load LoRA modules like this:

# requires both lang_code and style_or_domain
# for available ones, check the /loras folder
sat_lora = SaT("sat-3l", style_or_domain="ud", language="en")
sat_lora.split("Hello this is a test But this is different now Now the next one starts looool")
# now for a highly distinct domain
sat_lora_distinct = SaT("sat-12l", style_or_domain="code-switching", language="es-en")
sat_lora_distinct.split("in the morning over there cada vez que yo decía algo él me decía algo")

You can also freely adapt the segmentation threshold, with a higher threshold leading to more conservative segmentation:

sat.split("This is a test This is another test.", threshold=0.4)
# works similarly for lora; but thresholds are higher
sat_lora.split("Hello this is a test But this is different now Now the next one starts looool", threshold=0.7)

Advanced Usage

Get the newline or sentence boundary probabilities for a text:

# returns newline probabilities (supports batching!)
sat.predict_proba(text)

Load a SaT model in HuggingFace transformers:

# import library to register the custom models 
import wtpsplit.models
from transformers import AutoModelForTokenClassification

model = AutoModelForTokenClassification.from_pretrained("segment-any-text/sat-3l-sm") # or some other model name; see https://huggingface.co/segment-any-text

Adapt to your own corpus via LoRA

Our models can be efficiently adapted via LoRA in a powerful way. Only 10-100 training segmented training sentences should already improve performance considerably. To do so:

Clone the repository and install requirements:

git clone https://github.com/segment-any-text/wtpsplit
cd wtpsplit
pip install -r requirements.txt
pip install adapters==0.2.1 --no-dependencies
cd ..

1. Create data in this format:

import torch

torch.save(
    {
        "language_code": {
            "sentence": {
                "dummy-dataset": {
                    "meta": {
                        "train_data": ["train sentence 1", "train sentence 2"],
                    },
                    "data": [
                        "test sentence 1",
                        "test sentence 2",
                    ]
                }
            }
        }
    },
    "dummy-dataset.pth"
)

Note that there should not be any newlines within individual sentences! This now raises an error. Instead, each entry of a list should be a sentence, and there should be no "\n" characters. So your corpus should already be well-split.

1. Create/adapt config; provide base model via model_name_or_path and training data .pth via text_path:

configs/lora/lora_dummy_config.json

We recommend starting using this config, and adapting model_name_or_path, output_dir, and text_path if needed. You may also wish to adapt other aspects such as adapter_config and b