Baike.dev
All toolsOnline toolsAI codingTrendingOpen sourceNewsSubmit
Log in
< Back to tools
P

pragmatic_segmenter

> 编程语言
Open source

Pragmatic Segmenter is a rule-based sentence boundary detection gem that works out-of-the-box across many languages.

594 stars0 likes3 views
WebsiteGitHub

About

Pragmatic Segmenter is a rule-based sentence boundary detection gem that works out-of-the-box across many languages.

Pragmatic Segmenter

Pragmatic Segmenter is a rule-based sentence boundary detection gem that works out-of-the-box across many languages.

Install

Ruby Supports Ruby 2.1.5 and above

gem install pragmatic_segmenter

Ruby on Rails Add this line to your application’s Gemfile:

ruby
gem 'pragmatic_segmenter'

Usage

  • If no language is specified, the library will default to English.
  • To specify a language use its two character ISO 639-1 code.
…

Live Demo

Try out a live demo of Pragmatic Segmenter in the browser.

Background

According to Wikipedia, sentence boundary disambiguation (aka sentence boundary detection, sentence segmentation) is defined as:

Sentence boundary disambiguation (SBD), also known as sentence breaking, is the problem in natural language processing of deciding where sentences begin and end. Often natural language processing tools require their input to be divided into sentences for a number of reasons. However sentence boundary identification is challenging because punctuation marks are often ambiguous. For example, a period may denote an abbreviation, decimal point, an ellipsis, or an email address – not the end of a sentence. About 47% of the periods in the Wall Street Journal corpus denote abbreviations. As well, question marks and exclamation marks may appear in embedded quotations, emoticons, computer code, and slang. Languages like Japanese and Chinese have unambiguous sentence-ending markers.

The goal of Pragmatic Segmenter is to provide a "real-world" segmenter that works out of the box across many languages and does a reasonable job when the format and domain of the input text are unknown. Pragmatic Segmenter does not use any machine-learning techniques and thus does not require training data.

Pragmatic Segmenter aims to improve on other segmentation engines in 2 main areas:

  1. Language support (most segmentation tools only focus on English)
  2. Text cleaning and preprocessing

Pragmatic Segmenter is opinionated and made for the explicit purpose of segmenting texts to create translation memories. Therefore, things such as parenthesis within a sentence are kept as one segment, even if technically there are two or more sentences within the segment in order to maintain coherence. The algorithm is also conservative in that if it comes across an ambiguous sentence boundary it will ignore it rather than splitting.

What do you mean by opinionated?

Pragmatic Segmenter is specifically used for the purpose of segmenting texts for use in translation (and translation memory) related applications. Therefore Pragmatic Segmenter takes a stance on some formatting and segmentation gray areas with the goal of improving the segmentation for the above stated purpose. Some examples:

  • Removes 'table of contents' style long string of periods ('............')
  • Keeps parentheticals, quotations, and parentheticals or quotations within a sentence as one segment for clarity even though technically there may be multiple grammatical sentences within the segment
  • Strips out any xhtml code
  • Conservative in cases where the sentence boundary is ambigious and Pragmatic Segmenter does not have a built in rule

There is an option to turn off text cleaning and preprocessing if you so choose.

The Golden Rules

The Golden Rules are a set of tests I developed that can be run through a segmenter to check its accuracy in regards to edge case scenarios. Most of the papers cited below in Segmentation Papers and Books either use the WSJ corpus or Brown corpus from the Penn Treebank to test their segmentation algorithm. In my opinion there are 2 limits to using these corpora:

  1. The corpora may be too expensive for some people ($1,700).
  2. The majority of the sentences in the corpora are sentences that end with a regular word followed by a period, thus testing the same thing over and over again.

In the Brown Corpus 92% of potential sentence boundaries come after a regular word. The WSJ Corpus is richer with abbreviations and only 83% [53% according to Gale and Church, 1991] of sentences end with a regular word followed by a period.

Andrei Mikheev - Periods, Capitalized Words, etc.

Therefore, I created a set of distinct edge cases to compare segmentation tools on. As most segmentation tools have very high accuracy, in my opinion what is really important to test is how a segmenter handles the edge cases - not whether it can segment 20,000 sentences that end with a regular word followed by a period. These example tests I have named the “Golden Rules". This list is by no means complete and will evolve and expand over time. If you would like to contribute to (or complain about) the test set, please open an issue.

The Holy Grail of sentence segmentation appears to be Golden Rule #18 as no segmenter I tested was able to correctly segment that text. The difficulty being that an abbreviation (in this case a.m./A.M./p.m./P.M.) followed by a capitalized abbreviation (such as Mr., Mrs., etc.) or followed by a proper noun such as a name can be both a sentence boundary and a non sentence boundary.

Download the Golden Rules: [txt | Ruby RSpec]

Golden Rules (English)

1.) Simple period to end sentence

Hello World. My name is Jonas.
=> ["Hello World.", "My name is Jonas."]

2.) Question mark to end sentence

What is your name? My name is Jonas.
=> ["What is your name?", "My name is Jonas."]

3.) Exclamation point to end sentence

There it is! I found it.
=> ["There it is!", "I found it."]

4.) One letter upper case abbreviations

My name is Jonas E. Smith.
=> ["My name is Jonas E. Smith."]

5.) One letter lower case abbreviations

Please turn to p. 55.
=> ["Please turn to p. 55."]

6.) Two letter lower case abbreviations in the middle of a sentence

Were Jane and co. at the party?
=> ["Were Jane and co. at the party?"]

7.) Two letter upper case abbreviations in the middle of a sentence

They closed the deal with Pitt, Briggs & Co. at noon.
=> ["They closed the deal with Pitt, Briggs & Co. at noon."]

8.) Two letter lower case abbreviations at the end of a sentence

Let's ask Jane and co. They should know.
=> ["Let's ask Jane and co.", "They should know."]

9.) Two letter upper case abbreviations at the end of a sentence

They closed the deal with Pitt, Briggs & Co. It closed yesterday.
=> ["They closed the deal with Pitt, Briggs & Co.", "It closed yesterday."]

10.) Two letter (prepositive) abbreviations

I can see Mt. Fuji from here.
=> ["I can see Mt. Fuji from here."]

11.) Two letter (prepositive & postpositive) abbreviations

St. Michael's Church is on 5th st. near the light.
=> ["St. Michael's Church is on 5th st. near the light."]

12.) Possesive two letter abbreviations

That is JFK Jr.'s book.
=> ["That is JFK Jr.'s book."]

13.) Multi-period abbreviations in the middle of a sentence

I visited the U.S.A. last year.
=> ["I visited the U.S.A. last year."]

14.) Multi-period abbreviations at the end of a sentence

I live in the E.U. How about you?
=> ["I live in the E.U.", "How about you?"]

15.) U.S. as sentence boundary

I live in the U.S. How about you?
=> ["I live in the U.S.", "How about you?"]

16.) U.S. as non sentence boundary with next word capitalized

I work for the U.S. Government in Virginia.
=> ["I work for the U.S. Government in Virginia."]

17.) U.S. as non sentence boundary

I have lived in the U.S. for 20 years.
=> ["I have lived in the U.S. for 20 years."]

18.) A.M. / P.M. as non sentence boundary and sentence boundary

At 5 a.m. Mr. Smith went to the bank. He left the bank at 6 P.M. Mr. Smith then went to the store.
=> ["At 5 a.m. Mr. Smith went to the bank.", "He left the bank at 6 P.M.", "Mr. Smith then went to the store."]

19.) Number as non sentence boundary

She has $100.00 in her bag.
=> ["She has $100.00 in her bag."]

20.) Number as sentence boundary

She has $100.00. It is in her bag.
=> ["She has $100.00.", "It is in her bag."]

21.) Parenthetical inside sentence

He teaches science (He previously worked for 5 years as an engineer.) at the local University.
=> ["He teaches science (He previously worked for 5 years as an engineer.) at the local University."]

22.) Email addresses

Her email is [email protected]. I sent her an email.
=> ["Her email is [email protected].", "I sent her an email."]

23.) Web addresses

The site is: https://www.example.50.com/new-site/awesome_content.html. Please check it out.
=> ["The site is: https://www.example.50.com/new-site/awesome_content.html.", "Please check it out."]

24.) Single quotations inside sentence

She turned to him, 'This is great.' she said.
=> ["She turned to him, 'This is great.' she said."]

25.) Double quotations inside sentence

She turned to him, "This is great." she said.
=> ["She turned to him, \"This is great.\" she said."]

26.) Double quotations at the end of a sentence

She turned to him, \"This is great.\" She held the book out to show him.
=> ["She turned to him, \"This is great.\"", "She held the book out to show him."]

27.) Double punctuation (exclamation point)

Hello!! Long time no see.
=> ["Hello!!", "Long time no see."]

28.) Double punctuation (question mark)

Hello?? Who is there?
=> ["Hello??", "Who is there?"]

29.) Double punctuation (exclamation point / question mark)

Hello!? Is that you?
=> ["Hello!?", "Is that you?"]

30.) Double punctuation (question mark / exclamation point)

Hello?! Is that you?
=> ["Hello?!", "Is that you?"]

31.) List (period followed by parens and no period to end item)

1.) The first item 2.) The second item
=> ["1.) The first item", "2.) The second item"]

32.) List (period followed by parens and period to end item)

1.) The first item. 2.) The second item.
=> ["1.) The first item.", "2.) The second item."]

33.) List (parens and no period to end item)

1) The first item 2) The second item
=> ["1) The first item", "2) The second item"]

34.) List (parens and period to end item)

1) The first item. 2) The second item.
=> ["1) The first item.", "2) The second item."]

35.) List (period to mark list and no period to end item)

1. The first item 2. The second item
=> ["1. The first item", "2. The second item"]

36.) List (period to mark list and period to end item)

1. The first item. 2. The second item.
=> ["1. The first item.", "2. The second item."]

37.) List with bullet

• 9. The first item • 10. The second item
=> ["• 9. The first item", "• 10. The second item"]

38.) List with hypthen

⁃9. The first item ⁃10. The second item
=> ["⁃9. The first item", "⁃10. The second item"]

39.) Alphabetical list

a. The first item b. The second item c. The third list item
=> ["a. The first item", "b. The second item", "c. The third list item"]

40.) Errant newline in the middle of a sentence (PDF)

This is a sentence\ncut off in the middle because pdf.
=> ["This is a sentence\ncut off in the middle because pdf."]

41.) Errant newline in the middle of a sentence

It was a cold \nnight in the city.
=> ["It was a cold night in the city."]

42.) **Lower case list separated by ne

Issues· 0 open

View all issuesOpen on GitHub

No open issues yet, or sync has not completed.

> Tags

Ruby

No comments yet. Be the first to share.

> Details

PublishedAug 1, 2026
UpdatedSep 17, 2026
Category编程语言
PricingOpen source

> Related tools

T
TypeScript
JavaScript 的超集,为前端与全栈提供静态类型
P
Python
通用编程语言,广泛用于 Web、数据与 AI
G
Go
Google 推出的简洁高效系统语言