百科.dev
全部条目在线工具AI 编程趋势榜开源项目技术资讯提交条目
登录
< 返回工具列表
C

coreferee

> 编程语言
开源

英文、法文、德文和波兰文的核心引用解析,针对有限的训练数据进行优化,并可轻松扩展到其他语言

199 stars0 点赞1 次浏览
访问官网GitHub

工具介绍

英文、法文、德文和波兰文的核心引用解析,针对有限的训练数据进行优化,并可轻松扩展到其他语言

Coreferee

Author: Richard Paul Hudson, Explosion AI

  • 1. Introduction
    • 1.1 The basic idea
    • 1.2 Getting started
      • 1.2.1 English
      • 1.2.2 French
      • 1.2.3 German
      • 1.2.4 Polish
    • 1.3 Background information
    • 1.4 Facts and figures
      • 1.4.1 Covered relevant linguistic features
      • 1.4.2 Model performance
  • 2. Interacting with the data model
  • 3. How it works
    • 3.1 General operation and rules
      • 3.1.1 Anaphor pair analysis
      • 3.1.2 Noun pair detection
      • 3.1.3 Building the chains
    • 3.2 The neural ensemble
  • 4. Adding support for a new language
  • 5. Adding support for a custom spaCy model
  • 6. Version history
    • 6.1 Version 1.0.0
    • 6.2 Version 1.0.1
    • 6.3 Version 1.1.0
    • 6.4 Version 1.1.1
    • 6.5 Version 1.1.2
    • 6.6 Version 1.1.3
    • 6.7 Version 1.2.0
  • 7. Open issues/requests for assistance

1. Introduction

1.1 The basic idea

Coreferences are situations where two or more words within a text refer to the same entity, e.g. John went home because he was tired. Resolving coreferences is an important general task within the natural language processing field.

Coreferee is a Python 3 library (tested with versions 3.6—3.10) that is used together with spaCy (tested with versions 3.0.0—3.3.0) to resolve coreferences within English, French, German and Polish texts. It is designed so that it is easy to add support for new languages. It uses a mixture of neural networks and programmed rules.

The library was originally developed at msg systems, but is now being maintained at Explosion AI. Please direct any new issues or discussions to the Explosion repository.

1.2 Getting started

1.2.1 English

Presuming you have already installed spaCy and one of the English spacy models, install Coreferee from the command line by typing:

python3 -m pip install coreferee
python3 -m coreferee install en

Note that:

  • the required command may be python rather than python3 on some operating systems;
  • in order to use the transformer-based spaCy model en_core_web_trf with Coreferee, you will need to install the spaCy model en_core_web_lg as well (see the explanation here).

Then open a Python prompt (type python3 or python at the command line):

…
1.2.2 French

Presuming you have already installed spaCy and one of the French spacy models, install Coreferee from the command line by typing:

python3 -m pip install coreferee
python3 -m coreferee install fr

Note that the required command may be python rather than python3 on some operating systems.

Then open a Python prompt (type python3 or python at the command line):

…
1.2.3 German

Presuming you have already installed spaCy and one of the German spacy models, install Coreferee from the command line by typing:

python3 -m pip install coreferee
python3 -m coreferee install de

Note that the required command may be python rather than python3 on some operating systems.

Then open a Python prompt (type python3 or python at the command line):

…
1.2.4 Polish

Presuming you have already installed spaCy and one of the Polish spacy models, install Coreferee from the command line by typing:

python3 -m pip install coreferee
python3 -m coreferee install pl

Note that the required command may be python rather than python3 on some operating systems.

Then open a Python prompt (type python3 or python at the command line):

…

1.3 Background information

Handling coreference resolution successfully requires training corpora that have been manually annotated with coreferences. The state of the art in coreference resolution is progressing rapidly, but is largely focussed on techniques that require training corpora that are larger than what is available for most languages and software developers. The CONLL 2012 training corpus, which is most widely used, has the following restrictions:

  • CONLL 2012 covers English, Chinese and Arabic; there is nothing of comparable size for most other languages. For example, the corpus we used to train Coreferee for German is around a tenth of the size of CONLL 2012;

  • CONLL 2012 is not publicly available and has a relatively restrictive license.

Earlier versions of spaCy had an extension, Neuralcoref, that was excellent but that was never made publicly available for any language other than English. The aim of Coreferee, on the other hand, is to get coreference resolution working for a variety of languages: our focus is less on necessarily achieving the best possible precision and recall for English than on enabling the functionality to be reproduced for new languages as easily and as quickly as possible. Because training data is in such short supply for most languages and is very effort-intensive to produce, it is important to use what is available as effectively as possible.

There are three essential strategies that human readers employ to recognise coreferences within a text:

  1. Hard grammatical rules that completely preclude entities within a text from coreferring, e.g. The house stood tall. They went on walking. Such rules play an especially important role in languages that have grammatical gender, which includes most continental European languages.

  2. Pragmatic tendencies, e.g. a word that begins a sentence and that is a grammatical subject is more likely than a word that is in the middle of a sentence and that forms part of a prepositional phrase to be referred back to by a pronoun that follows it in the next sentence.

  3. Semantic restrictions, i.e. which entities can realistically do what to which entities in the world being described. For example, in the sentence The child saddled her up, a reader's experience of the world will make it clear that her must refer to a horse.

With unlimited training data, it would be possible to train a system to employ all three strategies effectively from first principles using word vectors. The features of Coreferee that allow effective learning with the limited training data that is available are:

  • Strategy 1) is covered by hardcoded rules for each language that the system is then not required to learn from the training data. Because detailed knowledge of the grammar of a specific natural language is a separate skill set from knowledge of machine learning, the two concerns have been fully separated in Coreferee: rules are covered in a separate module from tendencies. This means that a model for a new language can be generated by a competent Python programmer with no knowledge of machine learning or neural networks;

  • Because the pragmatic tendencies for strategy 2) are very complex and only partially understood by linguists, machine learning and neural networks represent the only realistic way of tackling them. In order to reduce the amount of training data required for neural networks to learn effectively, the syntactic and morphological information supplied by the spaCy models, which have typically been trained with considerably more training data than will be available for coreference resolution, is used as input to neural networks alongside the standard word vectors.

  • Especially with limited training data but probably even with the largest available training datasets, it is unlikely that a system will learn more than the very simplest tendencies for strategy 3). However, making word vectors available to neural networks ensures that Coreferee can make use of whatever tendencies are discernable.

Coreferee started life to assist the Holmes project, which is used for information extraction and intelligent search. Coreferee is in no way dependent on Holmes, but this original aim has led to several design decisions that may seem somewhat atypical. Several of them could easily be altered by someone with a requirement to do so:

  • A mention within Coreferee does not consist of a span, but rather of a single token or of a list of tokens that stand in a coordination relationship to one another.

  • Coreferee does not capture coreferences that are unambiguously evident from the structure of a sentence. For example, the identity of he and doctor in the sentence He was a doctor is not reported by Coreferee because it can easily be derived from a simple analysis of the copular structure of the phrase.

  • Repetitions of first- and second-person pronouns (I was tired. I went home) are not captured as they add no value either for information extraction or for intelligent search.

  • Coreferee focusses heavily on anaphors (for English: pronouns). There is only relatively limited capture of coreference between noun phrases, and it is entirely rule-based. (In turn, however, this serves the aim of working with limited training data: noun-phrase coreference is a more exacting task than anaphor resolution.)

  • Because search performance is much more important for Holmes than document parsing performance, Coreferee performs all analysis eagerly as each document passes through the pipe.

1.4 Facts and figures

1.4.1 Covered relevant linguistic features

ISO 639-1LanguageAnaphor expressionAgreement classesCoordination expression

PronominalVerbalPrepositionalConjunctiveComitative

enEnglishMy friend came in. He was happy.--Three singular (natural genders) and one plural class.Peter and Mary-

deGermanMein Freund kam rein. Er war glücklich.-Ich benutzte das Auto und hatte damit einige Probleme.Three singular (grammatical genders) and one plural class.Peter und Maria-

frFrenchMon ami entra. Il était heureux.--Two singular (grammatical genders) and two plural (grammatical genders) classes.Pierre et Marie-

plPolishWszedł mój kolega. Widzieliście, jaki on był szczęśliwy?Wszedł mój kolega. Szczęśliwy był.1-2Three singular (grammatical genders) and two plural (natural genders) classes.Piotr i Kasia1) Piotr z Kasią przyjechali do Warszawy; 2) Widziałem Piotra i przyszli z Kasią

  1. Only subject zero anaphors are covered. Object zero anaphors, e.g. Wypiłeś wodę? Tak, wypiłem. are not in scope because they are mainly used colloquially and do not normally occur in the types of text for which Coreferee is primarily designed. Handling them would require creating or locating a detailed dictionary of verb valencies.

  2. Polish has a restricted use of anaphoric prepositions in some formal registers, e.g. Skończyło się to dlań smutno. Because the Polish spaCy models were trained on news texts, they do not recognise such prepositions, meaning that Coreferee cannot capture them either.

1.4.2 Model performance

ISO 639-1LanguageTraining corporaTotal words in training corpora*_trf models*_lg models*_md models*_sm models

Anaphors in 20%Accu

Issues· 2 开放

查看全部 Issues在 GitHub 打开

暂无开放 Issues,或尚未同步最近议题。

> 标签

Python

暂无评论,来聊聊你的看法吧

> 工具信息

发布日期2026年8月1日
最后更新2026年9月18日
分类编程语言
定价开源

> 相关工具

T
TypeScript
JavaScript 的超集,为前端与全栈提供静态类型
P
Python
通用编程语言,广泛用于 Web、数据与 AI
G
Go
Google 推出的简洁高效系统语言