bert-extractive-summarizer

Bert Extractive Summarizer

This repo is the generalization of the lecture-summarizer repo. This tool utilizes the HuggingFace Pytorch transformers library to run extractive summarizations. This works by first embedding the sentences, then running a clustering algorithm, finding the sentences that are closest to the cluster's centroids. This library also uses coreference techniques, utilizing the https://github.com/huggingface/neuralcoref library to resolve words in summaries that need more context. The greedyness of the neuralcoref library can be tweaked in the CoreferenceHandler class.

As of the most recent version of bert-extractive-summarizer, by default, CUDA is used if a gpu is available.

Paper: https://arxiv.org/abs/1906.04165

Try the Online Demo:

Distill Bert Summarization Demo

Table of Contents

1. Install
2. Examples
   1. Simple Example
   2. SBert
   3. Retrieve Embeddings
   4. Use Coreference
   5. Custom Model Example
   6. Large Example
3. Calculating Elbow
4. Running the Service

Install

pip install bert-extractive-summarizer

Examples

Simple Example

from summarizer import Summarizer

body = 'Text body that you want to summarize with BERT'
body2 = 'Something else you want to summarize with BERT'
model = Summarizer()
model(body)
model(body2)

Specifying number of sentences

Number of sentences can be supplied as a ratio or an integer. Examples are provided below.

from summarizer import Summarizer
body = 'Text body that you want to summarize with BERT'
model = Summarizer()
result = model(body, ratio=0.2)  # Specified with ratio
result = model(body, num_sentences=3)  # Will return 3 sentences 

Using multiple hidden layers as the embedding output

You can also concat the summarizer embeddings for clustering. A simple example is below.

from summarizer import Summarizer
body = 'Text body that you want to summarize with BERT'
model = Summarizer('distilbert-base-uncased', hidden=[-1,-2], hidden_concat=True)
result = model(body, num_sentences=3)

Use SBert

One can use Sentence Bert with bert-extractive-summarizer with the newest version. It is based off the paper here: https://arxiv.org/abs/1908.10084, and the library here: https://www.sbert.net/. To get started, first install SBERT:

pip install -U sentence-transformers

Then a simple example is the following:

from summarizer.sbert import SBertSummarizer

body = 'Text body that you want to summarize with BERT'
model = SBertSummarizer('paraphrase-MiniLM-L6-v2')
result = model(body, num_sentences=3)

It is worth noting that all the features that you can do with the main Summarizer class, you can also do with SBert.

Retrieve Embeddings

You can also retrieve the embeddings of the summarization. Examples are below:

from summarizer import Summarizer
body = 'Text body that you want to summarize with BERT'
model = Summarizer()
result = model.run_embeddings(body, ratio=0.2)  # Specified with ratio. 
result = model.run_embeddings(body, num_sentences=3)  # Will return (3, N) embedding numpy matrix.
result = model.run_embeddings(body, num_sentences=3, aggregate='mean')  # Will return Mean aggregate over embeddings. 

Use Coreference

First ensure you have installed neuralcoref and spacy. It is worth noting that neuralcoref does not work with spacy > 0.2.1.

pip install spacy
pip install transformers # > 4.0.0
pip install neuralcoref

python -m spacy download en_core_web_md

Then to to use coreference, run the following:

from summarizer import Summarizer
from summarizer.text_processors.coreference_handler import CoreferenceHandler

handler = CoreferenceHandler(greedyness=.4)
# How coreference works:
# >>>handler.process('''My sister has a dog. She loves him.''', min_length=2)
# ['My sister has a dog.', 'My sister loves a dog.']

body = 'Text body that you want to summarize with BERT'
body2 = 'Something else you want to summarize with BERT'
model = Summarizer(sentence_handler=handler)
model(body)
model(body2)

Custom Model Example

…

Large Example

…

Calculating Elbow

As of bert-extractive-summarizer version 0.7.1, you can also calculate ELBOW to determine the optimal cluster. Below shows a sample example in how to retrieve the list of inertias.

from summarizer import Summarizer

body = 'Your Text here.'
model = Summarizer()
res = model.calculate_elbow(body, k_max=10)
print(res)

You can also find the optimal number of sentences with elbow using the following algorithm.

from summarizer import Summarizer

body = 'Your Text here.'
model = Summarizer()
res = model.calculate_optimal_k(body, k_max=10)
print(res)

Summarizer Options

…

Running the Service

There is a provided flask service and corresponding Dockerfile. Running the service is simple, and can be done though the Makefile with the two commands:

make docker-service-build
make docker-service-run

This will use the Bert-base-uncased model, which has a small representation. The docker run also accepts a variety of arguments for custom and different models. This can be done through a command such as:

docker build -t summary-service -f Dockerfile.service ./
docker run --rm -it -p 5000:5000 summary-service:latest -model bert-large-uncased

Other arguments can also be passed to the server. Below includes the list of available arguments.

• -greediness: Float parameter that determines how greedy nueralcoref should be
• -reduce: Determines the reduction statistic of the encoding layer (mean, median, max).
• -hidden: Determines the hidden layer to use for embeddings (default is -2)
• -port: Determines the port to use.
• -host: Determines the host to use.

Once the service is running, you can make a summarization command at the http://localhost:5000/summarize endpoint. This endpoint accepts a text/plain input which represents the text that you want to summarize. Parameters can also be passed as request arguments. The accepted arguments are:

• ratio: Ratio of sentences to summarize to from the original body. (default to 0.2)
• min_length: The minimum length to accept as a sentence. (default to 25)
• max_length: The maximum length to accept as a sentence. (default to 500)

An example of a request is the following:

…