prose is a natural language processing library for Go — pure Go, no cgo, no
external services.
- Tokenization — words and punctuation, with byte offsets, in any language
- Segmentation — sentences, via the Punkt algorithm, in 12 languages
- Tagging — part of speech, in 12 languages
- Entities — people, places and organizations (English)
- Readability — Flesch–Kincaid, Gunning fog, SMOG, Coleman–Liau and more
- Case conversion — title and sentence case, AP and Chicago styles
go get github.com/jdkato/prose/v3package main
import (
"fmt"
"log"
"github.com/jdkato/prose/v3"
)
func main() {
doc, err := prose.NewDocument("Marie Curie studied in Paris. She won the Nobel Prize twice.")
if err != nil {
log.Fatal(err)
}
for _, sent := range doc.Sentences() {
fmt.Println(sent.Text)
}
for _, tok := range doc.Tokens() {
fmt.Println(tok.Text, tok.Tag)
}
for _, ent := range doc.Entities() {
fmt.Println(ent.Text, ent.Label)
// Marie Curie PERSON
// Paris GPE
}
}Every token, sentence and entity carries a byte offset into the source, and the text at that offset is exactly the text you were given:
src[tok.Start:tok.End()] == tok.TextThat holds for multi-byte input, repeated words and irregular whitespace:
src := "He said “hi there” to me."
doc, _ := prose.NewDocument(src)
for _, tok := range doc.Tokens() {
fmt.Printf("%-8q [%d:%d] %q\n", tok.Text, tok.Start, tok.End(), tok.In(src))
}It is also why prose never normalizes text before tokenizing. Replacing curly
quotes with straight ones, or ’ with ', changes byte lengths and
silently shifts every offset that follows. Normalization that would move bytes
is the caller's decision, made before prose sees the text.
English is built in. Other languages are packages you import, so a binary carries only the languages it uses:
import (
"github.com/jdkato/prose/v3"
_ "github.com/jdkato/prose/v3/segment/lang/de"
_ "github.com/jdkato/prose/v3/tag/lang/de"
)
doc, err := prose.NewDocument("Dr. Müller fährt z.B. nach Berlin. Er bleibt dort.",
prose.WithLanguage("de"))
for _, tok := range doc.Tokens() {
fmt.Println(tok.Text, tok.Tag)
// Dr. NOUN
// Müller PROPN
// fährt VERB
// ...
}Each piece can be asked for a language on its own: tokenize.ForLanguage,
segment.ForLanguage and tag.ForLanguage. Import segment/lang/all and
tag/lang/all when the language is decided at run time.
| Languages | |
|---|---|
| Tagging | Danish, Dutch, English, Finnish, French, German, Italian, Norwegian, Portuguese, Spanish, Swedish, Turkish |
| Segmentation | The same twelve; punctuation rules for everything else, including Chinese, Japanese and Korean |
| Tokenization | Every language. French and Italian elisions (l'homme), French clitics (donne-moi), Spanish ¿¡, every quotation style, and script boundaries for Chinese and Japanese |
The English tagger uses the Penn Treebank tag set; every other model emits the
Universal Dependencies tag set, so
one rule reads across languages, and tag.ToUniversal maps Penn tags onto it.
The models, English included, are averaged perceptrons trained on the Universal
Dependencies treebanks; tag/lang/MODELS.md lists each
one's training data, license and accuracy, and cmd/train-tagger trains one
from any CoNLL-U treebank.
Entity extraction and readability are English only.
prose ships two trained models: a part-of-speech tagger (~3 MB) and an entity
recognizer (~4 MB). Each lives in its own package, so the Go linker leaves out
whatever you do not import.
Importing the root package links both. If you only need tags, import the tagger directly and the entity model never enters your binary — a 4.5 MB difference:
import "github.com/jdkato/prose/v3/tag"
tagger, err := tag.New()
if err != nil {
log.Fatal(err)
}
for _, tok := range tagger.Tag([]string{"Kubernetes", "orchestrates", "containers"}) {
fmt.Println(tok.Text, tok.Tag)
}Models load once per process and are shared, so building many documents does not repeat the work.
Each component stands alone if you want just that piece:
| Package | What it does |
|---|---|
prose |
Document, the whole pipeline |
tokenize |
words and punctuation |
segment |
sentences |
tag |
part-of-speech tags |
tag/lang |
tagging models, one package per language |
segment/lang |
segmentation models, one package per language |
ner |
named entities |
token |
the shared token type |
summarize |
readability metrics |
strcase |
title and sentence case |
Skipping entity extraction avoids most of the work:
doc, err := prose.NewDocument(text, prose.WithExtraction(false))WithTokenization, WithSegmentation, WithTagging and WithExtraction each
take a bool. Extraction requires tagging, and tagging requires tokenization, so
those are enabled automatically when needed.
Work scales with the length of the input — roughly 1.4 s for a million words —
so NewDocumentContext accepts a context.Context:
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
doc, err := prose.NewDocumentContext(ctx, veryLargeText)The individual tokenizers and taggers take no context: they work at microsecond scale, where there is nothing worth cancelling.
Tokenizer, Segmenter, Tagger and Extracter are all safe for concurrent
use. Their models are immutable once loaded, and per-call scratch space is
pooled.
import "github.com/jdkato/prose/v3/summarize"
d := summarize.NewDocument("The quick brown fox jumps over the lazy dog.")
fmt.Println(d.FleschKincaid(), d.GunningFog(), d.SMOG())import "github.com/jdkato/prose/v3/strcase"
tc := strcase.NewTitleConverter(strcase.APStyle)
fmt.Println(tc.Convert("the quick brown fox"))MIT. See LICENSE, which also records the upstream work prose
derives from.