Skip to content
jdkatoPublic

About

📖 A Golang library for text processing, including tokenization, part-of-speech tagging, and named-entity extraction.

Topics

Resources

Stars

3.1k stars

Watchers

2 watching

Forks

Repository files navigation

prose

CI Go Reference Coverage Status Go Report Card

prose is a natural language processing library for Go — pure Go, no cgo, no external services.

  • Tokenization — words and punctuation, with byte offsets, in any language
  • Segmentation — sentences, via the Punkt algorithm, in 12 languages
  • Tagging — part of speech, in 12 languages
  • Entities — people, places and organizations (English)
  • Readability — Flesch–Kincaid, Gunning fog, SMOG, Coleman–Liau and more
  • Case conversion — title and sentence case, AP and Chicago styles

Install

go get github.com/jdkato/prose/v3

Quick start

package main

import (
    "fmt"
    "log"

    "github.com/jdkato/prose/v3"
)

func main() {
    doc, err := prose.NewDocument("Marie Curie studied in Paris. She won the Nobel Prize twice.")
    if err != nil {
        log.Fatal(err)
    }

    for _, sent := range doc.Sentences() {
        fmt.Println(sent.Text)
    }

    for _, tok := range doc.Tokens() {
        fmt.Println(tok.Text, tok.Tag)
    }

    for _, ent := range doc.Entities() {
        fmt.Println(ent.Text, ent.Label)
        // Marie Curie PERSON
        // Paris GPE
    }
}

Everything knows where it came from

Every token, sentence and entity carries a byte offset into the source, and the text at that offset is exactly the text you were given:

src[tok.Start:tok.End()] == tok.Text

That holds for multi-byte input, repeated words and irregular whitespace:

src := "He said “hi there” to me."
doc, _ := prose.NewDocument(src)

for _, tok := range doc.Tokens() {
    fmt.Printf("%-8q [%d:%d] %q\n", tok.Text, tok.Start, tok.End(), tok.In(src))
}

It is also why prose never normalizes text before tokenizing. Replacing curly quotes with straight ones, or ’ with ', changes byte lengths and silently shifts every offset that follows. Normalization that would move bytes is the caller's decision, made before prose sees the text.

Languages

English is built in. Other languages are packages you import, so a binary carries only the languages it uses:

import (
    "github.com/jdkato/prose/v3"

    _ "github.com/jdkato/prose/v3/segment/lang/de"
    _ "github.com/jdkato/prose/v3/tag/lang/de"
)

doc, err := prose.NewDocument("Dr. Müller fährt z.B. nach Berlin. Er bleibt dort.",
    prose.WithLanguage("de"))

for _, tok := range doc.Tokens() {
    fmt.Println(tok.Text, tok.Tag)
    // Dr. NOUN
    // Müller PROPN
    // fährt VERB
    // ...
}

Each piece can be asked for a language on its own: tokenize.ForLanguage, segment.ForLanguage and tag.ForLanguage. Import segment/lang/all and tag/lang/all when the language is decided at run time.

Languages
Tagging Danish, Dutch, English, Finnish, French, German, Italian, Norwegian, Portuguese, Spanish, Swedish, Turkish
Segmentation The same twelve; punctuation rules for everything else, including Chinese, Japanese and Korean
Tokenization Every language. French and Italian elisions (l'homme), French clitics (donne-moi), Spanish ¿¡, every quotation style, and script boundaries for Chinese and Japanese

The English tagger uses the Penn Treebank tag set; every other model emits the Universal Dependencies tag set, so one rule reads across languages, and tag.ToUniversal maps Penn tags onto it. The models, English included, are averaged perceptrons trained on the Universal Dependencies treebanks; tag/lang/MODELS.md lists each one's training data, license and accuracy, and cmd/train-tagger trains one from any CoNLL-U treebank.

Entity extraction and readability are English only.

Pay only for what you use

prose ships two trained models: a part-of-speech tagger (~3 MB) and an entity recognizer (~4 MB). Each lives in its own package, so the Go linker leaves out whatever you do not import.

Importing the root package links both. If you only need tags, import the tagger directly and the entity model never enters your binary — a 4.5 MB difference:

import "github.com/jdkato/prose/v3/tag"

tagger, err := tag.New()
if err != nil {
    log.Fatal(err)
}

for _, tok := range tagger.Tag([]string{"Kubernetes", "orchestrates", "containers"}) {
    fmt.Println(tok.Text, tok.Tag)
}

Models load once per process and are shared, so building many documents does not repeat the work.

Packages

Each component stands alone if you want just that piece:

Package What it does
prose Document, the whole pipeline
tokenize words and punctuation
segment sentences
tag part-of-speech tags
tag/lang tagging models, one package per language
segment/lang segmentation models, one package per language
ner named entities
token the shared token type
summarize readability metrics
strcase title and sentence case

Choosing stages

Skipping entity extraction avoids most of the work:

doc, err := prose.NewDocument(text, prose.WithExtraction(false))

WithTokenization, WithSegmentation, WithTagging and WithExtraction each take a bool. Extraction requires tagging, and tagging requires tokenization, so those are enabled automatically when needed.

Cancellation

Work scales with the length of the input — roughly 1.4 s for a million words — so NewDocumentContext accepts a context.Context:

ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()

doc, err := prose.NewDocumentContext(ctx, veryLargeText)

The individual tokenizers and taggers take no context: they work at microsecond scale, where there is nothing worth cancelling.

Concurrency

Tokenizer, Segmenter, Tagger and Extracter are all safe for concurrent use. Their models are immutable once loaded, and per-call scratch space is pooled.

Readability and case conversion

import "github.com/jdkato/prose/v3/summarize"

d := summarize.NewDocument("The quick brown fox jumps over the lazy dog.")
fmt.Println(d.FleschKincaid(), d.GunningFog(), d.SMOG())
import "github.com/jdkato/prose/v3/strcase"

tc := strcase.NewTitleConverter(strcase.APStyle)
fmt.Println(tc.Convert("the quick brown fox"))

License

MIT. See LICENSE, which also records the upstream work prose derives from.

About

📖 A Golang library for text processing, including tokenization, part-of-speech tagging, and named-entity extraction.

Topics

Resources

Stars

3.1k stars

Watchers

2 watching

Forks

Releases

Packages

Used by

Contributors

Languages