Zipf's Law Next Word Predictor

A cheap, single-purpose statistical text model: next-word suggestions and text completion for plain prose, built with n-grams and Zipf's law. No neural networks, no GPU, trains in seconds.

Overview

This project explores how Zipf's law - a fundamental principle in linguistics stating that the frequency of any word is inversely proportional to its rank in the frequency table - can be used for practical text prediction. While modern LLMs use sophisticated neural architectures, this experiment shows how far basic statistical patterns can be pushed with a few tweaks.

How the model scores words

The predictor combines three ideas:

1. N-gram model for capturing local context (configurable n, default trigrams)
2. Zipf blend: instead of trusting ranks alone, each candidate distribution blends real evidence with the Zipf prior: P(word) ∝ zipf_weight · (count / total) + (1 − zipf_weight) · (1 / rank^s)
   • zipf_weight = 1.0 → pure counts
   • zipf_weight = 0.0 → legacy rank-only Zipf behavior
3. Interpolated smoothing across context lengths: the final score is a weighted mix of the unigram, bigram, and trigram distributions (interp_weights, by default exponentially favoring longer contexts), so unseen contexts degrade gracefully instead of collapsing

Temperature sampling still works on top of the final distribution for controlling generation diversity.

Usage

python example.py   # guided demo: self-tuning + chef-vs-coder personalization

Python API

from ZipfNextWordPredictor import ZipfNextWordPredictor

# Train directly from texts...
predictor = ZipfNextWordPredictor(
    corpus_texts=[your_text],
    n_gram_size=3,
    zipf_exponent=1.0,
    zipf_weight=0.7,
)

# ...or from .txt files / folders
stats = predictor.train_from_files("./data/corpus")
predictor.save("model.pkl")

# Later, skip training entirely
predictor = ZipfNextWordPredictor.load("model.pkl")

# Get word predictions
predictions = predictor.predict_next_words("Once upon a")

# Complete a sentence
completion = predictor.predict_completion("The story begins", num_words=5)

# Measure quality on held-out text
results = predictor.evaluate(held_out_text, k=3)
# -> {"top_k_accuracy": ..., "perplexity": ..., "tokens": ...}

# Let the model find its own Zipf curve
report = predictor.tune(held_out_text)
print(report["best"])   # best zipf_weight + interpolation profile, by perplexity
predictor.save("model_tuned.pkl")

Self-tuning

The model sweeps the full blend space — 11 zipf_weight values (0.0 = pure ranks → 1.0 = pure counts) across 4 interpolation profiles (44 combinations) — scores each on held-out text, and applies the winner (lowest perplexity).

python cli.py tune --text heldout.txt -m model.pkl --save

Personalization: it learns from its user

The creative heart of this project: a user layer that learns from everything typed, without ever touching the base corpus model. Predictions blend both:

P_final = (1 − α) · P_corpus + α · P_user

where α grows with evidence (α = personalization · u / (u + halflife)), so a brand-new user gets safe predictions and a seasoned user gets ones shaped by their own vocabulary and habits. This is the same insight SwiftKey was founded on — the personalization is the product.

model.observe("The princess decided to write python code.")  # learn, anytime
model.predict_next_words("the princess decided to")           # adapts instantly
model.user_stats()    # {"words": ..., "vocab": ..., "effective_alpha": ...}
model.forget()        # privacy reset: erase everything learned
model.save_user_profile("user.pkl")   # portable personal data

CLI:

# Learn from a file into a user profile (accumulates across runs)
python cli.py observe --text myposts.txt -m model.pkl --profile user.pkl

# Suggestions with personalization
python cli.py suggest "the princess decided to" -m model.pkl --profile user.pkl

# Interactive REPL that learns as you type and saves the profile on exit
python cli.py chat -m model.pkl --profile user.pkl --alpha 0.8

CLI

# Train from a .txt file or a folder of .txt files
python cli.py train --corpus ./data --out model.pkl --n 3 --zipf-weight 0.7

# One-shot suggestions
python cli.py suggest "once upon a" -m model.pkl -k 5

# Interactive REPL (suggestions + completion as you type)
python cli.py chat -m model.pkl

# Evaluate on held-out text
python cli.py evaluate --text heldout.txt -m model.pkl -k 3

# Auto-tune the blend on held-out text (persist with --save)
python cli.py tune --text heldout.txt -m model.pkl --save

Significance

This implementation helps understand:

• The role of statistical patterns in language modeling
• How Zipf's law relates to modern LLM prediction patterns
• Baseline performance of statistical approaches vs neural models

Use evaluate() to compare settings — e.g. sweep zipf_weight from 0.0 (pure ranks) to 1.0 (pure counts) on held-out text to see the effect of the blend yourself.

Limitations

• No semantic understanding
• Limited to patterns present in training corpus
• Lacks long-range dependencies
• Simple statistical approach vs modern attention mechanisms

Experimental Insights

This project demonstrates how fundamental linguistic patterns captured by Zipf's law contribute to the statistical foundation of text prediction, providing insights into one aspect of how modern LLMs work at a basic level.