fineweb

Open-source code for training and scaling up energy-based foundation models.

LiteGPT: A 124M Small Language Model (SLM) pre-trained on FineWeb and fine-tuned on Alpaca.

Train a SmolLM-style llm on fineweb-edu in JAX/Flax with an assortment of optimizers.

A 134M-parameter GPT written from scratch in PyTorch, no nn.Transformer and no HuggingFace, pretrained on a 7B tokens corpus of FineWeb-Edu, where it consumed 1.2B tokens on one free Kaggle P100. Held-out perplexity 36.12 against a Chinchilla prediction of 38.70.

Exploring the BBC News subset of the FineWeb dataset (via HuggingFaceFW/fineweb's dated subsets on HF), originally a Common Crawl dataset.

Local public-text ingestion and research engine. Backfill archives, follow feeds, search a unified corpus and track topics through a Python CLI, FastAPI dashboard and DuckDB.

A 66M parameter decoder-only transformer language model implemented from scratch in PyTorch. Features a custom SentencePiece tokenizer, RoPE positional embeddings, SwiGLU feed-forward network, per-layer KV cache for efficient autoregressive inference, and a Svelte-based streaming chat interface.

A 197M-param decoder-only LM trained from scratch on FineWeb-Edu — PyTorch with RoPE, GQA, SwiGLU, 32k BPE, built for 6GB VRAM

Training GPT-2 on FineWeb-Edu in JAX/Flax

OpenAI Parameter Golf competition entry — adaptive Hessian-sensitivity GPTQ clipping on PR #1855's stack. 1.06310 BPB (3-seed mean), PR #1962.

How much wall-clock training time data-selection methods actually save for GPT-2 small, with the cost of selection charged against the savings. The data axis of the NanoGPT speedrun, sized for free Kaggle and Colab GPUs.

Teacher-student knowledge distillation for educational content scoring: Llama-3.3-70B labels web text 0-5, a Ridge regressor learns it at 96.6% F1 for a fraction of the cost

Prune Qwen2.5-0.5B from 24 to 7 layers, distill it back on 800M tokens, and test what the loss curve can and can't tell you.

Do quality filters pull benchmark questions into pretraining corpora? Injects MMLU, GSM8K, GPQA, ARC, HellaSwag, PIQA and TruthfulQA items into a corpus and ranks them with six quality classifiers, including DCLM, FineWeb-Edu and Gaperon.

A minimal GPT model pretraining pipeline (mostly taken from Andrej Karpathy's build-nanogpt repo)

Decoder-only LLM from scratch with reproducible data pipelines, tokenizer/sharding workflows, and GPU training.

FineWeb-Edu dataset analysis using Apache Spark - DSC 232R group project

Qwen3-style language modeling with FineWeb streaming and tensor parallelism.