codezip

Lossless compression tool for Python source files. Combines code-aware preprocessing with general-purpose compressors, races the candidates in parallel, picks the smallest verified result, and guarantees byte-for-byte round-trip integrity.

How it works (v2)

1. Preprocessing — ~100 common Python tokens (keywords, indentation patterns, type hints, etc.) are replaced with 1-byte dictionary codes using greedy longest-match. The mapping is bijective with an escape rule (0x00), fully reversible. Since v2 the encoder is a C-level bytes-regex substitution and the decoder a bytes.find loop — 6.8x faster than the v1 per-byte Python loops, bit-identical output.

2. Candidate race — instead of v1's sequential extreme-lzma-first search, all candidates for the chosen level compress in parallel threads (the C codecs release the GIL) over shared preprocessed input, and the smallest verified payload wins. RAW always competes, so a result is guaranteed.

3. Levels — fast (zlib-1/zstd-3 class, ~2x of balanced speed), balanced (default: zlib-9, lzma-6, zstd-10 + preprocessed twins), max (--best: adds the old v1 extreme LZMA2 + lzma-9e + zstd-19 for the last tenth of a percent).

4. Verification & integrity — each candidate is decompressed and compared byte-for-byte before it may win; the .czc container carries SHA-256 + original size checked on every decompression.

5. Optional C extensions — zstandard is used when installed (new method codes ZS/PZS) and skipped otherwise; the stdlib-only path always works. v1 files (codes 0–5) decode unchanged.

File format (.czc)

Magic (4B) + Version (1B) + Flags/Method (1B)
  bit 7 = canonical mode flag
  bits 0-6 = compression method (RAW=0, Z=1, L=2, PZ=3, PL=4, LX=5,
    ZS=6, PZS=7; codes 0–5 decode identically to v1, 6–7 need v2 + zstandard)
+ FilenameLen (2B, big-endian) + Filename (UTF-8)
+ OriginalSize (8B) + SHA256 (32B) + Payload

Requirements

• Python 3.10+
• Stdlib only by default; zstandard (pip) unlocks ZS/PZS candidates and is recommended for the best ratio
• Canonical mode requires ruff or black — install with pip install ruff (recommended) or pip install black

Usage

Lossless mode (default) — byte-for-byte identical output

# Compress (balanced level by default)
python codezip.py compress myscript.py
python codezip.py compress myscript.py --level fast   # quicker
python codezip.py compress myscript.py --level max    # best ratio (v1 extreme + more)
# Decompress
python codezip.py decompress myscript.py.czc

Canonical mode — normalize formatting before compression

The code is run through ruff format or black first, formatted, then compressed. Decompression restores the canonical version (not the original — whitespace may differ but semantics are identical). Uses .canonical.czc extension by default.

python codezip.py compress myscript.py --mode canonical
python codezip.py compress myscript.py --mode canonical --formatter black

Decompression works identically — output is the canonical version.

Benchmark

python codezip.py bench codezip.py            # single file, in-memory (no temp files)
python codezip.py bench a.py b.py --best     # several files + slow extremes (lzma -9e, codezip max)
python bench/bench_many.py --best            # whole corpus + byte-weighted totals
python bench/fetch_corpus.py                 # re-download corpus pinned by bench/corpus_manifest.json

Compares codezip levels against gzip -9, bzip2 -9, lzma std, zstd (-3/-19 when installed) and, with --best, lzma -9e.

Benchmark corpus (CC0-grade, fetched — not vendored)

Only public-domain dedications and attribution-free licenses, each pinned to a commit with sha256 in bench/corpus_manifest.json. Third-party files are deliberately NOT committed (bench/corpus/*.py is gitignored); reproduce them byte-identically any time with python bench/fetch_corpus.py (verified 8/8, certutil cross-checked):

File	Source	License
cfgov_forms.py (11.7 KB)	cfpb/cfgov-refresh cfgov/v1/forms.py	CC0-1.0
cfgov_hooks.py (13.1 KB)	cfpb/cfgov-refresh cfgov/v1/wagtail_hooks.py	CC0-1.0
youtubedl_init.py (20.7 KB)	ytdl-org/youtube-dl youtube_dl/__init__.py	Unlicense
ytdlp_init.py (47.9 KB)	yt-dlp/yt-dlp yt_dlp/__init__.py	Unlicense
powertools_logger.py (51.1 KB)	aws-powertools/powertools-lambda-python .../logging/logger.py	MIT-0
ytdlp_options.py (100.9 KB)	yt-dlp/yt-dlp yt_dlp/options.py	Unlicense
powertools_apigw.py (136.3 KB)	aws-powertools/powertools-lambda-python .../event_handler/api_gateway.py	MIT-0
ytdlp_common.py (204 KB)	yt-dlp/yt-dlp yt_dlp/extractor/common.py	Unlicense

bench/bench_many.py --best totals over the 588 KB corpus (Python 3.12, Windows, zstandard installed — timings are indicative):

Method	Size	Ratio	Enc(ms)	Dec(ms)
codezip max	118,596	79.8%	1171.8	23.4
codezip balanced	118,958	79.7%	415.8	21.1
bzip2 -9	118,926	79.7%	64.3	24.4
lzma std	121,180	79.3%	215.9	10.7
lzma -9e	121,220	79.3%	425.0	17.6
zstd -19	122,411	79.1%	348.8	1.7
gzip -9	134,342	77.1%	137.0	4.0
codezip fast	137,223	76.6%	219.6	14.5
zstd -3	143,925	75.4%	4.7	1.2

Honest reading: codezip takes the ratio crown (max 79.8%, balanced 79.7% tied with bzip2 -9) but encodes slower — the candidate race runs several codecs (incl. lzma-6) per file, so balanced costs ~6x bzip2's encode time for the same ratio; decoding is on par with the field. fast is not competitive on ratio here (zstd-3 class without the stronger twins) and exists for quick round-trips, not benchmarks. v2 balanced encodes an order of magnitude faster than the v1 always-extreme strategy at the same ratio, and the v1 preprocessing bottleneck alone is 6.8x faster (bit-identical output).

Validation (cross-checked, not self-reported)

• bench/verify_czc.py — independent checker (own header parse, own split-based preprocess decoder, raw stdlib codecs): 3/3 CLI-made archives verified; restored bytes certutil-identical to originals.
• Full matrix: 8 corpus files + codezip.py itself × 3 levels, plus NUL/empty/tiny edge cases and 1 MB random binary — all round-trip PASS, deterministic across runs (thread race has a fixed tie-break).
• The cross-check caught one real bug during development (in the independent decoder itself, on \x00\x00 pairs) — proof it exercises a genuinely different code path.

CLI output examples

Compress

Compression complete -> codezip.py.czc
  Mode: Lossless
  Level: balanced
  Method: PL
  Original size: 22,186 bytes
  Compressed size: 6,270 bytes
  Ratio: 71.7%

Decompress

Decompression complete -> codezip.py
  Mode: Lossless
  Method: PL
  Size: 22,186 bytes
  Integrity check: PASSED (SHA-256 match)

Benchmark (bench/corpus/ytdlp_init.py, 47,928 bytes)

Method	Size	Ratio	Enc(ms)	Dec(ms)	OK
gzip -9	12,994	72.9%	6.2	0.2	PASS
bzip2 -9	12,211	74.5%	3.6	2.3	PASS
lzma std	12,176	74.6%	17.6	1.0	PASS
zstd -3	13,832	71.1%	0.5	0.1	PASS
zstd -19	12,292	74.4%	17.1	0.2	PASS
codezip fast [PZS]	13,462	71.9%	20.2	0.8	PASS
codezip balanced [PL]	11,992	75.0%	32.2	1.8	PASS