Real-Time Video Abstraction

A modern, high-performance PyTorch implementation of Holger Winnemöller, Sven C. Olsen, and Bruce Gooch's "Real-Time Video Abstraction" (SIGGRAPH '06).

Source Video • Real-Time Abstraction

Source photograph	Stylized (Comic Preset)

Every tone and stroke in the stylized output is computed deterministically in a single forward pass. Nothing was trained or fitted.

Traditional cartoon filters flicker wildly across video frames because hard thresholds pop in and out. The paper's core insight: temporal coherence comes from replacing binary step functions with smooth, continuous transitions (specifically $\tanh$).

Stage	Role in Pipeline	Mathematical Mechanism
Color Decoupling	Isolates brightness from chromaticity	sRGB $\leftrightarrow$ CIELab D65 ($L^, a^, b^*$)
Bilateral Flattening	Suppresses low-contrast texture while keeping silhouettes sharp	Multi-pass iterated bilateral filtering
Soft Line Drawing	Detects edges with anti-aliased, flicker-free ink lines	Separable 1D DoG + $\tanh$ soft thresholding
Soft Quantization	Creates discrete cartoon color bands without contour popping	Periodic $\tanh$ luminance binning + chroma boost

Two key advantages emerge:

• Temporal Stability. Smooth $\tanh$ transitions ensure that as pixel values fluctuate across frames, tones and edges change continuously rather than popping suddenly.
• Extreme Throughput. All operators are mathematically separable or localized, enabling 90+ FPS at 1080p and 330+ FPS at standard resolution on modern GPUs.

Structure

The pipeline processes frames directly in GPU VRAM without CPU-GPU synchronization bottlenecks.

stylize.py      command-line entry point: image, video, or live webcam in, stylized out

core/           the core algorithmic stages
    color.py        vectorized sRGB <-> CIELab D65 conversion on GPU
    edge.py         separable 1D Difference-of-Gaussians & soft tanh thresholding
    bilateral.py    iterated edge-preserving bilateral filter
    quantize.py     soft tanh luminance quantization & chroma saturation boost
    pipeline.py     unifies stages into presets and an end-to-end nn.Module

utils/          I/O helpers and visualization tools
    io.py           GPU-resident tensor loading and saving
    visualization.py side-by-side comparison grids

---

Installation

Requires Python >= 3.10 and PyTorch >= 2.0 with CUDA support:

conda create -n video-abs python=3.12 -y
conda activate video-abs
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu126
pip install opencv-python numpy pillow matplotlib tqdm

---

Usage

1. Stylize a Photograph

python stylize.py --input assets/pushu.jpg --preset comic

The result is saved to test/pushu_comic.png.

2. Stylize a Video File

python stylize.py --input input_video.mp4 --output output_stylized.mp4 --preset fast_realtime --batch-size 8

3. Live Interactive Webcam Mode

python stylize.py --webcam

Interactive Webcam Controls:

Key	Action
1 ~ 6	Switch presets in real-time (comic, subtle, bold, pencil, color_wash, fast_realtime)
O	Toggle side-by-side split screen (Original vs Stylized)
S	Save high-resolution snapshot to disk
Q / ESC	Quit

---

Command-Line Options Reference

Option	Type	Default	Description
--input, -i	str	assets/pushu.jpg	Path to input image or video file
--output, -o	str	None	Path to output destination file
--preset, -p	str	comic	Built-in style preset (see below)
--webcam, -w	flag	False	Launch interactive real-time webcam demo
--cam-id	int	0	Camera device index
--batch-size	int	4	Number of video frames per GPU batch
--device	str	auto	Execution device (cuda, cpu, or auto)
--iterations	int	from preset	Number of bilateral smoothing passes ($k$)
--num-bins	int	from preset	Number of luminance quantization tonal bins ($n$)
--sigma-e	float	from preset	Difference-of-Gaussians spatial sigma ($\sigma_e$)
--saturation	float	from preset	Chromatic saturation multiplier ($S$)

---

Artistic Presets & Feature Showcase

Preset	Visual Output	Characteristics & Settings
comic (Default)		Classic comic-book look with bold ink outlines ($\sigma_e = 1.2$), 8 luminance bins, and $+35%$ saturation boost.
subtle		High-fidelity cartoon with delicate linework ($\sigma_e = 0.7$), 14 tonal bins, and natural colors.
bold		Heavy graphic-novel aesthetic with thick outlines ($\sigma_e = 1.8$), 6 broad color bands, and $+45%$ saturation.
pencil		Monochrome graphite sketch capturing fine texture and hatching contours without bilateral flattening.
color_wash		Painterly watercolor effect with heavy bilateral flattening ($k=5$), 12 soft tones, and minimal linework.

---

Step-by-Step Verification Tests

Run individual tests to inspect intermediate representations:

python test/step1_test_color.py      # CIELab round-trip precision & channel decomposition
python test/step2_test_edge.py       # Difference-of-Gaussians soft line extraction
python test/step3_test_bilateral.py  # GPU iterated bilateral flattening
python test/step4_test_quantize.py   # Soft luminance quantization & composite
python test/step5_test_pipeline.py   # Full preset gallery & throughput benchmark

---

References

• Winnemöller, H., Olsen, S. C., & Gooch, B. (2006). Real-time video abstraction. ACM Transactions on Graphics (TOG), 25(3), 1221-1226.