Storage I/O lab

Five small C programs that make the Linux storage stack visible.

Which pages of a file are in RAM right now. Where a filename's bytes physically sit on the disk. Whether your config file survives a kill -9 at the worst possible instant. Each program answers one of those by asking the kernel directly, and prints the answer in a form you can read.

This is a learning lab, not a library. The programs are instruments: each one exists to make a single kernel behaviour observable, and the interesting part is usually what surprises you when you run it.

$ ./ch1-cache/resident /mnt/work/small.dat        # after one 4 KiB read
page        0  ####.... ........ ........ ........ ........ ........ ........ ........    4/64
page       64  ........ ........ ........ ........ ........ ........ ........ ........    0/64
page      128  ........ ........ ........ ........ ........ ........ ........ ........    0/64
page      192  ........ ........ ........ ........ ........ ........ ........ ........    0/64

resident: 4 / 256 pages (1.6%)  16.0 KiB / 1.0 MiB

One read was requested. Four pages arrived. Chapter 1 is about why.

Where each chapter cuts into the stack

flowchart TD
    app["your program: read() / write() / mmap()"]
    vfs["VFS: getdents64, statx, rename"]
    pc["page cache: dirty pages, readahead"]
    fs["filesystem: ext2 / ext4 inodes and blocks"]
    blk["block layer + device"]

    app --> vfs --> pc --> fs --> blk
    app -. "O_DIRECT bypasses" .-> fs

    ch1["ch1 · which pages are resident, and COW"]:::c --- pc
    ch2["ch2 · name to inode to block, by hand"]:::c --- fs
    ch3["ch3 · when a write is actually durable"]:::c --- pc
    ch0["ch0 · what alignment the device demands"]:::c --- blk

    classDef c fill:#1f2933,stroke:#f0883e,color:#e6edf3

Loading

The chapters

	Program	The question it answers	Status
ch0	dioalign	What alignment does direct I/O demand on this file, and how do I know the kernel actually told me?	done
ch1	resident, cowpfn	Which pages of this file are in RAM right now, without perturbing the answer by looking? And does fork() really not copy anything?	done
ch2	dirdump	Where do a file's bytes physically live, walking name to inode to block with no filesystem driver helping?	done
ch3	atomic_replace	Which of these eight syscalls can I skip and still survive a crash at any instruction?	done
ch4	directread	When does O_DIRECT win, when does it lose, and what do you have to rebuild to make it win?	planned
ch5	rawring	Which io_uring operations silently punt to a kernel thread pool?	planned

Each chapter directory has its own README with the run commands, real captured output, and what to look for in it.

Run it

You need Linux. Not WSL1, not macOS: half of this is Linux-only by definition (mincore behaviour, /proc/self/pagemap, getdents64, drop_caches, STATX_DIOALIGN). A VM or container against a Linux kernel is fine, that is what this was built on.

sudo apt install -y build-essential gdb e2fsprogs vmtouch
git clone https://github.com/valentynkit/linux-io-lab && cd linux-io-lab
make

make builds every program under ch*/. make ch1 builds one chapter. make clean removes the binaries. Warnings are errors and AddressSanitizer is on, deliberately: this is code you are supposed to break.

The bench

Every experiment runs on a loop-mounted disk image, never on your real root filesystem. Two images, because chapter 2 wants a filesystem simple enough to read by hand:

mkdir -p ~/iolab && cd ~/iolab
dd if=/dev/zero of=ext4.img bs=1M count=1024
dd if=/dev/zero of=ext2.img bs=1M count=64
mkfs.ext4 ext4.img && mkfs.ext2 -b 4096 ext2.img

sudo mkdir -p /mnt/work /mnt/dissect
sudo mount -o loop ext4.img /mnt/work        # the working filesystem
sudo mount -o loop ext2.img /mnt/dissect     # the one you take apart
sudo chown $USER /mnt/work /mnt/dissect

Two reasons for the loop images rather than a directory on your root disk. Dropping caches and flipping writeback sysctls affects the whole machine, so you want the thing you are measuring to be small and yours. And chapter 2 reads the filesystem's raw bytes with dd, which requires a filesystem you can unmount without ending your session.

Sanity check the bench before trusting any number from it:

mount | grep /mnt/work                     # ext4, not something else
losetup -a                                 # which loop device is which
cat /sys/block/loop1/queue/read_ahead_kb   # readahead window, ch1 needs it

stat -f cannot tell ext2 from ext4: both report magic 0xEF53. Use mount.

How to use the lab

The programs are worth about ten minutes each if you just run them. The value is in the loop around them:

1. Predict. Write down the expected output in one sentence before running. Every chapter README opens with the predictions worth committing to.
2. Run.
3. Reconcile. Where the prediction was wrong, that is the actual lesson, and it is the only part worth remembering.

A wrong prediction is the product. If everything you predict comes true, the experiment was too easy and you learned nothing.

One environment rule that will save you an afternoon: never separate loading the cache from measuring it. Chain the drop, the load, and the measurement on one command line with no pause between them. Something else on the machine will evict your pages the moment you look away, and you will spend twenty minutes debugging a kernel that is behaving correctly.

What this code is and is not

• Written to be read, in the order the syscalls happen. Error paths are explicit and repetitive on purpose, because half the lesson is which calls can fail and what the failure means.
• Not production code. No retry logic, no configurability, no library surface. atomic_replace in particular is a demonstration of a durability skeleton, not a drop-in replacement for a real one.
• The numbers in the chapter READMEs are real, captured from actual runs, and specific to one bench: 4 KiB pages, 128 KiB readahead, ext4 and ext2 on loop devices, aarch64, kernel 7.0. Yours will differ. The shapes should not.

Reading that goes with it

• man 2 mincore, man 2 getdents64, man 2 statx, man 2 rename, man 2 fsync, man 7 io_uring. Most chapters here are one man page taken seriously.
• Dan Luu, Files are hard.
• Poirier, The Second Extended File System.
• LWN, PostgreSQL's fsync() surprise.
• Kernel docs: admin-guide/mm/pagemap.rst, admin-guide/sysctl/vm.rst.

License

MIT, see LICENSE. Take the programs and break them.