linux-kernel-vfb-driver

An out-of-tree Linux character-device driver (fb536) that presents an in-memory pseudo framebuffer with per-descriptor viewports, six pixel-composition write modes, and selective wait-queue notification.

What it is

fb536 is an out-of-tree Linux kernel module that implements a pseudo framebuffer as a character device. Each minor device — 4 by default — owns a vmalloc'd byte matrix, 1000 x 1000 by default and resizable at runtime to any size from 256 to 10000 per dimension, where every byte is one pixel. User programs drive it through ordinary file operations (open, read, write, lseek, close) plus an eight-command ioctl interface declared in fb536/fb536.h:

ioctl	Purpose
FB536_IOCRESET	Zero the entire framebuffer
FB536_IOCTSETSIZE	Resize; the argument packs (width << 16) | height (destructive: buffer reallocated and zeroed)
FB536_IOCQGETSIZE	Return the current size as (width << 16) | height via the ioctl return value
FB536_IOCSETVIEWPORT	Set this descriptor's viewport rectangle (a struct fb_viewport)
FB536_IOCGETVIEWPORT	Copy the current viewport back to user space
FB536_IOCTSETOP	Select the write mode: SET, ADD, SUB, AND, OR, XOR (0-5)
FB536_IOCQGETOP	Return the current write mode via the ioctl return value
FB536_IOCWAIT	Block until the calling descriptor is notified of a change

Each open file descriptor gets its own viewport (initially the whole framebuffer) and its own write mode (initially SET), so several descriptors can window into different regions of the same device independently.

The code is coursework: the header of fb536/test_fb536.c marks it "CEng 536 - Fall 2025 - Homework 3", and both that file and the module's MODULE_AUTHOR string credit the scullc example from Linux Device Drivers (Rubini and Corbet) as the scaffold. The viewport mapping, composition operators, and selective-notification logic are the driver's own additions on top of that scaffold. As a whole it exercises the core char-driver toolkit: cdev registration, a file_operations table, copy_to_user/copy_from_user, module parameters, a per-device mutex, and wait queues.

How it works

All driver logic is in fb536/main.c; the Makefile builds the module from main.o alone.

State. Each minor is a struct fb536_dev holding a vmalloc'd pixel buffer, its width/height/size, a mutex, a cdev, and a list of open descriptors. Each open() kzallocs a struct fb536_file_desc — viewport, write op, list node, a private wait queue, and a wake flag — links it into the device's list, and stores it in filp->private_data; release() unlinks and frees it.

I/O path. read and write are viewport-relative and row-major: the file offset is split into a viewport row and column, mapped to a global buffer offset (global_row * dev->width + global_col), and processed a row-chunk at a time for reads or a byte at a time for writes. Both clip to the viewport, return short counts at its end, and return 0 once the offset reaches the viewport size. If a viewport no longer fits inside the framebuffer — for example when a concurrent FB536_IOCTSETSIZE shrank it underneath an existing viewport — reads and writes return 0. Writes apply the selected operator per byte, with saturation for ADD (clamped to 255) and SUB (clamped to 0); SET/AND/OR/XOR are plain assignment or bitwise operations. lseek handles SEEK_SET/SEEK_CUR/SEEK_END relative to the viewport size and rejects a negative result with -EINVAL.

Notification. After a successful write the driver builds the rectangle of viewport rows it touched and walks the device's descriptor list, setting the wake flag and calling wake_up_interruptible only on descriptors whose viewport intersects that rectangle. FB536_IOCRESET and FB536_IOCTSETSIZE notify every descriptor (they pass a NULL region); FB536_IOCSETVIEWPORT wakes the descriptor whose viewport it just changed (and rejects, with -EINVAL, a rectangle that would extend past the framebuffer). FB536_IOCWAIT clears the descriptor's wake flag and sleeps in wait_event_interruptible until the flag is set, returning -ERESTARTSYS if interrupted by a signal. FB536_IOCWAIT is rejected with -EINVAL on write-only descriptors, and FB536_IOCTSETOP/FB536_IOCQGETOP are rejected with -EINVAL on read-only ones.

Locking. A single per-device mutex serializes buffer access, descriptor-list manipulation, resize, reset, and notification. The FB536_IOCWAIT sleep itself happens outside the lock; writers set the wake flag before calling wake_up_interruptible, and the waiter re-checks the flag, so a wakeup delivered between unlock and sleep is not lost. FB536_IOCTSETOP and FB536_IOCQGETOP read/write the per-descriptor op without taking the lock.

Lifecycle. fb536_init calls alloc_chrdev_region for a dynamic major, or register_chrdev_region if a major= parameter is supplied, then allocates and zeroes each minor's buffer and adds its cdev. Module parameters (all read-only in sysfs): major (dynamic by default), numminors (default 4), width and height (default 1000 each).

Building & running

Building the module needs the matching kernel headers (/lib/modules/$(uname -r)/build), make, and a C compiler; building and running the test program needs gcc.

Build against the running kernel:

cd fb536
make            # KERNELDIR defaults to /lib/modules/$(uname -r)/build
make clean

KERNELDIR can point at another tree: make KERNELDIR=/path/to/kernel/build. The in-VM build/load sequence used during development is recorded in fb536/qemu_commands.sh:

cd /root/fb536
make clean
make KBUILD_MODPOST_WARN=1
ls -lh fb536.ko
insmod fb536.ko
dmesg | tail -20

Load with non-default parameters if desired:

sudo insmod fb536.ko numminors=8 width=300 height=300

The driver does not create a device class or nodes, so nodes are made by hand from the dynamically assigned major:

major=$(awk '$2=="fb536" {print $1}' /proc/devices)
sudo mknod /dev/fb536_0 c "$major" 0
sudo chmod 666 /dev/fb536_0

fb536/run_test.sh automates the whole loop as root: it rmmods any old module, insmods fb536.ko, reads the major from /proc/devices, creates /dev/fb536_0-/dev/fb536_3 (chmod 666), compiles the suite with gcc -o test_fb536 test_fb536.c -pthread, and runs it:

cd fb536
sudo ./run_test.sh

Test programs. Three standalone programs share fb536/fb536.h:

• fb536/test_fb536.c — the suite run_test.sh builds and runs. Its main() covers basic open/close, size boundaries, reset, viewport operations, all six write ops (with ADD/SUB saturation) and op get/set, seek semantics, and multi-descriptor viewport/op independence. It also defines a threaded selective-wakeup waiter/writer test, but that call is commented out in main() (a source note says pthread_cancel does not work against the blocking ioctl), so it does not run.
• fb536/comprehensive_test.c — a 13-test suite that rmmod/insmods the module before each test for a clean state (uses sudo internally). It adds size-boundary edges (255 and 10001 rejected, 256 and 10000 accepted), one-pixel viewport-overflow rejection, the shrink-under-viewport EOF case, O_RDONLY/O_WRONLY ioctl rejections, a y*width + x memory-layout check, viewport row wrapping, a five-thread race test, and large (10000-byte) read/write. It is not invoked by run_test.sh.
• fb536/test_viewport_boundary_fixed.c — a focused reproducer for the shrink-under-viewport EOF case (open at 500x500, resize to 300x300 through a second descriptor, then expect read/write to return 0).

Kernel/VM environment. Development targeted a Debian guest on Linux 6.12.57; kernelconfig.config is that kernel's configuration (its header reads Linux/x86 6.12.57), and the qemu-debian-create-image submodule references the tool used to build the VM image. .gitignore excludes *.qcow2, *.deb, and *.tar.xz, so the image artifacts are not committed.

Status & caveats

• Coursework-grade code; it passes its own tests but is not production hardened.
• No class_create/device_create: device nodes must be created with mknod (or via run_test.sh).
• No mmap — pixels are reachable only through read/write.
• One mutex per device serializes all I/O on that device; there is no finer-grained or reader/writer locking.
• FB536_IOCTSETOP/FB536_IOCQGETOP touch the per-descriptor op without holding the device lock.
• FB536_IOCWAIT clears its wake flag on entry, so a change made before the call is not remembered; it waits for the next notification.
• Resizing is destructive (buffer reallocated and zeroed) and limited to 256-10000 per dimension.
• No test that actually runs exercises the blocking FB536_IOCWAIT/selective-wakeup path end to end: the threaded test in test_fb536.c is commented out, and comprehensive_test.c only checks that FB536_IOCWAIT is rejected on a write-only descriptor.
• ldd4 and qemu-debian-create-image are recorded as submodule gitlinks with no .gitmodules file, so git submodule update --init cannot resolve them; clone the upstream repositories manually if you need them.
• Everything that makes up the driver lives under fb536/.