Skip to content

stack.md

Latest commit

 

History

History
506 lines (398 loc) · 24.3 KB

stack.md

File metadata and controls

506 lines (398 loc) · 24.3 KB
Raw

Networking

TCP/IP Stack

Directory: src/net/ (13 files, ~2700 lines). Modular protocol stack with socket abstraction, NIC drivers, and ARP cache.

Module Overview

File Lines Responsibility
types.rs ~80 MacAddr, Ipv4Addr, SocketAddrV4, TcpState, SocketType, SocketDirection
ethernet.rs ~120 Ethernet frame header (14 bytes: dst MAC, src MAC, ethertype), FCS computation, build_ethernet_frame()
arp.rs ~250 ARP protocol, 64-entry cache with 300s timeout, static entries, request/reply, arp_resolve()
ipv4.rs ~300 IPv4 header (20 bytes), header checksum, packet building, classification (loopback/multicast/link-local)
icmp.rs ~100 ICMP echo request/reply, checksum, build_port_unreachable()
udp.rs ~150 UDP header (8 bytes), pseudo-header checksum, build_udp_datagram()
tcp.rs ~800 TCP state machine (11 states), connection lifecycle, send/recv buffers (16 KB sliding window), segment building
socket.rs ~700 SocketManager, bind/connect/listen/send/recv/close, KWait wake, udp_dispatch(), tcp_dispatch()
nic.rs ~240 NetworkInterface trait (9 methods), NicRegistry (4 slots), IP/next-hop/gateway, Route local-routing decision, vendor/device/description per NIC. NICs registered via NEM bridge
loopback.rs ~120 Loopback interface (127.0.0.0/8, #484): TX queue, LoopbackInterface, loopback_send/pump, \Device\Loopback
net_bridge.rs ~100 NEM network bridge: hst_register_network_device, wraps NEM callbacks as NetworkInterface
counters.rs ~150 Global + per-interface counters (RX/TX packets/bytes/errors per NIC slot + loopback), exposed via NetStats (28); periodic dump every 1000 ticks
tests.rs ~300 18+ integration tests

TCP State Machine

11 states defined in TcpState:

pub enum TcpState {
    Closed,      // Initial state, no connection
    Listen,      // Waiting for SYN
    SynSent,     // SYN sent, awaiting SYN+ACK
    SynRcvd,     // SYN received, SYN+ACK sent
    Established, // Connection established, data transfer
    FinWait1,    // FIN sent, awaiting ACK
    FinWait2,    // FIN ACKed, awaiting peer's FIN
    CloseWait,   // FIN received, waiting for app to close
    Closing,     // Both FINs sent, awaiting final ACK
    LastAck,     // FIN received, FIN sent, awaiting FIN ACK
    TimeWait,    // All done, waiting for delayed segments
}

Connection lifecycle: build_tcp_segment(), send_tcp_segment(), tcp_send_syn_ack(), tcp_handle_ack(). Send/recv buffers use a 16 KB sliding window.

Data path (#486): tcp_send() buffers; tcp_tick() (from net_tick) flushes up to MSS per connection, tracks send_base/send_next, advances the window on ACKs, retransmits on a 200 ms TSC-based RTO (8 retries max), and drives FIN (fin_pending when closing with unsent data). SynReceived connections also retransmit their SYN-ACK and time out with the same bound. Incoming data is stored in the connection buffer, ACKed, and wakes SocketRead waiters; incoming FINs park in CloseWait. tcp_dispatch() never holds SOCKET_MANAGER across protocol actions (nested locks deadlocked SYN handling).

RX dispatch and locking

network_poll_all() drains a bounded batch of frames while holding NIC_REGISTRY, then dispatches them after releasing it. The dispatcher must not hold the registry: the TCP send path re-locks it (nic_next_hop, nic_default_id, nic_send_packet), and spin::Mutex is non-reentrant, so holding it deadlocked netpump on the first TCP segment received over a real NIC. net_handle_incoming_packet(nic_id, frame) takes the interface id (not a borrowed interface) and emits replies through net_send_frame(), which routes loopback frames through the loopback queue.

RX validation

Received IPv4 datagrams are bounded by the header's total length (so Ethernet padding never enters a transport checksum), the IPv4 header checksum is verified, and TCP/UDP checksums are verified (tcp_checksum_valid, udp_checksum_valid; a zero UDP checksum means "not computed" and is accepted). Invalid frames are counted as RX errors and dropped.

Connection teardown

socket_free calls tcp_release_connection: a connection that still has buffered/unacknowledged data is not freed immediately — it is marked reap_pending and tcp_tick transmits the remaining payload and FIN before reaping it once Closed. Freed immediately only when already drained. This prevents losing the last write when an application closes right after send. There is no TIME_WAIT / delayed-ACK handling yet.

Socket wait (Ring 3)

ObSetInfoClass::SocketListen/SocketClose carry no payload and are accepted with an empty buffer by the generic ob_set_info validation. ob_wait on a socket blocks on WaitReason::SocketConnect / SocketAccept / SocketRead (socket_wait_reason), and the handshake, inbound data, FIN and connection failure all wake the corresponding waiters. Before this, ob_wait returned NoSys for sockets, so Ring 3 callers had to busy-poll socket_recv.

Tests and validation

Covered by the kernel tests net_tcp_loopback_e2e (handshake + both directions + orderly close over loopback), net_tcp_nic_path_syn_synack_no_deadlock (real dispatch path through a scripted NIC), net_tcp_synack_retransmit_bounded, net_tcp_nic_bad_checksum_dropped, net_tcp_close_flushes_buffered_data, and the checksum-validation tests. Off-subnet TCP uses nic_next_hop (gateway- aware); #315 is resolved. The Ring 3 utilities netsrv/netcli (real client and echo server: connect/accept, finite timeouts, clean close) and the tcptest self-test validate the public socket ABI. In VirtualBox NAT (e1000, guest 10.0.2.15) netcli 10.0.2.2 5555 reached a host echo server and a host client reached netsrv 6666 through a NAT port-forward.

ObType Integration

ObType::Socket = 18. Sockets are Ob objects managed through the standard handle/namespace system.

ObInfoClass (query)

Class ID Name Returns
17 SocketInfo Socket state, type, local/remote addresses
18 SocketAddr Bound address and port
19 TcpStatus TCP connection state
20 NicInfo NIC metadata: MAC, IP, link status, vendor/device PCI IDs, driver name, description
23 SocketRecv Received data (non-blocking read)
43 SocketAccept Accept a pending connection from a listener (4-byte fd out)

ObSetInfoClass (mutate)

Class ID Name Effect
18 SocketConnect Initiate TCP connection or set UDP remote
19 SocketBind Bind socket to local address/port
20 SocketListen Start listening for TCP connections
21 SocketSend Send data on connected socket
22 SocketClose Close socket (FIN or RST)
27 SetNicIp Set NIC IP address from userspace
28 SetNicGateway Set NIC default gateway (0.0.0.0 = unset)

Socket creation via ob_create with attrs encoding: bits 0-7 = socket type (1=TCP, 2=UDP, 3=Raw), bits 8-23 = port for well-known bindings.

KWait Integration

Socket objects support KWait (kernel wait) for blocking I/O operations:

Wait Reason Triggers On
SocketRead Data arrives in socket receive buffer
SocketConnect TCP handshake completes (SYN+ACK received)
SocketAccept New connection arrives on listening socket

Blocking syscalls suspend the calling thread via KWait and wake when the corresponding event fires.

Ring 3 listener / accept semantics

A listening socket stays in Listen; each incoming pure SYN (SYN without ACK) creates a distinct socket + TCP connection, sends the SYN-ACK and queues the connection on the listener (MAX_ACCEPT_BACKLOG = 8; excess SYNs are dropped, the listener stays usable). ob_query_info(SocketAccept) pops an established connection, wraps it in a new Socket Ob object/handle and returns the new fd (EAGAIN when none is ready). This allows sequential — and, up to the backlog and MAX_SOCKETS, concurrent — connections without cross-talk. A SYN-ACK is never routed to a listener; it reaches the client connection.

Blocking and timeouts

ob_wait on a socket blocks on SocketAccept (listener), SocketConnect (handshake in progress) or SocketRead (data/EOF). A non-zero ob_wait timeout arms a TSC deadline serviced by the RX pump (net_tick → kwait::deadline_tick), which wakes the waiter when it expires. The timeout wake carries no distinct status (ob_wait returns 0); the caller must re-check readiness. Indefinite waits (timeout = 0) are unchanged.

Send / receive return values

SocketSend returns the number of bytes buffered; a full send buffer returns EIO (retry after draining). SocketRecv returns bytes copied, or EAGAIN when none are buffered. There is no distinct EOF: SocketRecv returns EAGAIN both when no data is available and when the peer has closed; the connection state (TcpStatus) distinguishes CloseWait/Closed. Zero-length sends/recvs are rejected as EINVAL.

Packet Flow

NIC (e1000)
  -> poll_packet() -> 2048 byte buffer
    -> ethernet::EthernetHeader parse (dst MAC, src MAC, ethertype)
      |
      +-> ARP (ethertype 0x0806):
      |     -> arp_resolve() -> cache lookup (broadcast IP -> broadcast MAC directly)
      |     -> if request for our IP: build reply
      |     -> if reply: update cache
      |
      +-> IPv4 (ethertype 0x0800):
            -> Ipv4Header parse (version, IHL, total_len, protocol)
              |
              +-> ICMP (proto 1):
              |     -> echo request -> build echo reply -> send
              |     -> port unreachable -> build and send
              |
              +-> TCP (proto 6):
              |     -> tcp_dispatch() -> socket lookup by (src IP, src port, dst IP, dst port)
              |     -> segment processing (SYN, ACK, FIN, data)
              |     -> data delivery to socket recv buffer
              |     -> KWait wake for SocketRead/SocketConnect
              |
              +-> UDP (proto 17):
                    -> udp_dispatch() -> socket lookup by (dst port)
                    -> data delivery to socket recv buffer
                    -> KWait wake for SocketRead

Loopback (127.0.0.0/8, #484)

Virtual interface outside NicRegistry (registering it would change default_nic_id() and consume a NIC slot). Frames to 127/8 are queued in loopback.rs and drained through the single dispatch path net_handle_incoming_packet(), both from network_poll_all() and synchronously on the send path (so back-to-back send+recv never sees EAGAIN).

Rules: no NIC required (works with 0 NICs), no ARP, no gateway lookup, no new syscalls (sockets keep using ObType::Socket), \Device\Loopback in the Ob namespace. Synthetic MAC 02:00:00:00:00:01 (MacAddr::loopback()), never on the wire. nic_route() classifies destinations as Route::{Loopback, OnLink, ViaGateway, Unreachable}. ping 127.0.0.1 answers through the real ICMP dispatch (request → echo reply → notify); RAX 36 reports a nominal 1 µs since 0 means failure. The interface is also enumerated by the NicInfo query (sentinel nic_id, read-only) and shown by ipconfig as the Loopback adapter. Validated in VirtualBox: 5 kernel tests (net_loopback_*) green plus a Ring-3 ping 127.0.0.1 check in cmdtest.nxe.

NIC Initialization (Phase 3.88)

  1. Create \Device\Tcp and \Device\Udp namespace entries in Ob
  2. Probe PCI bus for Intel e1000 devices (vendor 0x8086, devices 0x100E/0x1004/0x100F/0x10D3)
  3. Initialize found NIC: map MMIO BAR, allocate RX/TX ring buffers, configure descriptors
  4. Set initial IP configuration (default: 0.0.0.0 — no kernel DHCP)
  5. Userspace DHCP service (dhcpd.nxe) configures IP asynchronously
  6. ARP cache begins accepting entries

Link state (is_link_up)

NIC drivers run as NEM modules (drivers/e1000). The kernel NEM bridge (src/drivers/nem/loader/net_bridge.rs) registers a bootstrap NetworkInterface at driver-load time, so it cannot call back into a driver that is still loading. Instead:

  • The e1000 driver exports driver_link_up(), which reads STATUS.LU (with a link-ready flag cached during init_e1000_hw) and publishes the state through hst_set_network_link_state. Since the NEM bridge has no link-query slot, the driver also re-publishes from its RX poll entry (throttled: 1 MMIO read per 512 polls, #529); unload publishes down via driver_fini.
  • The Ring-0 netpump kernel thread calls nic::nic_poll_link_state() once per network_poll_all() and stores the result in NicSlot::link_up; the NicInfo query and the netapplier link-up edge detection read that cached value.
  • A NIC is never advertised as link-up before its driver reports a real link.

The e1000 is polled, not interrupt-driven: RX is drained by network_poll_all() from the Ring-0 netpump worker and from the sys_yield syscall path. (netd is now the Ring 3 network service — see userland.md — not the RX pump.) Issue #339 established that this polling (not driver link/ring bring-up) is the relevant variable for the first DHCP DISCOVER.

NicRegistry

Manages up to 4 NIC slots. Each slot holds a Box<dyn NetworkInterface>.

pub struct NicRegistry {
    pub nics: [Option<NicSlot>; 4],
}

DHCP Client — Userspace Service

DHCP is NOT handled by the kernel. The protocol runs entirely in userspace as the dhcpd.nxe service, following the NT-like design principle where protocol stacks live in user mode and the kernel only provides transport primitives.

Architecture

dhcpd.nxe (Ring 3 user service)
  │
  ├─ Creates UDP socket (ObType::Socket = 18)
  │   ├─ bind(0.0.0.0:68)        ← DHCP client port
  │   └─ connect(255.255.255.255:67)  ← DHCP server port (broadcast)
  │
  ├─ Performs DORA sequence:
  │   ├─ DISCOVER → wait → OFFER
  │   ├─ REQUEST  → wait → ACK
  │   └─ On ACK: publish IP/mask/gw/DNS to the Registry
  │              (the netapplier service applies it to the NIC)
  │
  ├─ Supervises the lease (#316): unicast renew at server T1 (default 50%),
  │            broadcast rebind at server T2 (default 87.5%), DORA restart on NAK/expiry
  ├─ Falls back to APIPA (169.254.1.1) if DHCP fails
  └─ Publishes IP configuration to the Registry (never applies it directly)

Kernel (Ring 0) provides:
  ├─ NIC access (e1000 kernel stub or NEM driver)
  ├─ Ethernet / ARP / IPv4 / UDP protocol processing
  ├─ Socket abstraction with recv/send via Ob syscalls
  └─ No DHCP protocol logic — removed in v0.48.8+

DORA Sequence (RFC 2131)

  1. DISCOVER: Broadcast UDP packet with DHCPDISCOVER message type
  2. OFFER: Received on bound UDP socket (port 68), parsed for offered IP and options
  3. REQUEST: Unicast or broadcast DHCPREQUEST with offered server ID
  4. ACK: Final acknowledgment; the lease is published to the Registry and the netapplier service applies it to the NIC (SetNicIp) — dhcpd never configures the NIC directly

Lease Renewal

After binding, dhcpd.nxe enters a loop that:

  • Tracks elapsed time via sys_yield() batches (approximating seconds)
  • At 50% of lease time, sends unicast DHCPREQUEST to the serving DHCP server
  • On ACK: lease is extended, timer resets
  • On failure: retries up to MAX_RETRIES, then restarts DORA

Why Userspace?

The previous kernel-based DHCP implementation had fundamental architectural problems:

  • build_dhcp_packet() used Vec (heap allocation) from timer IRQ context
  • nic_send_packet() acquired NIC_REGISTRY.lock() (spinlock) from IRQ context, risking deadlock
  • dhcp_tick() in the idle loop never ran because user threads were always Ready
  • Result: DHCP never progressed, and the DHCP client always fell back to APIPA

Moving DHCP to userspace resolves all these issues:

  • All memory allocation happens in process context (safe)
  • No spinlocks held in IRQ context
  • Socket operations are fully preemptible
  • Clean separation of concerns: kernel transports packets, user mode runs protocols

Registry Integration

dhcpd.nxe stores network configuration in:

\Registry\Machine\System\CurrentControlSet\Services\Network\Interfaces\0
  DHCPEnabled = 1 (DWORD)
  IPAddress   = <assigned IP> (DWORD, big-endian byte order)
  SubnetMask  = <subnet mask> (DWORD)
  DHCPBound   = 1/0 (DWORD, 1 when DHCP lease is active)
  DHCPServer  = <server IP> (DWORD)

dhcpd only publishes here; the resident netapplier service reads this key and applies IP/mask/gateway to the runtime NIC (single applier; see #365).

Change notification (#364). The applier no longer spin-polls the key. A config writer that applies a NIC address (ObSetInfoClass::SetNicIp/SetNicGateway, i.e. libnet::config::apply) makes the kernel signal EVENT_NET_CONFIG_CHANGED (18) via kwait; the applier blocks on an Event object (ob_create(Event, attrs = 18) + ob_wait) and re-reads the Registry on wake. ob_wait carries a finite timeout (2 s) so polling remains a fallback if the notification object is unavailable (degradation).

DNS Resolver (Userland)

DNS resolution is not a kernel subsystem: like DHCP, it runs in user mode on top of the kernel's UDP sockets. The protocol is implemented once in the libdns crate and used by libnet (and therefore by nslookup and ping).

Layer Path Responsibility
Wire format + resolver core libdns/src/lib.rs Query/response encoding, A/CNAME parsing, hostname validation, DnsError, retries/server selection
Userland transport + config libnet/src/dns.rs UDP via net.nxl, Registry servers (DnsServer, DnsServer2/3), bounded cache, resolve() / resolve_with_server()
Tools userbin/nslookup, userbin/ping Consume the shared resolver API

The kernel-side DNS parser/cache (src/net/dns.rs) has been retired (#312): it was only reachable from dns_tick()/tests and duplicated libdns. DNS now has a single implementation — libdns (wire format) + libnet::dns (transport) — and the kernel only polls the NIC and runs ARP/TCP ticks.

See docs/networking/userland.md §3.4–3.5 for the API, configuration and error taxonomy.

UDP Send Path (next-hop routing)

Making DNS work on a real network required three fixes in the kernel/userland send path (both were latent because DHCP only uses broadcast):

  • Source address. A UDP socket bound to 0.0.0.0 (the DNS resolver binds port 0) had an unspecified source address in the IP header. socket_send_udp_raw now fills it from the NIC, except for broadcast datagrams (DHCP DISCOVER keeps 0.0.0.0).
  • Next-hop routing. IPv4 next-hop is centralized in nic_next_hop(dest) (NicRegistry::next_hop_ip): on-link → the destination; off-link → the NIC's configured gateway; off-link without a gateway (0.0.0.0, unset) → no valid next hop and the send fails cleanly (it never ARPs the remote destination). The gateway is a real per-NIC property (NicSlot::gateway), set by ObSetInfoClass::SetNicGateway (28) from DHCP Option 3 / static config, and persisted in Network\Interfaces\0\Gateway. socket_send_udp_raw and icmp_ping both use nic_next_hop. There is no implicit 10.0.1.1 fallback. Bound-interface selection (#317): UDP (socket_send_udp_raw_on) and TCP (tcp_nic_for → resolve_src_ip / tcp_mac_for / send_tcp_segment) now select the source address/MAC, the next hop (nic_next_hop_for(nic_id, dst)) and the ARP resolution (arp_resolve_on(nic_id, target)) from the socket's bound nic_id (ObSetInfoClass::SocketBindNic = 29), falling back to the default NIC when unbound. icmp_ping has no socket and keeps using the default NIC.
  • Datagram dispatch. udp_dispatch matched connected sockets only by remote.port; it now matches the socket's local port (dst_port) and, when set, the remote port, so a previous socket cannot capture replies addressed to a different local port.

Because netpump (not the syscall) drives RX polling and there is no blocking socket wait syscall, the userland resolver waits for the reply with an RDTSC time budget rather than a bare sys_yield loop (which returns immediately when no other thread is runnable).

QEMU Networking

NeoDOS uses QEMU's user-mode networking (SLiRP) by default, which requires no root/sudo privileges.

Default Mode (sudo-free)

-netdev user,id=net0,net=10.0.1.0/24,dhcpstart=10.0.1.80,host=10.0.1.1 \
-device e1000,netdev=net0

QEMU's built-in DHCP server assigns IPs in the 10.0.1.x range starting at 10.0.1.80. The host is accessible at 10.0.1.1 and provides NAT to the outside world.

TAP Mode (requires privileges)

neodev run --net tap

TAP networking requires:

  • /dev/net/tun access (requires CAP_NET_ADMIN or root)
  • A preconfigured tap0 interface on the host

Useful when the guest needs direct network access (e.g., DHCP from a real LAN server).

Source Files

File Path
types.rs src/net/types.rs
ethernet.rs src/net/ethernet.rs
arp.rs src/net/arp.rs
ipv4.rs src/net/ipv4.rs
icmp.rs src/net/icmp.rs
udp.rs src/net/udp.rs
tcp.rs src/net/tcp.rs
socket.rs src/net/socket.rs
nic.rs src/net/nic.rs
e1000 drivers/e1000/ (NEM)
tests/ src/net/tests/

User-mode DHCP service: userbin/dhcpd/src/main.rs

ARP Resolution Flow

When a packet needs to be sent to an IP address (e.g., ICMP ping), the ARP resolution follows this sequence:

  1. Cache lookup: arp_lookup(target_ip) checks the 64-entry ARP cache.

    • Broadcast IP → broadcast MAC (no ARP needed).
    • Cache hit → returns MAC immediately.
    • Cache miss → proceeds to send ARP request.

  2. ARP Request: An Ethernet frame is built with:

    • Destination MAC: FF:FF:FF:FF:FF:FF (broadcast)
    • Ethertype: ETH_TYPE_ARP (0x0806)
    • ARP operation: ARP_OP_REQUEST (1)
    • Sender MAC/IP: NIC's MAC and IP
    • Target IP: the IP to resolve

  3. Wait for Reply: The sender polls network_poll_all() in a tight loop with RDTSC-based timeout (500ms). Each poll checks for received packets via the e1000 RX ring.

  4. ARP Reply Reception: net_handle_incoming_packet() processes incoming Ethernet frames:

    • If ethertype is ETH_TYPE_ARP (0x0806) and operation is ARP_OP_REPLY (2):
      • Extracts sender IP and MAC from the ARP payload
      • Calls arp_insert() to add/update the cache entry
    • arp_insert() updates existing entries or inserts new ones (LRU eviction at 64 entries)

  5. Resolution Complete: The polling loop in icmp_ping() detects the cache entry and returns the resolved MAC address.

  6. ICMP Echo Request: The resolved MAC is used as the destination in the Ethernet frame, and the ICMP echo request is sent.

  7. ICMP Echo Reply Wait: Polls LAST_PING_REPLY atomic with RDTSC-based timeout (configurable, default 1s).

Important implementation details

  • The icmp_ping() function in icmp.rs contains its own ARP resolution logic (inline in or_else closure) rather than calling arp_resolve(). This is because icmp_ping() needs to block waiting for the reply, while arp_resolve() is fire-and-forget.
  • The ARP request Ethernet frame must use ETH_TYPE_ARP (0x0806) as the ethertype. Using ETH_TYPE_IPV4 (0x0800) will cause the receiver to reject the frame.
  • The ARP cache is protected by spin::Mutex. Entries expire after 300 seconds (checked every 10 ticks).
  • QEMU user-mode (SLiRP) networking supports ARP. VirtualBox Bridge Mode forwards ARP to the physical network.

Known limitations

  • Only one ARP resolution can be in-flight at a time (no pending queue for concurrent requests).
  • The e1000 RX descriptor status is read without explicit read_volatile, relying on the compiler generating fresh memory reads via &mut self reference.
  • Gratuitous ARP sent automatically on IP address change via nic_set_ip().

Tests

17+ tests in src/net/tests/ covering: MAC address formatting, IPv4 header checksum, ARP cache operations, TCP state machine transitions, TCP full lifecycle (listen -> connect -> established -> close), ICMP echo request/reply, socket creation/lookup/bind/connect, UDP header construction, NIC registry add/remove.

Kernel DHCP tests removed (DHCP is now a userspace service).