Skip to content

Executing any of tools (ifconfig, netstat, ipfw) breaks running primary process #860

Closed

Description

@blaa

Hello,
I'm chasing a issue as in the title.

Reproduction steps:

  1. Build f-stack 1.24 on Debian Bookworm VM using VirtIO network drivers and DPDK from packages (22.11.7)
  2. Build nginx app, run the app.
  3. Curl can correctly connect to the nginx.
  4. Execute ff_ifconfig --help (so it doesn't really do anything at all. Only ff_ipc_init and rte_eal_init)
  5. nginx stops accepting connections.

ifconfig output:

/data/f-stack # ff_ifconfig --help
Device 0000:00:12.0 is not driven by the primary process
EAL: Requested device 0000:00:12.0 cannot be used
ff_ifconfig: illegal option -- -
usage: ifconfig -p <f-stack proc_id> [-f type:format] [-g groupname] [-C] [-L] interface address_family
                [address [dest_address]] [parameters]
       ifconfig -p <f-stack proc_id> interface create
       ifconfig -p <f-stack proc_id> -a [-g groupname] [-C] [-L] [-d] [-m] [-u] [-v] [address_family]
       ifconfig -p <f-stack proc_id> -l [-d] [-u] [address_family]
       ifconfig -p <f-stack proc_id> [-g groupname] [-C] [-L] [-d] [-m] [-u] [-v]

Devbind:

Network devices using DPDK-compatible driver
============================================
0000:00:13.0 'Virtio network device 1000' drv=uio_pci_generic unused=virtio_pci
0000:00:14.0 'Virtio network device 1000' drv=uio_pci_generic unused=virtio_pci

Network devices using kernel driver
===================================
0000:00:12.0 'Virtio network device 1000' if=ens18 drv=virtio-pci unused=virtio_pci,uio_pci_generic *Active*

The same happens on our own code that we were trying to use TRANSPARENT IP sockets - using ipfw.
Both codes use kqueue and loop keeps executing, just there are no incoming events on new connections.

Activity

  1. blaa commented on Feb 18, 2025

    @blaa
    Author

    If I remove ff_ipc_init from ifconfig.c the ifconfig --help won't break the primary process. But, ifconfig prints the interfaces correctly still... and then breaks the process. It calls rte_eal_init later then:

    #0  0x0000555556de4290 in rte_eal_init ()
    #1  0x0000555556498f59 in ff_ipc_init () at ff_ipc.c:68
    #2  0x0000555556498ff7 in ff_ipc_msg_alloc () at ff_ipc.c:94
    #3  0x00005555564a593a in sysctl (name=0x7fffffffdc30, namelen=6, old=0x0, oldlenp=0x7fffffffdc28, new=0x0, newlen=0) at sysctl.c:46
    #4  0x00005555564a3035 in getifaddrs (pif=0x7fffffffe140) at getifaddrs.c:125
    #5  0x000055555648f6a9 in main (argc=0, argv=0x7fffffffe300) at ifconfig.c:691
    

    When stepping through the code it seems rte_eal_init is not a problem, but quitting the process after execution is. Disabling rte_eal_cleanup calls doesn't help.

  2. blaa commented on Feb 18, 2025

    @blaa
    Author

    This seems to be the case:
    https://lore.kernel.org/r/dpdk-dev/20241128091552.2406f1a3@hermes.local/T/
    So it's a DPDK bug that is present on Debian Stable / Bookworm.

    EDIT: Also the same bug seems to be present in the included DPDK with f-stack.

  3. jfb8856606 commented on Mar 18, 2026

    @jfb8856606
    Contributor

    Root Cause Analysis

    After investigating the DPDK source code and upstream mailing list, here is the complete picture.

    Bug-introducing commit

    1cab1a40ea9b — "bus: cleanup devices on shutdown" (2022-10-04, DPDK 22.11)

    This commit added eal_bus_cleanup() to rte_eal_cleanup() without distinguishing between primary and secondary processes.

    What happens when a secondary process exits

    rte_eal_cleanup()
      rte_mp_channel_cleanup()   <- closes IPC socket (mp_fd = -1)
      eal_bus_cleanup()          <- calls drv->remove() on ALL PCI devices  ← BUG
        pci_cleanup()
          drv->remove(dev)       <- eth_virtio_pci_remove()
            virtio_dev_close()
              virtio_reset(hw)   <- writes 0x00 to VIRTIO_CONFIG_STATUS_RESET !
    

    virtio_reset() issues a full hardware reset of the NIC. The primary process still holds the device but its hardware state has been destroyed — it stops receiving packets.

    This is why even ff_ifconfig --help (which triggers rte_eal_init → rte_eal_cleanup in secondary mode) is enough to break the primary process.

    Affected DPDK / F-Stack versions

    Version Affected
    DPDK 22.11.x (all point releases) ✅ Yes
    DPDK 23.11.x incl. F-Stack bundled 23.11.5 ✅ Yes
    DPDK 24.11.x (up to 24.11.4) ✅ Yes
    DPDK 25.07+ ✅ Fixed

    Upstream fix

    Upstream DPDK commit 4bc53f8f0d64 — "eal: fix MP socket cleanup" (2025-07-19, merged for DPDK 25.07) by Yang Ming (Nokia).

    The fix reorders eal_bus_cleanup() before rte_mp_channel_cleanup() and adds per-driver secondary-process guards.

    Fix for F-Stack bundled DPDK 23.11.5

    PR #1050 backports the fix using the simpler guard approach:

    // lib/eal/linux/eal.c  rte_eal_cleanup()
    - eal_bus_cleanup();
    + if (rte_eal_process_type() == RTE_PROC_PRIMARY)
    +     eal_bus_cleanup();

    PR: #1050

  4. jfb8856606 commented on Mar 18, 2026

    @jfb8856606
    Contributor

    Fixed by PR #1050, now merged into dev branch.

  5. added a commit that references this issue on Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions