Perftest: use memfd for hugepage host memory

[Hong-L666]

Fixes #395

This changes the --use_hugepages host allocation path from SysV shm to a hugetlb memfd mapping.

The current code uses shmget(SHM_HUGETLB) for hugepage-backed host memory. That is fine for the normal path, where perftest allocates one large buffer and shares it between QPs.

With --mr_per_qp, perftest allocates one host buffer per QP. If --use_hugepages is also set, each buffer allocation creates one SysV shm segment. With a large QP count this can hit kernel.shmmni, even when the hugepage pool still has enough free pages.

Using memfd_create(MFD_HUGETLB) keeps the allocation backed by hugetlb pages, but avoids consuming one SysV shm id per buffer. The memfd is closed after mmap() succeeds, and the mapping is released with munmap().

Testing:

• git diff --check
• ./autogen.sh
• ./configure
• make

Runtime test:

Server:

ib_write_bw -d hrn5_0 -s 2 -Q 1 -q 4096 --use_hugepages --run --mr_per_qp

Client:

ib_write_bw -d hrn5_2 -s 2 -Q 1 -q 4096 --use_hugepages --run --mr_per_qp 192.168.100.1

Result:

The test passed with this patch. The same command failed before this patch
when the SysV shm segment count reached kernel.shmmni.

[Hong-L666]

Hi @sshaulnv , friendly ping on this PR when you have a chance.

This fixes the kernel.shmmni exhaustion that occurs when --use_hugepages
is combined with --mr_per_qp. With --mr_per_qp, perftest allocates and
registers a separate host buffer for each QP. The existing hugepage
allocation path creates one SysV shared-memory segment for each buffer,
so a large number of QPs can exhaust kernel.shmmni. This patch replaces
the SysV shm-based hugepage allocation with hugetlb-backed memfd mappings.

I tested it with 4096 QPs using ib_write_bw; the test passed with this patch, while the same setup failed before the change.

I also noticed that the two GitHub Actions workflows are currently marked as “action required”. Could you please approve the workflows and take a look at the PR when convenient?

I’m happy to add more tests, clarify the kernel compatibility or fallback behavior, or revise the implementation based on your feedback. Thanks!

[sshaulnv]

The old shmget(SHM_HUGETLB) call used whatever huge page size the system defaults to, but the new memfd_create(MFD_HUGETLB | MFD_HUGE_2MB) hard-selects the 2 MB pool specifically - so on any system where Hugepagesize isn't 2048 kB (default_hugepagesz=1G, or 64 K-granule arm64 where it's 512 MB), --use_hugepages now fails against an empty 2 MB pool even though the user's configured huge pages are sitting right there:

Failed to mmap hugepage memfd (errno=12: Cannot allocate memory)
Failed to allocate hugepage region.
failed to create mr
Failed to create MR
 Couldn't create IB resources

[Hong-L666]

Thanks for catching this. You're right—the explicit MFD_HUGE_2MB changes the existing behavior by selecting the 2 MB pool instead of the system's default HugeTLB pool.
I've updated the implementation to use MFD_HUGETLB and align the mapping to the actual HugeTLB page size reported by the memfd. I'll run the build and runtime tests tomorrow and follow up here with the results.

[Hong-L666]

I have completed the compilation and traffic testing. Could you please review it further and let me know if there are any areas that need improvement?

[Hong-L666]

Hi @sshaulnv , gentle follow-up on the updated patch when you have a chance. The review feedback has been addressed, and the build and runtime tests have passed.

Please let me know if any further changes are needed. Thanks!

[sshaulnv]

Hi @Hong-L666 , sorry for the delay, will review this week.