Hi Team,
We are seeing crashes when running libisyntax on a shared multi-user Linux server.
After some debugging, this seems to be related to temporary files created under /dev/shm during libisyntax initialization. These files use fixed global names:
/dev/shm/sem.worksem
/dev/shm/sem.completionsem
/dev/shm/sem.benaphore2
Under normal execution, these files appear to be cleaned up correctly. However, if a job is terminated unexpectedly, the files remain on the system and stay owned by the user who ran the job.
When another user later runs libisyntax, libisyntax tries to reuse these existing files. In this situation, the user who originally created the files can run successfully, while other users encounter crashes during initialization.
We now rely on a small LD_PRELOAD wrapper that intercepts sem_open() and remaps only these three semaphore names to per-process unique names:
/worksem → /worksem_u_p
/completionsem → /completionsem_u_p
/benaphore2 → /benaphore2_u_p
This workaround worked for us right now. However, we’re not sure whether similar failures could still happen in the future (e.g., if multiple users run libisyntax at the same time or if jobs crash again). Ideally, we’d like a smooth, built-in fix in libisyntax itself, so it fails gracefully and handles cleanup properly.
Thank you and we‘re happy to share any details that you may need!
Hi Team,
We are seeing crashes when running libisyntax on a shared multi-user Linux server.
After some debugging, this seems to be related to temporary files created under /dev/shm during libisyntax initialization. These files use fixed global names:
/dev/shm/sem.worksem
/dev/shm/sem.completionsem
/dev/shm/sem.benaphore2
Under normal execution, these files appear to be cleaned up correctly. However, if a job is terminated unexpectedly, the files remain on the system and stay owned by the user who ran the job.
When another user later runs libisyntax, libisyntax tries to reuse these existing files. In this situation, the user who originally created the files can run successfully, while other users encounter crashes during initialization.
We now rely on a small LD_PRELOAD wrapper that intercepts
sem_open()and remaps only these three semaphore names to per-process unique names:/worksem → /worksem_u_p
/completionsem → /completionsem_u_p
/benaphore2 → /benaphore2_u_p
This workaround worked for us right now. However, we’re not sure whether similar failures could still happen in the future (e.g., if multiple users run libisyntax at the same time or if jobs crash again). Ideally, we’d like a smooth, built-in fix in libisyntax itself, so it fails gracefully and handles cleanup properly.
Thank you and we‘re happy to share any details that you may need!