github-starred/nvitop

Watch

Fork 0

mirror of https://github.com/XuehaiPan/nvitop.git synced 2026-10-23 08:07:34 -06:00

[GH-ISSUE #222] [BUG] Intermittent segfault on quit: atexit nvmlShutdown() races with un-joined snapshot daemon threads #216

New issue

Closed

opened 2026-10-02 23:14:14 -06:00 by gitea-mirror · 1 comment

gitea-mirror commented

2026-10-02 23:14:14 -06:00

Owner

Originally created by @zengchang233 on GitHub (Jul 9, 2026).
Original GitHub issue: https://github.com/XuehaiPan/nvitop/issues/222

Originally assigned to: @XuehaiPan on GitHub.

Required prerequisites

I have read the documentation https://nvitop.readthedocs.io.
I have searched the Issue Tracker and Discussions that this hasn't already been reported. (+1 or comment there if it has.)
Consider asking first in a Discussion.

What version of nvitop are you using?

1.7.0

Operating system and environment

OS: Ubuntu 24.04.3 LTS (x86_64)
Python: 3.14.6 (conda, Anaconda build)
nvitop: 1.7.0 (pip)
nvidia-ml-py: 13.595.45
NVIDIA driver: 580.126.20
GPUs: 8x B300 SXM6 AC

Problem description

Pressing q to quit the TUI intermittently crashes with:

[1]    2774 segmentation fault (core dumped)  nvitop

dmesg shows the faulting thread is one of the snapshot daemons, dying inside libc/libnvidia-ml with a near-NULL dereference:

device-snapshot[2842255]: segfault at 18 ip 00007f84e154506f sp 00007f84d8db4790 error 4 in libc.so.6[7f84e14ca000+188000]

Root cause analysis

nvitop.api.libnvml registers an atexit hook on first NVML use:

https://github.com/XuehaiPan/nvitop/blob/v1.7.0/nvitop/api/libnvml.py#L275-L276

    nvmlInit()
    _atexit.register(nvmlShutdown)

Meanwhile the TUI panels (DevicePanel, HostPanel, ProcessPanel) each run a snapshot daemon thread (device-snapshot-daemon, etc.). On quit, destroy() only calls self._daemon_running.clear() — the threads are never join()ed:

https://github.com/XuehaiPan/nvitop/blob/v1.7.0/nvitop/tui/screens/main/panels/device.py#L531-L533

    def destroy(self) -> None:
        super().destroy()
        self._daemon_running.clear()

A daemon thread sleeping in time.sleep(SNAPSHOT_INTERVAL) or mid-take_snapshots() doesn't observe the cleared event before the main thread reaches interpreter finalization. The atexit hook then calls nvmlShutdown() while the snapshot thread is still inside a libnvidia-ml call → use-after-free of NVML internal state → SIGSEGV. The race window is a few milliseconds, so the crash is intermittent (for me: roughly 1 in 3 quits on an otherwise idle 8-GPU box).

Steps to Reproduce

The race is hard to hit through the TUI deterministically, but the underlying mechanism reproduces reliably:

# nvml_race.py — simulates snapshot daemon threads + normal interpreter exit
import threading, time
from nvitop.api import Device  # registers atexit(nvmlShutdown) on first NVML use

devs = Device.all()

def hammer():
    while True:
        for d in devs:
            try:
                d.memory_used(); d.gpu_utilization(); d.power_usage(); d.temperature()
            except Exception:
                pass

for _ in range(4):
    threading.Thread(target=hammer, daemon=True).start()

time.sleep(0.7)
# normal exit -> atexit nvmlShutdown() races with in-flight NVML calls

for i in $(seq 1 15); do python nvml_race.py; echo "run $i: rc=$?"; done

Results on my machine: 6/15 runs exit with rc=139 (SIGSEGV).

Adding atexit.unregister(nvitop.api.libnvml.nvmlShutdown) before exit: 0/15 crashes (15/15 clean).

Commenting out the _atexit.register(nvmlShutdown) line in libnvml.py also fixes quitting the real TUI (verified with repeated pty-driven nvitop → q cycles).

Expected Behavior

nvitop exits cleanly on q without a segmentation fault.

Additional context

Possible fixes, in order of decreasing safety:

Drop the atexit nvmlShutdown entirely. The OS reclaims NVML resources at process exit; an explicit shutdown at interpreter finalization buys nothing but this race.
Join the snapshot threads in destroy() (requires interruptible sleep, e.g. Event.wait(interval) instead of time.sleep(interval), so join is bounded).
Guard NVML calls and nvmlShutdown() with a shared lock — likely too coarse given snapshot frequency.

Happy to send a PR for option 1 or 2 if you have a preference.

Originally created by @zengchang233 on GitHub (Jul 9, 2026). Original GitHub issue: https://github.com/XuehaiPan/nvitop/issues/222 Originally assigned to: @XuehaiPan on GitHub. ### Required prerequisites - [x] I have read the documentation <https://nvitop.readthedocs.io>. - [x] I have searched the [Issue Tracker](https://github.com/XuehaiPan/nvitop/issues) and [Discussions](https://github.com/XuehaiPan/nvitop/discussions) that this hasn't already been reported. (+1 or comment there if it has.) - [x] Consider asking first in a [Discussion](https://github.com/XuehaiPan/nvitop/discussions). ### What version of nvitop are you using? 1.7.0 ### Operating system and environment - OS: Ubuntu 24.04.3 LTS (x86_64) - Python: 3.14.6 (conda, Anaconda build) - nvitop: 1.7.0 (pip) - nvidia-ml-py: 13.595.45 - NVIDIA driver: 580.126.20 - GPUs: 8x B300 SXM6 AC ### Problem description Pressing <kbd>q</kbd> to quit the TUI intermittently crashes with: ``` [1] 2774 segmentation fault (core dumped) nvitop ``` `dmesg` shows the faulting thread is one of the snapshot daemons, dying inside libc/libnvidia-ml with a near-NULL dereference: ``` device-snapshot[2842255]: segfault at 18 ip 00007f84e154506f sp 00007f84d8db4790 error 4 in libc.so.6[7f84e14ca000+188000] ``` #### Root cause analysis `nvitop.api.libnvml` registers an atexit hook on first NVML use: https://github.com/XuehaiPan/nvitop/blob/v1.7.0/nvitop/api/libnvml.py#L275-L276 ```python nvmlInit() _atexit.register(nvmlShutdown) ``` Meanwhile the TUI panels (`DevicePanel`, `HostPanel`, `ProcessPanel`) each run a snapshot **daemon** thread (`device-snapshot-daemon`, etc.). On quit, `destroy()` only calls `self._daemon_running.clear()` — the threads are never `join()`ed: https://github.com/XuehaiPan/nvitop/blob/v1.7.0/nvitop/tui/screens/main/panels/device.py#L531-L533 ```python def destroy(self) -> None: super().destroy() self._daemon_running.clear() ``` A daemon thread sleeping in `time.sleep(SNAPSHOT_INTERVAL)` or mid-`take_snapshots()` doesn't observe the cleared event before the main thread reaches interpreter finalization. The atexit hook then calls `nvmlShutdown()` while the snapshot thread is still inside a `libnvidia-ml` call → use-after-free of NVML internal state → SIGSEGV. The race window is a few milliseconds, so the crash is intermittent (for me: roughly 1 in 3 quits on an otherwise idle 8-GPU box). ### Steps to Reproduce The race is hard to hit through the TUI deterministically, but the underlying mechanism reproduces reliably: ```python # nvml_race.py — simulates snapshot daemon threads + normal interpreter exit import threading, time from nvitop.api import Device # registers atexit(nvmlShutdown) on first NVML use devs = Device.all() def hammer(): while True: for d in devs: try: d.memory_used(); d.gpu_utilization(); d.power_usage(); d.temperature() except Exception: pass for _ in range(4): threading.Thread(target=hammer, daemon=True).start() time.sleep(0.7) # normal exit -> atexit nvmlShutdown() races with in-flight NVML calls ``` ```bash for i in $(seq 1 15); do python nvml_race.py; echo "run $i: rc=$?"; done ``` Results on my machine: **6/15 runs exit with rc=139 (SIGSEGV)**. Adding `atexit.unregister(nvitop.api.libnvml.nvmlShutdown)` before exit: **0/15 crashes** (15/15 clean). Commenting out the `_atexit.register(nvmlShutdown)` line in `libnvml.py` also fixes quitting the real TUI (verified with repeated pty-driven `nvitop` → `q` cycles). ### Expected Behavior `nvitop` exits cleanly on <kbd>q</kbd> without a segmentation fault. ### Additional context Possible fixes, in order of decreasing safety: 1. **Drop the atexit `nvmlShutdown` entirely.** The OS reclaims NVML resources at process exit; an explicit shutdown at interpreter finalization buys nothing but this race. 2. Join the snapshot threads in `destroy()` (requires interruptible sleep, e.g. `Event.wait(interval)` instead of `time.sleep(interval)`, so join is bounded). 3. Guard NVML calls and `nvmlShutdown()` with a shared lock — likely too coarse given snapshot frequency. Happy to send a PR for option 1 or 2 if you have a preference.

gitea-mirror

2026-10-02 23:14:14 -06:00

closed this issue
added the
pynvml

api

bug
labels

gitea-mirror commented

2026-10-02 23:14:15 -06:00

Author

Owner

@XuehaiPan commented on GitHub (Jul 10, 2026):

Hi @zengchang233, thanks for the detailed report and spot-on diagnosis. The cause is that atexit nvmlShutdown() freed NVML state while a background thread was still mid-libnvidia-ml call (use-after-free).

I have fixed this in PR #223: nvmlQuery drains in-flight NVML calls before the atexit shutdown, so NVML is never torn down under a live call. Verified with your nvml_race.py on a 4×H100 host — main ~15/25 segfaults, fixed 0/25.

It'll ship in the next release; you can test against main now. Thanks for the clean reproducer!

@XuehaiPan commented on GitHub (Jul 10, 2026): Hi @zengchang233, thanks for the detailed report and spot-on diagnosis. The cause is that `atexit` `nvmlShutdown()` freed NVML state while a background thread was still mid-`libnvidia-ml` call (use-after-free). I have fixed this in PR #223: `nvmlQuery` drains in-flight NVML calls before the `atexit` shutdown, so NVML is never torn down under a live call. Verified with your `nvml_race.py` on a 4×H100 host — `main` ~15/25 segfaults, fixed 0/25. It'll ship in the next release; you can test against `main` now. Thanks for the clean reproducer!

gitea-mirror referenced this issue

2026-10-02 23:15:06 -06:00

[PR #216] [MERGED] feat(examples): add examples/ folder and relocate the deprecated nvitop.callbacks #219