github-starred/nvitop

Watch

Fork 0

mirror of https://github.com/XuehaiPan/nvitop.git synced 2026-10-23 08:07:34 -06:00

[GH-ISSUE #221] [BUG] GPU clock frequency is masked by the UTL bar #214

New issue

Closed

opened 2026-10-02 23:14:14 -06:00 by gitea-mirror · 2 comments

gitea-mirror commented

2026-10-02 23:14:14 -06:00

Owner

Originally created by @FlorinAndrei on GitHub (Jun 23, 2026).
Original GitHub issue: https://github.com/XuehaiPan/nvitop/issues/221

Originally assigned to: @XuehaiPan on GitHub.

Required prerequisites

I have read the documentation https://nvitop.readthedocs.io.
I have searched the Issue Tracker that this hasn't already been reported. (comment there if it has.)
I have tried the latest version of nvitop in a new isolated virtual environment.

What version of nvitop are you using?

1.7.0

Operating system and version

Ubuntu 24.04

NVIDIA driver version

580.159.03

NVIDIA-SMI

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.159.03             Driver Version: 580.159.03     CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GB10                    Off |   0000000F:01:00.0 Off |                  N/A |
| N/A   41C    P0             10W /  N/A  | Not Supported          |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A            2724      G   /usr/lib/xorg/Xorg                       18MiB |
|    0   N/A  N/A            4107      C   /app/llama-server                       170MiB |
|    0   N/A  N/A            4797      G   /usr/bin/gnome-shell                      6MiB |
|    0   N/A  N/A           35172      C   /app/llama-server                     64965MiB |
+-----------------------------------------------------------------------------------------+

Python environment

3.13.14 (main, Jun 11 2026, 12:30:59) [GCC 13.3.0] linux
nvidia-ml-py==13.580.82
nvitop==1.7.0

Problem description

The GPU clock frequency indicator shares a row with the UTL bar.

Most of the time, I can read the GPU clock frequency just fine. But when UTL is high, it covers the GPU clock frequency, and I cannot read that anymore.

GPU clock frequency is an important metric. It should never be hidden.

Steps to Reproduce

Run nvitop

Create a task that raises GPU utilization close to 100%.

Watch the GPU clock indicator.

Traceback

Logs

Expected behavior

GPU clock frequency should never be masked.

Additional context

My system has a bug specifically related to the GPU clock. Troubleshooting the bug, I would like to use nvitop, since it provides so much information. But I cannot use nvitop because I cannot read the most important metric in this case.

Originally created by @FlorinAndrei on GitHub (Jun 23, 2026). Original GitHub issue: https://github.com/XuehaiPan/nvitop/issues/221 Originally assigned to: @XuehaiPan on GitHub. ### Required prerequisites - [x] I have read the documentation <https://nvitop.readthedocs.io>. - [x] I have searched the [Issue Tracker](https://github.com/XuehaiPan/nvitop/issues) that this hasn't already been reported. (comment there if it has.) - [x] I have tried the latest version of nvitop in a new isolated virtual environment. ### What version of nvitop are you using? 1.7.0 ### Operating system and version Ubuntu 24.04 ### NVIDIA driver version 580.159.03 ### NVIDIA-SMI ```text +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 580.159.03 Driver Version: 580.159.03 CUDA Version: 13.0 | +-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA GB10 Off | 0000000F:01:00.0 Off | N/A | | N/A 41C P0 10W / N/A | Not Supported | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | 0 N/A N/A 2724 G /usr/lib/xorg/Xorg 18MiB | | 0 N/A N/A 4107 C /app/llama-server 170MiB | | 0 N/A N/A 4797 G /usr/bin/gnome-shell 6MiB | | 0 N/A N/A 35172 C /app/llama-server 64965MiB | +-----------------------------------------------------------------------------------------+ ``` ### Python environment 3.13.14 (main, Jun 11 2026, 12:30:59) [GCC 13.3.0] linux nvidia-ml-py==13.580.82 nvitop==1.7.0 ### Problem description The GPU clock frequency indicator shares a row with the UTL bar. Most of the time, I can read the GPU clock frequency just fine. But when UTL is high, it covers the GPU clock frequency, and I cannot read that anymore. GPU clock frequency is an important metric. It should never be hidden. ### Steps to Reproduce Run nvitop Create a task that raises GPU utilization close to 100%. Watch the GPU clock indicator. ### Traceback ```pytb ``` ### Logs ```text ``` ### Expected behavior GPU clock frequency should never be masked. ### Additional context My system has a bug specifically related to the GPU clock. Troubleshooting the bug, I would like to use nvitop, since it provides so much information. But I cannot use nvitop because I cannot read the most important metric in this case.

gitea-mirror

2026-10-02 23:14:14 -06:00

closed this issue
added the
bug
label

gitea-mirror commented

2026-10-02 23:14:15 -06:00

Author

Owner

@XuehaiPan commented on GitHub (Jul 10, 2026):

Hi @FlorinAndrei, thanks for the report and sorry for the late reply. This is intentional rather than a bug. When the terminal isn't wide enough to show everything on a row, nvitop deliberately keeps the most relevant value instead of overflowing. Which value that is comes down to preference: most users want the utilization rate, while some prefer the clock speed, so the utilization rate takes that slot by default.

A few ways to see the clock:

Enlarge the terminal window — the simplest fix. With more width, nvitop has room to show the clock next to the utilization bar.
Use the monitor-colored example for an always-visible readout. It prints SM Clock and Memory Clock on their own rows that are never truncated, which is handy for troubleshooting clock behavior. Run it without cloning the repo:
```
uv run https://github.com/XuehaiPan/nvitop/raw/HEAD/examples/monitor-colored/monitor_colored.py
```
Export to Prometheus and build a custom dashboard — nvitop-exporter publishes the clocks (gpu_clock_sm, gpu_clock_memory, …) alongside every other metric, so you can put the clock front and center on a Grafana dashboard.
Build your own monitor on the nvitop API — nvitop is also a library: device.sm_clock(), device.memory_clock(), and friends return the raw values, so you can render exactly what you want (the monitor-colored example above is a good starting point). See the API reference.

Thanks again!

@XuehaiPan commented on GitHub (Jul 10, 2026): Hi @FlorinAndrei, thanks for the report and sorry for the late reply. This is intentional rather than a bug. When the terminal isn't wide enough to show everything on a row, `nvitop` deliberately keeps the most relevant value instead of overflowing. Which value that is comes down to preference: most users want the utilization rate, while some prefer the clock speed, so the utilization rate takes that slot by default. A few ways to see the clock: 1. **Enlarge the terminal window** — the simplest fix. With more width, `nvitop` has room to show the clock next to the utilization bar. 2. **Use the [`monitor-colored` example](https://github.com/XuehaiPan/nvitop/tree/main/examples/monitor-colored)** for an always-visible readout. It prints `SM Clock` and `Memory Clock` on their own rows that are never truncated, which is handy for troubleshooting clock behavior. Run it without cloning the repo: ```bash uv run https://github.com/XuehaiPan/nvitop/raw/HEAD/examples/monitor-colored/monitor_colored.py ``` 3. **Export to Prometheus and build a custom dashboard** — [`nvitop-exporter`](https://github.com/XuehaiPan/nvitop/tree/HEAD/nvitop-exporter) publishes the clocks (`gpu_clock_sm`, `gpu_clock_memory`, …) alongside every other metric, so you can put the clock front and center on a Grafana dashboard. 4. **Build your own monitor on the nvitop API** — `nvitop` is also a library: [`device.sm_clock()`](https://github.com/XuehaiPan/nvitop/issues/221#issuecomment-4934620037), [`device.memory_clock()`](https://nvitop.readthedocs.io/en/latest/api/device.html#nvitop.Device.memory_clock), and friends return the raw values, so you can render exactly what you want (the [`monitor-colored` example](https://github.com/XuehaiPan/nvitop/tree/main/examples/monitor-colored) above is a good starting point). See the [API reference](https://nvitop.readthedocs.io/en/latest/api/device.html). Thanks again!

gitea-mirror commented

2026-10-02 23:14:15 -06:00

Author

Owner

@FlorinAndrei commented on GitHub (Jul 10, 2026):

@XuehaiPan

I understand the reasoning. But just looking at this part:

Enlarge the terminal window — the simplest fix. With more width, nvitop has room to show the clock next to the utilization bar.

The bar covers the frequency even when running nvitop in a maximized terminal. The terminal uses the whole screen. And yet the bar still covers the numbers. I'm running nvitop in a maximized terminal on a MacBook Pro, the screen is very high resolution (Retina display), and yet the bars take over the whole space. This does not seem to serve a useful purpose.

Could we have a certain size threshold, above which the bar does not keep growing? We don't need an infinite bar, just one long enough. The fix seems simple: just cap the bar length.

Also, past a certain screen size, allowing the bars to get even bigger is not useful as a visual indicator.

This is true of all 4 bars, not just UTL.

@FlorinAndrei commented on GitHub (Jul 10, 2026): @XuehaiPan I understand the reasoning. But just looking at this part: > Enlarge the terminal window — the simplest fix. With more width, nvitop has room to show the clock next to the utilization bar. The bar covers the frequency even when running nvitop in a maximized terminal. The terminal uses the whole screen. And yet the bar still covers the numbers. I'm running nvitop in a maximized terminal on a MacBook Pro, the screen is very high resolution (Retina display), and yet the bars take over the whole space. This does not seem to serve a useful purpose. Could we have a certain size threshold, above which the bar does not keep growing? We don't need an infinite bar, just one long enough. The fix seems simple: just cap the bar length. Also, past a certain screen size, allowing the bars to get even bigger is not useful as a visual indicator. This is true of all 4 bars, not just UTL.

gitea-mirror referenced this issue

2026-10-02 23:15:05 -06:00

[PR #214] [MERGED] feat(cli): add option --readonly to CLI #218