[GH-ISSUE #217] [Feature Request] Configurable timebase for utilization history graphs #215

New issue

Closed

opened 2026-10-02 23:14:14 -06:00 by gitea-mirror · 1 comment

gitea-mirror commented

2026-10-02 23:14:14 -06:00

Owner

Originally created by @maxine-at-forecast on GitHub (May 19, 2026).
Original GitHub issue: https://github.com/XuehaiPan/nvitop/issues/217

Required prerequisites

I have searched the Issue Tracker that this hasn't already been reported. (comment there if it has.)
I have tried the latest version of nvitop in a new isolated virtual environment.

Motivation

First — thank you for nvitop! It's been the daily-driver GPU monitor on every box I touch, and the metrics screen in particular is wonderful.

I'd love a way to widen the time window shown in the utilization / memory history graphs (on the metrics screen, Enter on a process). When watching a long-running training or inference job, the current ~1s/column resolution means the visible window is only as wide as the terminal — usually a minute or two. It's great for spotting kernel-level stutter but makes it hard to eyeball longer-term trends ("did util drop off 5 minutes ago when the dataloader switched shards?") without resizing the terminal absurdly wide.

This is distinct from the existing --interval flag, which (correctly) controls the process snapshot rate. I was initially confused and tried nvitop --interval 20 expecting wider columns, but of course that flag governs the process table refresh, not the graph aggregation.

Looking at the source, the four graphs on the metrics screen all hardcode interval=1.0:

https://github.com/XuehaiPan/nvitop/blob/main/nvitop/tui/screens/metrics.py — the four BufferedHistoryGraph(interval=1.0, ...) instantiations (cpu_percent, used_host_memory, used_gpu_memory, gpu_sm_utilization).

And BufferedHistoryGraph already supports a configurable interval for buffering — averages get folded into one column every interval seconds:

https://github.com/XuehaiPan/nvitop/blob/main/nvitop/tui/library/history.py

So the underlying machinery is already in place; the graphs just need a way to plumb a non-default interval.

I want to be clear that I understand why the NVML sample rate should stay fast — you covered this thoroughly in #65, and the graphs need fresh data to feel responsive. This request is only about the display aggregation layer: keep sampling at ~1Hz, but average N samples into each column so the same number of columns covers N× the wall-clock window. No change to NVML query frequency.

Solution

A CLI flag (or M-screen keybind) to set the graph column width in seconds. Something like:

--graph-interval SEC   Seconds of data aggregated per column in the metrics
                       screen history graphs (default: 1.0). Higher values
                       widen the visible time window at the cost of
                       intra-column resolution. Does not affect NVML
                       sampling rate.

Implementation sketch: thread the value from MainScreen / CLI args into the four BufferedHistoryGraph(interval=..., ...) calls in metrics.py. The buffering logic in history.py already handles averaging across the wider interval, so this looks like it's mostly plumbing.

Happy to take a stab at a PR if you'd be open to it — wanted to check on the design (CLI flag vs. interactive keybind vs. both) before writing code.

Alternatives

Resize the terminal wider. Works but caps out at monitor width, and is awkward when you're SSH'd into a remote box with a fixed pty size.
Patch metrics.py locally (sed -i 's/interval=1.0/interval=5.0/g' in the venv). Fine as a one-off but doesn't survive pip install --upgrade.
Run an external metric collector (Prometheus / Grafana / ResourceMetricCollector). The Grafana dashboard you mentioned in #117 is great for long-term, but it's a heavier setup than I want when I just need "show me the last 10 minutes in the terminal I already have open."

Additional context

Use case: I'm running a 24h ML training kickoff and use the metrics screen to spot regressions (memory leaks, dataloader stalls, kernel-launch gaps). Right now I have to either keep an eye on it continuously or maximize the terminal. A 5–10s column would let a normal-width terminal show 5–10 minutes of history, which is the sweet spot for catching most issues.

Disclosure: This issue was researched, drafted, and filed by Claude (Anthropic's AI assistant) on behalf of a user. I reviewed the source paths cited above before posting and tried not to waste your time — but happy to clarify or rework anything if I've misunderstood the architecture.

Originally created by @maxine-at-forecast on GitHub (May 19, 2026). Original GitHub issue: https://github.com/XuehaiPan/nvitop/issues/217 ### Required prerequisites - [X] I have searched the [Issue Tracker](https://github.com/XuehaiPan/nvitop/issues) that this hasn't already been reported. (comment there if it has.) - [X] I have tried the latest version of nvitop in a new isolated virtual environment. ### Motivation First — thank you for `nvitop`! It's been the daily-driver GPU monitor on every box I touch, and the metrics screen in particular is wonderful. I'd love a way to **widen the time window** shown in the utilization / memory history graphs (on the metrics screen, `Enter` on a process). When watching a long-running training or inference job, the current ~1s/column resolution means the visible window is only as wide as the terminal — usually a minute or two. It's great for spotting kernel-level stutter but makes it hard to eyeball longer-term trends ("did util drop off 5 minutes ago when the dataloader switched shards?") without resizing the terminal absurdly wide. This is distinct from the existing `--interval` flag, which (correctly) controls the *process snapshot* rate. I was initially confused and tried `nvitop --interval 20` expecting wider columns, but of course that flag governs the process table refresh, not the graph aggregation. Looking at the source, the four graphs on the metrics screen all hardcode `interval=1.0`: https://github.com/XuehaiPan/nvitop/blob/main/nvitop/tui/screens/metrics.py — the four `BufferedHistoryGraph(interval=1.0, ...)` instantiations (`cpu_percent`, `used_host_memory`, `used_gpu_memory`, `gpu_sm_utilization`). And `BufferedHistoryGraph` already supports a configurable `interval` for buffering — averages get folded into one column every `interval` seconds: https://github.com/XuehaiPan/nvitop/blob/main/nvitop/tui/library/history.py So the underlying machinery is already in place; the graphs just need a way to plumb a non-default `interval`. I want to be clear that I understand why the **NVML sample rate** should stay fast — you covered this thoroughly in #65, and the graphs need fresh data to feel responsive. This request is only about the **display aggregation** layer: keep sampling at ~1Hz, but average N samples into each column so the same number of columns covers N× the wall-clock window. No change to NVML query frequency. ### Solution A CLI flag (or `M`-screen keybind) to set the graph column width in seconds. Something like: ``` --graph-interval SEC Seconds of data aggregated per column in the metrics screen history graphs (default: 1.0). Higher values widen the visible time window at the cost of intra-column resolution. Does not affect NVML sampling rate. ``` Implementation sketch: thread the value from `MainScreen` / CLI args into the four `BufferedHistoryGraph(interval=..., ...)` calls in `metrics.py`. The buffering logic in `history.py` already handles averaging across the wider interval, so this looks like it's mostly plumbing. Happy to take a stab at a PR if you'd be open to it — wanted to check on the design (CLI flag vs. interactive keybind vs. both) before writing code. ### Alternatives - **Resize the terminal wider.** Works but caps out at monitor width, and is awkward when you're SSH'd into a remote box with a fixed pty size. - **Patch `metrics.py` locally** (`sed -i 's/interval=1.0/interval=5.0/g'` in the venv). Fine as a one-off but doesn't survive `pip install --upgrade`. - **Run an external metric collector** (Prometheus / Grafana / `ResourceMetricCollector`). The Grafana dashboard you mentioned in #117 is great for long-term, but it's a heavier setup than I want when I just need "show me the last 10 minutes in the terminal I already have open." ### Additional context Use case: I'm running a 24h ML training kickoff and use the metrics screen to spot regressions (memory leaks, dataloader stalls, kernel-launch gaps). Right now I have to either keep an eye on it continuously or maximize the terminal. A 5–10s column would let a normal-width terminal show 5–10 minutes of history, which is the sweet spot for catching most issues. --- *Disclosure: This issue was researched, drafted, and filed by [Claude](https://www.anthropic.com/claude) (Anthropic's AI assistant) on behalf of a user. I reviewed the source paths cited above before posting and tried not to waste your time — but happy to clarify or rework anything if I've misunderstood the architecture.*

gitea-mirror

2026-10-02 23:14:14 -06:00

closed this issue
added the
cli / tui

enhancement
labels

gitea-mirror commented

2026-10-02 23:14:15 -06:00

Author

Owner

@XuehaiPan commented on GitHub (May 20, 2026):

Hi @maxine-at-forecast, hardcoding the interval to 1.0s is intentional to make each bin on the graph represent 1 second. Making the interval configurable will make it hard to calculate the tick labels of the x-axis (e.g., passing interval=3.0s). Also, the collected metrics will be pushed and averaged in the buffer. So you will never get the original raw data if you want to change the interval (i.e., the interactive keybinding solution).

A typical use case is that when the user sees something in a coarse-grained graph with a large interval, they will ultimately want to zoom in to a fine-grained graph to figure out what is going on. Making the graph interval configurable cannot fulfill that need.

Run an external metric collector (Prometheus / Grafana / ResourceMetricCollector). The Grafana dashboard you mentioned in #117 is great for long-term, but it's a heavier setup than I want when I just need "show me the last 10 minutes in the terminal I already have open."

I opened another solution in:

#218

I wonder if it resolves your request.

@XuehaiPan commented on GitHub (May 20, 2026): Hi @maxine-at-forecast, hardcoding the interval to 1.0s is intentional to make each bin on the graph represent 1 second. Making the interval configurable will make it hard to calculate the tick labels of the x-axis (e.g., passing `interval=3.0s`). Also, the collected metrics will be pushed and averaged in the buffer. So you will never get the original raw data if you want to change the interval (i.e., the interactive keybinding solution). A typical use case is that when the user sees something in a coarse-grained graph with a large interval, they will ultimately want to zoom in to a fine-grained graph to figure out what is going on. Making the graph interval configurable cannot fulfill that need. --- > * **Run an external metric collector** (Prometheus / Grafana / `ResourceMetricCollector`). The Grafana dashboard you mentioned in #117 is great for long-term, but it's a heavier setup than I want when I just need "show me the last 10 minutes in the terminal I already have open." I opened another solution in: - #218 I wonder if it resolves your request.