github-starred/nvitop

Fork 0

mirror of https://github.com/XuehaiPan/nvitop.git synced 2026-05-15 14:15:55 -06:00

[GH-ISSUE #173] [BUG][upstream] used value of host memory is different from the result of `free` command #108

New issue

Closed

opened 2026-05-05 03:25:41 -06:00 by gitea-mirror · 3 comments

gitea-mirror commented

2026-05-05 03:25:41 -06:00

Owner

Originally created by @1saac-k on GitHub (Jul 25, 2025).
Original GitHub issue: https://github.com/XuehaiPan/nvitop/issues/173

Originally assigned to: @XuehaiPan on GitHub.

Required prerequisites

I have read the documentation https://nvitop.readthedocs.io.
I have searched the Issue Tracker that this hasn't already been reported. (comment there if it has.)
I have tried the latest version of nvitop in a new isolated virtual environment.

What version of nvitop are you using?

1.5.2.dev5 (the latest from source code)

Operating system and version

Ubuntu 24.04 LTS

NVIDIA driver version

575.51.03

NVIDIA-SMI

Python environment

3.10.16 (main, Feb 12 2025, 14:50:02) [Clang 19.1.6 ] linux
nvidia-ml-py==12.575.51
nvitop @ file:///home/<**>/ws/nvitop_origin

Problem description

The used value of CPU memory is incorrect due to a bug in psutil, which is a dependency of nvitop.

Refer to https://github.com/giampaolo/psutil/issues/2604 (reported by me)

Steps to Reproduce

Here, we will ignore the difference in the used/USED value.

(nvitop_origin) (Supermicro-EMR) ~/ws/nvitop_origin $ free -h -w
               total        used        free      shared     buffers       cache   available
Mem:           1.0Ti        24Gi       891Gi        13Mi       1.4Gi        96Gi       983Gi
Swap:          2.0Gi          0B       2.0Gi

(nvitop_origin) (Supermicro-EMR) ~/ws/nvitop_origin $ nvitop --once  | grep '^\[.*MEM'
[ MEM: ▊ 2.4%             USED: 18.21GiB ]  [ SWP: ▏ 0.0%                     ]

In other shell,

(LMCache) (Supermicro-EMR) ~/ws/LMCache $ python
Python 3.12.3 (main, Feb  4 2025, 14:48:35) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> buffer = torch.empty(16 * 1024**3, dtype=torch.uint8, pin_memory=True)
>>>

Then,

free command: used/shared/cache (it's not page cache) increases in free command
nvitop: percent (4.1%) is OK, but USED does not increase

(nvitop_origin) (Supermicro-EMR) ~/ws/nvitop_origin $ free -h -w
               total        used        free      shared     buffers       cache   available
Mem:           1.0Ti        40Gi       874Gi        16Gi       1.4Gi       112Gi       966Gi
Swap:          2.0Gi          0B       2.0Gi

(nvitop_origin) (Supermicro-EMR) ~/ws/nvitop_origin $ nvitop --once  | grep '^\[.*MEM'
[ MEM: █▎ 4.1%            USED: 18.85GiB ]  [ SWP: ▏ 0.0%                     ]

Traceback

Logs

Expected behavior

Increase USED as free command.

Additional context

It would be best if this is fixed in psutil, as it's a bug in psutil.

However, we can also apply a workaround patch in nvitop as follows:

diff --git a/nvitop-exporter/nvitop_exporter/exporter.py b/nvitop-exporter/nvitop_exporter/exporter.py
index 186759a..9f1d720 100644
--- a/nvitop-exporter/nvitop_exporter/exporter.py
+++ b/nvitop-exporter/nvitop_exporter/exporter.py
@@ -497,7 +497,7 @@ class PrometheusExporter:  # pylint: disable=too-many-instance-attributes
             (self.host_uptime, host.uptime()),
             (self.host_cpu_percent, host.cpu_percent()),
             (self.host_virtual_memory_total, virtual_memory.total / MiB),
-            (self.host_virtual_memory_used, virtual_memory.used / MiB),
+            (self.host_virtual_memory_used, (virtual_memory.total - virtual_memory.available) / MiB),
             (self.host_virtual_memory_free, virtual_memory.free / MiB),
             (self.host_virtual_memory_percent, virtual_memory.percent),
             (self.host_swap_memory_total, swap_memory.total / MiB),
diff --git a/nvitop/tui/screens/main/panels/host.py b/nvitop/tui/screens/main/panels/host.py
index de61296..f116f5a 100644
--- a/nvitop/tui/screens/main/panels/host.py
+++ b/nvitop/tui/screens/main/panels/host.py
@@ -298,7 +298,7 @@ class HostPanel(BasePanel):  # pylint: disable=too-many-instance-attributes
                     'MEM',
                     self.virtual_memory.percent,
                     width_left - 4,
-                    extra_text=f'  USED: {bytes2human(self.virtual_memory.used, min_unit=GiB)}',
+                    extra_text=f'  USED: {bytes2human((self.virtual_memory.total - self.virtual_memory.available), min_unit=GiB)}',
                 ),
             )
             swap_bar = '[ {} ]'.format(make_bar('SWP', self.swap_memory.percent, width_right - 4))

Below is result after run torch.empty(..)

Before patch: [ MEM: ██▌ 4.0% USED: 18.77GiB ]
After patch: [ MEM: ██▌ 4.0% USED: 40.76GiB ]

Originally created by @1saac-k on GitHub (Jul 25, 2025). Original GitHub issue: https://github.com/XuehaiPan/nvitop/issues/173 Originally assigned to: @XuehaiPan on GitHub. ### Required prerequisites - [x] I have read the documentation <https://nvitop.readthedocs.io>. - [x] I have searched the [Issue Tracker](https://github.com/XuehaiPan/nvitop/issues) that this hasn't already been reported. (comment there if it has.) - [x] I have tried the latest version of nvitop in a new isolated virtual environment. ### What version of nvitop are you using? 1.5.2.dev5 (the latest from source code) ### Operating system and version Ubuntu 24.04 LTS ### NVIDIA driver version 575.51.03 ### NVIDIA-SMI ```text ``` ### Python environment 3.10.16 (main, Feb 12 2025, 14:50:02) [Clang 19.1.6 ] linux nvidia-ml-py==12.575.51 nvitop @ file:///home/<**>/ws/nvitop_origin ### Problem description The `used` value of CPU memory is incorrect due to a bug in `psutil`, which is a dependency of `nvitop`. Refer to https://github.com/giampaolo/psutil/issues/2604 (reported by me) ### Steps to Reproduce Here, we will ignore the difference in the `used`/`USED` value. ``` bash (nvitop_origin) (Supermicro-EMR) ~/ws/nvitop_origin $ free -h -w total used free shared buffers cache available Mem: 1.0Ti 24Gi 891Gi 13Mi 1.4Gi 96Gi 983Gi Swap: 2.0Gi 0B 2.0Gi (nvitop_origin) (Supermicro-EMR) ~/ws/nvitop_origin $ nvitop --once | grep '^\[.*MEM' [ MEM: ▊ 2.4% USED: 18.21GiB ] [ SWP: ▏ 0.0% ] ``` In other shell, ``` bash (LMCache) (Supermicro-EMR) ~/ws/LMCache $ python Python 3.12.3 (main, Feb 4 2025, 14:48:35) [GCC 13.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import torch >>> buffer = torch.empty(16 * 1024**3, dtype=torch.uint8, pin_memory=True) >>> ``` Then, - free command: `used`/`shared`/`cache` (it's not page cache) increases in free command - nvitop: percent (4.1%) is OK, but `USED` does not increase ``` bash (nvitop_origin) (Supermicro-EMR) ~/ws/nvitop_origin $ free -h -w total used free shared buffers cache available Mem: 1.0Ti 40Gi 874Gi 16Gi 1.4Gi 112Gi 966Gi Swap: 2.0Gi 0B 2.0Gi (nvitop_origin) (Supermicro-EMR) ~/ws/nvitop_origin $ nvitop --once | grep '^\[.*MEM' [ MEM: █▎ 4.1% USED: 18.85GiB ] [ SWP: ▏ 0.0% ] ``` ### Traceback ```pytb ``` ### Logs ```text ``` ### Expected behavior Increase `USED` as free command. ### Additional context It would be best if this is fixed in `psutil`, as it's a bug in `psutil`. However, we can also apply a workaround patch in nvitop as follows: ``` bash diff --git a/nvitop-exporter/nvitop_exporter/exporter.py b/nvitop-exporter/nvitop_exporter/exporter.py index 186759a..9f1d720 100644 --- a/nvitop-exporter/nvitop_exporter/exporter.py +++ b/nvitop-exporter/nvitop_exporter/exporter.py @@ -497,7 +497,7 @@ class PrometheusExporter: # pylint: disable=too-many-instance-attributes (self.host_uptime, host.uptime()), (self.host_cpu_percent, host.cpu_percent()), (self.host_virtual_memory_total, virtual_memory.total / MiB), - (self.host_virtual_memory_used, virtual_memory.used / MiB), + (self.host_virtual_memory_used, (virtual_memory.total - virtual_memory.available) / MiB), (self.host_virtual_memory_free, virtual_memory.free / MiB), (self.host_virtual_memory_percent, virtual_memory.percent), (self.host_swap_memory_total, swap_memory.total / MiB), diff --git a/nvitop/tui/screens/main/panels/host.py b/nvitop/tui/screens/main/panels/host.py index de61296..f116f5a 100644 --- a/nvitop/tui/screens/main/panels/host.py +++ b/nvitop/tui/screens/main/panels/host.py @@ -298,7 +298,7 @@ class HostPanel(BasePanel): # pylint: disable=too-many-instance-attributes 'MEM', self.virtual_memory.percent, width_left - 4, - extra_text=f' USED: {bytes2human(self.virtual_memory.used, min_unit=GiB)}', + extra_text=f' USED: {bytes2human((self.virtual_memory.total - self.virtual_memory.available), min_unit=GiB)}', ), ) swap_bar = '[ {} ]'.format(make_bar('SWP', self.swap_memory.percent, width_right - 4)) ``` Below is result after run torch.empty(..) - Before patch: `[ MEM: ██▌ 4.0% USED: 18.77GiB ]` - After patch: `[ MEM: ██▌ 4.0% USED: 40.76GiB ]`

gitea-mirror

2026-05-05 03:25:41 -06:00

closed this issue
added the
bug

upstream
labels

gitea-mirror commented

2026-05-05 03:25:43 -06:00

Author

Owner

@XuehaiPan commented on GitHub (Jul 25, 2025):

Hi @kyet, thanks for filing the issue and also reporting it to the upstream. Let's wait for the response from the upstream maintainer.

@XuehaiPan commented on GitHub (Jul 25, 2025): Hi @kyet, thanks for filing the issue and also reporting it to the upstream. Let's wait for the response from the upstream maintainer.

gitea-mirror commented

2026-05-05 03:25:44 -06:00

Author

Owner

@1saac-k commented on GitHub (Aug 14, 2025):

@XuehaiPan My upstream PR has been merged. It will be included in psutil 7.1.0.

Let's wait for next release of psutil.

@1saac-k commented on GitHub (Aug 14, 2025): @XuehaiPan My upstream PR has been merged. It will be included in psutil 7.1.0. - https://github.com/giampaolo/psutil/pull/2612 - https://github.com/giampaolo/psutil/commit/9c7c46b413809814e477c2a255ca7017a7f43395 Let's wait for next release of psutil.

gitea-mirror commented

2026-05-05 03:25:45 -06:00

Author

Owner

@XuehaiPan commented on GitHub (Aug 16, 2025):

Thanks for the information and contribution! @kyet

@XuehaiPan commented on GitHub (Aug 16, 2025): Thanks for the information and contribution! @kyet