[GH-ISSUE #95] [Question] How to count the total amount of one user's GPU memory on all graphics cards by using nvitop? #55

New issue

Closed

opened 2026-05-05 03:23:54 -06:00 by gitea-mirror · 2 comments

gitea-mirror commented

2026-05-05 03:23:54 -06:00

Owner

Originally created by @Charlie0257 on GitHub (Sep 1, 2023).
Original GitHub issue: https://github.com/XuehaiPan/nvitop/issues/95

Originally assigned to: @XuehaiPan on GitHub.

Required prerequisites

I have read the documentation https://nvitop.readthedocs.io.
I have searched the Issue Tracker that this hasn't already been reported. (comment there if it has.)
I have tried the latest version of nvitop in a new isolated virtual environment.

Questions

Thanks for the wonderful project!

How to count the total amount of one user's GPU memory on all graphics cards by using nvitop?

Thanks for any suggestions! :) @XuehaiPan

Originally created by @Charlie0257 on GitHub (Sep 1, 2023). Original GitHub issue: https://github.com/XuehaiPan/nvitop/issues/95 Originally assigned to: @XuehaiPan on GitHub. ### Required prerequisites - [X] I have read the documentation <https://nvitop.readthedocs.io>. - [X] I have searched the [Issue Tracker](https://github.com/XuehaiPan/nvitop/issues) that this hasn't already been reported. (comment there if it has.) - [X] I have tried the latest version of nvitop in a new isolated virtual environment. ### Questions Thanks for the wonderful project! How to count the total amount of one user's GPU memory on all graphics cards by using nvitop? Thanks for any suggestions! :) @XuehaiPan

gitea-mirror

2026-05-05 03:23:54 -06:00

closed this issue
added the
api

question
labels

gitea-mirror commented

2026-05-05 03:23:57 -06:00

Author

Owner

@XuehaiPan commented on GitHub (Sep 1, 2023):

Hi @Charlie0257, thanks for raising this. Here is a code snippet:

# memory_usage.py

import itertools
from collections import defaultdict

from nvitop import Device, GpuProcess, NA, bytes2human


devices = Device.all()  # a list of all GPUs on the system
all_gpu_processes = list(  # a list of all GPU processes on all devices
    itertools.chain.from_iterable(
        (device.processes().values() for device in devices),
    ),
)

used_gpu_memory = defaultdict(int)
used_host_memory = defaultdict(int)
with GpuProcess.failsafe():  # ignore NoSuchProcess and AccessDenied exceptions
    for process in all_gpu_processes:
        username = process.username()
        if username is NA:
            continue  # the process is gone

        # Here we use int(memory) to convert N/A value to 0
        used_gpu_memory[username] += int(process.gpu_memory())
        used_host_memory[username] += int(process.host_memory())

print('used_gpu_memory:', dict(used_gpu_memory))  # dict of {username: used_gpu_memory in bytes}
print('used_host_memory:', dict(used_host_memory))  # dict of {username: used_host_memory in bytes}
print()

print(
    'used_gpu_memory:',
    {username: bytes2human(memory) for username, memory in used_gpu_memory.items()},
)
print(
    'used_host_memory:',
    {username: bytes2human(memory) for username, memory in used_host_memory.items()},
)

Here is an example output:

$ python3 memory_usage.py
used_gpu_memory: {'me': 509974937600}
used_host_memory: {'me': 334980202496}

used_gpu_memory: {'me': '475.0GiB'}
used_host_memory: {'me': '312.0GiB'}

Hope this resolves your question.

@XuehaiPan commented on GitHub (Sep 1, 2023): Hi @Charlie0257, thanks for raising this. Here is a code snippet: ```python # memory_usage.py import itertools from collections import defaultdict from nvitop import Device, GpuProcess, NA, bytes2human devices = Device.all() # a list of all GPUs on the system all_gpu_processes = list( # a list of all GPU processes on all devices itertools.chain.from_iterable( (device.processes().values() for device in devices), ), ) used_gpu_memory = defaultdict(int) used_host_memory = defaultdict(int) with GpuProcess.failsafe(): # ignore NoSuchProcess and AccessDenied exceptions for process in all_gpu_processes: username = process.username() if username is NA: continue # the process is gone # Here we use int(memory) to convert N/A value to 0 used_gpu_memory[username] += int(process.gpu_memory()) used_host_memory[username] += int(process.host_memory()) print('used_gpu_memory:', dict(used_gpu_memory)) # dict of {username: used_gpu_memory in bytes} print('used_host_memory:', dict(used_host_memory)) # dict of {username: used_host_memory in bytes} print() print( 'used_gpu_memory:', {username: bytes2human(memory) for username, memory in used_gpu_memory.items()}, ) print( 'used_host_memory:', {username: bytes2human(memory) for username, memory in used_host_memory.items()}, ) ``` Here is an example output: ```console $ python3 memory_usage.py used_gpu_memory: {'me': 509974937600} used_host_memory: {'me': 334980202496} used_gpu_memory: {'me': '475.0GiB'} used_host_memory: {'me': '312.0GiB'} ``` Hope this resolves your question.

gitea-mirror commented

2026-05-05 03:23:58 -06:00

Author

Owner

@Charlie0257 commented on GitHub (Sep 1, 2023):

Thanks for this answer! :)

I will close this issue.

@Charlie0257 commented on GitHub (Sep 1, 2023): Thanks for this answer! :) I will close this issue.