[PR #70] [MERGED] fix(api/device): further isolate the `CUDA_VISIBLE_DEVICE` parser in a subprocess #148

New issue

Closed

opened 2026-05-05 03:26:45 -06:00 by gitea-mirror · 0 comments

gitea-mirror commented

2026-05-05 03:26:45 -06:00

Owner

📋 Pull Request Information

Original PR: https://github.com/XuehaiPan/nvitop/pull/70
Author: @XuehaiPan
Created: 4/10/2023
Status: ✅ Merged
Merged: 4/11/2023
Merged by: @XuehaiPan

Base: main ← Head: spawn-subprocess

📝 Commits (2)

21e0ea0 fix(api/device): further isolate the CUDA_VISIBLE_DEVICE parser in a subprocess
12278cc docs(CHANGELOG): update CHANGELOG.md

📊 Changes

2 files changed (+30 additions, -3 deletions)

View changed files

📝 CHANGELOG.md (+1 -1)
📝 nvitop/api/device.py (+29 -2)

📄 Description

Issue Type

Bug fix

Description

Further isolate the CUDA_VISIBLE_DEVICE parser in a subprocess. We are using both subprocess and mutliprocessing modules.

Motivation and Context

Fixes #69

Testing

# test.py

from nvitop import CudaDevice

print(CudaDevice.count())

Before:

$ python3 test.py
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/linuxbrew/.linuxbrew/opt/python@3.11/lib/python3.11/multiprocessing/spawn.py", line 120, in spawn_main
    exitcode = _main(fd, parent_sentinel)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/linuxbrew/.linuxbrew/opt/python@3.11/lib/python3.11/multiprocessing/spawn.py", line 129, in _main
    prepare(preparation_data)
  File "/home/linuxbrew/.linuxbrew/opt/python@3.11/lib/python3.11/multiprocessing/spawn.py", line 240, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "/home/linuxbrew/.linuxbrew/opt/python@3.11/lib/python3.11/multiprocessing/spawn.py", line 291, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen runpy>", line 291, in run_path
  File "<frozen runpy>", line 98, in _run_module_code
  File "<frozen runpy>", line 88, in _run_code
  File "/home/PanXuehai/Projects/nvitop/test.py", line 5, in <module>
    print(CudaDevice.count())
          ^^^^^^^^^^^^^^^^^^
  File "/home/PanXuehai/Projects/nvitop/nvitop/api/device.py", line 2132, in count
    return len(super().parse_cuda_visible_devices())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/PanXuehai/Projects/nvitop/nvitop/api/device.py", line 488, in parse_cuda_visible_devices
    return parse_cuda_visible_devices(cuda_visible_devices)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/PanXuehai/Projects/nvitop/nvitop/api/device.py", line 2357, in parse_cuda_visible_devices
    return _parse_cuda_visible_devices(cuda_visible_devices, format='index')
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/PanXuehai/Projects/nvitop/nvitop/api/device.py", line 2491, in _parse_cuda_visible_devices
    raw_uuids = _parse_cuda_visible_devices_to_uuids(cuda_visible_devices, verbose=False)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/PanXuehai/Projects/nvitop/nvitop/api/device.py", line 2616, in _parse_cuda_visible_devices_to_uuids
    parser.start()
  File "/home/linuxbrew/.linuxbrew/opt/python@3.11/lib/python3.11/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^
  File "/home/linuxbrew/.linuxbrew/opt/python@3.11/lib/python3.11/multiprocessing/context.py", line 288, in _Popen
    return Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^
  File "/home/linuxbrew/.linuxbrew/opt/python@3.11/lib/python3.11/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/home/linuxbrew/.linuxbrew/opt/python@3.11/lib/python3.11/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/home/linuxbrew/.linuxbrew/opt/python@3.11/lib/python3.11/multiprocessing/popen_spawn_posix.py", line 42, in _launch
    prep_data = spawn.get_preparation_data(process_obj._name)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/linuxbrew/.linuxbrew/opt/python@3.11/lib/python3.11/multiprocessing/spawn.py", line 158, in get_preparation_data
    _check_not_importing_main()
  File "/home/linuxbrew/.linuxbrew/opt/python@3.11/lib/python3.11/multiprocessing/spawn.py", line 138, in _check_not_importing_main
    raise RuntimeError('''
RuntimeError: 
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.

After:

$ python3 test.py
1

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/XuehaiPan/nvitop/pull/70 **Author:** [@XuehaiPan](https://github.com/XuehaiPan) **Created:** 4/10/2023 **Status:** ✅ Merged **Merged:** 4/11/2023 **Merged by:** [@XuehaiPan](https://github.com/XuehaiPan) **Base:** `main` ← **Head:** `spawn-subprocess` --- ### 📝 Commits (2) - [`21e0ea0`](https://github.com/XuehaiPan/nvitop/commit/21e0ea00e28bb5e8a63e740330de3ab5762940a3) fix(api/device): further isolate the `CUDA_VISIBLE_DEVICE` parser in a subprocess - [`12278cc`](https://github.com/XuehaiPan/nvitop/commit/12278ccce39383f26330d17e524f42d59668fbea) docs(CHANGELOG): update CHANGELOG.md ### 📊 Changes **2 files changed** (+30 additions, -3 deletions) <details> <summary>View changed files</summary> 📝 `CHANGELOG.md` (+1 -1) 📝 `nvitop/api/device.py` (+29 -2) </details> ### 📄 Description  #### Issue Type  - Bug fix #### Description  Further isolate the `CUDA_VISIBLE_DEVICE` parser in a subprocess. We are using both `subprocess` and `mutliprocessing` modules. #### Motivation and Context    Fixes #69 #### Testing   ```python # test.py from nvitop import CudaDevice print(CudaDevice.count()) ``` Before: ```console $ python3 test.py Traceback (most recent call last): File "<string>", line 1, in <module> File "/home/linuxbrew/.linuxbrew/opt/python@3.11/lib/python3.11/multiprocessing/spawn.py", line 120, in spawn_main exitcode = _main(fd, parent_sentinel) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/linuxbrew/.linuxbrew/opt/python@3.11/lib/python3.11/multiprocessing/spawn.py", line 129, in _main prepare(preparation_data) File "/home/linuxbrew/.linuxbrew/opt/python@3.11/lib/python3.11/multiprocessing/spawn.py", line 240, in prepare _fixup_main_from_path(data['init_main_from_path']) File "/home/linuxbrew/.linuxbrew/opt/python@3.11/lib/python3.11/multiprocessing/spawn.py", line 291, in _fixup_main_from_path main_content = runpy.run_path(main_path, ^^^^^^^^^^^^^^^^^^^^^^^^^ File "<frozen runpy>", line 291, in run_path File "<frozen runpy>", line 98, in _run_module_code File "<frozen runpy>", line 88, in _run_code File "/home/PanXuehai/Projects/nvitop/test.py", line 5, in <module> print(CudaDevice.count()) ^^^^^^^^^^^^^^^^^^ File "/home/PanXuehai/Projects/nvitop/nvitop/api/device.py", line 2132, in count return len(super().parse_cuda_visible_devices()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/PanXuehai/Projects/nvitop/nvitop/api/device.py", line 488, in parse_cuda_visible_devices return parse_cuda_visible_devices(cuda_visible_devices) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/PanXuehai/Projects/nvitop/nvitop/api/device.py", line 2357, in parse_cuda_visible_devices return _parse_cuda_visible_devices(cuda_visible_devices, format='index') ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/PanXuehai/Projects/nvitop/nvitop/api/device.py", line 2491, in _parse_cuda_visible_devices raw_uuids = _parse_cuda_visible_devices_to_uuids(cuda_visible_devices, verbose=False) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/PanXuehai/Projects/nvitop/nvitop/api/device.py", line 2616, in _parse_cuda_visible_devices_to_uuids parser.start() File "/home/linuxbrew/.linuxbrew/opt/python@3.11/lib/python3.11/multiprocessing/process.py", line 121, in start self._popen = self._Popen(self) ^^^^^^^^^^^^^^^^^ File "/home/linuxbrew/.linuxbrew/opt/python@3.11/lib/python3.11/multiprocessing/context.py", line 288, in _Popen return Popen(process_obj) ^^^^^^^^^^^^^^^^^^ File "/home/linuxbrew/.linuxbrew/opt/python@3.11/lib/python3.11/multiprocessing/popen_spawn_posix.py", line 32, in __init__ super().__init__(process_obj) File "/home/linuxbrew/.linuxbrew/opt/python@3.11/lib/python3.11/multiprocessing/popen_fork.py", line 19, in __init__ self._launch(process_obj) File "/home/linuxbrew/.linuxbrew/opt/python@3.11/lib/python3.11/multiprocessing/popen_spawn_posix.py", line 42, in _launch prep_data = spawn.get_preparation_data(process_obj._name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/linuxbrew/.linuxbrew/opt/python@3.11/lib/python3.11/multiprocessing/spawn.py", line 158, in get_preparation_data _check_not_importing_main() File "/home/linuxbrew/.linuxbrew/opt/python@3.11/lib/python3.11/multiprocessing/spawn.py", line 138, in _check_not_importing_main raise RuntimeError(''' RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping phase. This probably means that you are not using fork to start your child processes and you have forgotten to use the proper idiom in the main module: if __name__ == '__main__': freeze_support() ... The "freeze_support()" line can be omitted if the program is not going to be frozen to produce an executable. ``` After: ```console $ python3 test.py 1 ``` --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>