github-starred/nvitop

Fork 0

mirror of https://github.com/XuehaiPan/nvitop.git synced 2026-05-15 14:15:55 -06:00

[GH-ISSUE #112] [BUG] Pytorch lightning callback #68

New issue

Closed

opened 2026-05-05 03:24:24 -06:00 by gitea-mirror · 3 comments

gitea-mirror commented

2026-05-05 03:24:24 -06:00

Owner

Originally created by @marios1861 on GitHub (Dec 12, 2023).
Original GitHub issue: https://github.com/XuehaiPan/nvitop/issues/112

Originally assigned to: @XuehaiPan on GitHub.

Required prerequisites

I have read the documentation https://nvitop.readthedocs.io.
I have searched the Issue Tracker that this hasn't already been reported. (comment there if it has.)
I have tried the latest version of nvitop in a new isolated virtual environment.

What version of nvitop are you using?

1.3.1

Operating system and version

Ubuntu 20.04.6 LTS

NVIDIA driver version

535.129.03

NVIDIA-SMI

No response

Python environment

poetry environment

Problem description

When using pytorch lightning callback, model_helpers function in lightning.pytorch.utilities raises ValueError("Expected a parent"), even though

elif isinstance(instance, pl.Callback):
  parent = pl.Callback

should be the chosen branch, isinstance(instance, pl.Callback) returns false. After further inspection, the older package pytorch_lightning is being used, which can be replaced without any further changes with lightning.pytorch.

Steps to Reproduce

gpu_stats = GpuStatsLogger()
trainer = pl.Trainer(logger=etc, callbacks=[gpu_stats])

Traceback

Traceback (most recent call last):
  File "...", line 40, in <module>
    trainer = pl.Trainer(..., callbacks=[callback])
  File "/.../lib/python3.8/site-packages/lightning/pytorch/utilities/argparse.py", line 70, in insert_env_defaults
    return fn(self, **kwargs)
  File "/.../lib/python3.8/site-packages/lightning/pytorch/trainer/trainer.py", line 431, in __init__
    self._callback_connector.on_trainer_init(
  File "/.../lib/python3.8/site-packages/lightning/pytorch/trainer/connectors/callback_connector.py", line 79, in on_trainer_init
    _validate_callbacks_list(self.trainer.callbacks)
  File "/.../lib/python3.8/site-packages/lightning/pytorch/trainer/connectors/callback_connector.py", line 227, in _validate_callbacks_list
    stateful_callbacks = [cb for cb in callbacks if is_overridden("state_dict", instance=cb)]
  File "/.../lib/python3.8/site-packages/lightning/pytorch/trainer/connectors/callback_connector.py", line 227, in <listcomp>
    stateful_callbacks = [cb for cb in callbacks if is_overridden("state_dict", instance=cb)]
  File "/.../lib/python3.8/site-packages/lightning/pytorch/utilities/model_helpers.py", line 39, in is_overridden
    raise ValueError("Expected a parent")

Logs

No response

Expected behavior

No response

Additional context

No response

Originally created by @marios1861 on GitHub (Dec 12, 2023). Original GitHub issue: https://github.com/XuehaiPan/nvitop/issues/112 Originally assigned to: @XuehaiPan on GitHub. ### Required prerequisites - [X] I have read the documentation <https://nvitop.readthedocs.io>. - [X] I have searched the [Issue Tracker](https://github.com/XuehaiPan/nvitop/issues) that this hasn't already been reported. (comment there if it has.) - [X] I have tried the latest version of nvitop in a new isolated virtual environment. ### What version of nvitop are you using? 1.3.1 ### Operating system and version Ubuntu 20.04.6 LTS ### NVIDIA driver version 535.129.03 ### NVIDIA-SMI _No response_ ### Python environment poetry environment ### Problem description When using pytorch lightning callback, `model_helpers` function in `lightning.pytorch.utilities` raises `ValueError("Expected a parent")`, even though ```python elif isinstance(instance, pl.Callback): parent = pl.Callback ``` should be the chosen branch, `isinstance(instance, pl.Callback)` returns false. After further inspection, the older package pytorch_lightning is being used, which can be replaced without any further changes with lightning.pytorch. ### Steps to Reproduce ```python gpu_stats = GpuStatsLogger() trainer = pl.Trainer(logger=etc, callbacks=[gpu_stats]) ``` ### Traceback ```pytb Traceback (most recent call last): File "...", line 40, in <module> trainer = pl.Trainer(..., callbacks=[callback]) File "/.../lib/python3.8/site-packages/lightning/pytorch/utilities/argparse.py", line 70, in insert_env_defaults return fn(self, **kwargs) File "/.../lib/python3.8/site-packages/lightning/pytorch/trainer/trainer.py", line 431, in __init__ self._callback_connector.on_trainer_init( File "/.../lib/python3.8/site-packages/lightning/pytorch/trainer/connectors/callback_connector.py", line 79, in on_trainer_init _validate_callbacks_list(self.trainer.callbacks) File "/.../lib/python3.8/site-packages/lightning/pytorch/trainer/connectors/callback_connector.py", line 227, in _validate_callbacks_list stateful_callbacks = [cb for cb in callbacks if is_overridden("state_dict", instance=cb)] File "/.../lib/python3.8/site-packages/lightning/pytorch/trainer/connectors/callback_connector.py", line 227, in <listcomp> stateful_callbacks = [cb for cb in callbacks if is_overridden("state_dict", instance=cb)] File "/.../lib/python3.8/site-packages/lightning/pytorch/utilities/model_helpers.py", line 39, in is_overridden raise ValueError("Expected a parent") ``` ### Logs _No response_ ### Expected behavior _No response_ ### Additional context _No response_

gitea-mirror

2026-05-05 03:24:24 -06:00

closed this issue
added the
bug
label

gitea-mirror commented

2026-05-05 03:24:27 -06:00

Author

Owner

@XuehaiPan commented on GitHub (Dec 12, 2023):

Hi @marios1861, does lightning.pytorch.callbacks.DeviceStatsMonitor fit your use case? The callback in nvitop has not been updated for years. I would rather mark it as deprecated. Because we don't have a release schedule that's aligned with Lightning-AI/lightning. See my previous discussion in https://github.com/XuehaiPan/nvitop/pull/84#issuecomment-1663400377.

@XuehaiPan commented on GitHub (Dec 12, 2023): Hi @marios1861, does [`lightning.pytorch.callbacks.DeviceStatsMonitor`](https://github.com/Lightning-AI/pytorch-lightning/blob/master/src/lightning/pytorch/callbacks/device_stats_monitor.py) fit your use case? The callback in `nvitop` has not been updated for years. I would rather mark it as deprecated. Because we don't have a release schedule that's aligned with [Lightning-AI/lightning](https://github.com/Lightning-AI/lightning). See my previous discussion in https://github.com/XuehaiPan/nvitop/pull/84#issuecomment-1663400377.

gitea-mirror commented

2026-05-05 03:24:28 -06:00

Author

Owner

@XuehaiPan commented on GitHub (Dec 12, 2023):

When using pytorch lightning callback, model_helpers function in lightning.pytorch.utilities raises ValueError("Expected a parent"), even though
elif isinstance(instance, pl.Callback):
  parent = pl.Callback
should be the chosen branch, isinstance(instance, pl.Callback) returns false. After further inspection, the older package pytorch_lightning is being used, which can be replaced without any further changes with lightning.pytorch.

@marios1861 I guess you are using lightning.pytorch rather than pytorch_lightning, right?

@XuehaiPan commented on GitHub (Dec 12, 2023): > When using pytorch lightning callback, `model_helpers` function in `lightning.pytorch.utilities` raises `ValueError("Expected a parent")`, even though > > ```python > elif isinstance(instance, pl.Callback): > parent = pl.Callback > ``` > > should be the chosen branch, `isinstance(instance, pl.Callback)` returns false. After further inspection, the older package pytorch_lightning is being used, which can be replaced without any further changes with lightning.pytorch. @marios1861 I guess you are using `lightning.pytorch` rather than `pytorch_lightning`, right?

gitea-mirror commented

2026-05-05 03:24:29 -06:00

Author

Owner

@marios1861 commented on GitHub (Dec 14, 2023):

Hello @XuehaiPan, yes I am using lightning.pytorch (the package was renamed). lightning.pytorch.callbacks.DeviceStatsMonitor is fine as well, but with adaptor modules (nvitop - lightning in this case) it's not very clear where the responsibility of implementation lies. Since lightning provides the base class for implementing callbacks, I would provide the implementation in nvitop. If that isn't your decision that's fine too.

@marios1861 commented on GitHub (Dec 14, 2023): Hello @XuehaiPan, yes I am using lightning.pytorch ([the package was renamed](https://github.com/Lightning-AI/pytorch-lightning/discussions/16688)). [lightning.pytorch.callbacks.DeviceStatsMonitor](https://github.com/Lightning-AI/pytorch-lightning/blob/master/src/lightning/pytorch/callbacks/device_stats_monitor.py) is fine as well, but with adaptor modules (nvitop - lightning in this case) it's not very clear where the responsibility of implementation lies. Since lightning provides the base class for implementing callbacks, I would provide the implementation in nvitop. If that isn't your decision that's fine too.