For now, there may be some Windows-specific gotchas here:
A couple of the TODOs in the screenshot look less like GTX 1650 limitations and more like Windows / driver / memory-accounting boundaries, so I think you can probably make Neon more predictable without changing its basic direction.
The two most immediate points are:
For the NVML dependency, the NVIDIA-maintained Python package is currently nvidia-ml-py, while the import surface is still
pynvml
:
pip install nvidia-ml-py
python
from pynvml import nvmlInit
The separate pynvml
PyPI project is now deprecated and points users toward nvidia-ml-py
.
The --gpu-reset is not recognized
line on driver 591.86 may not mean “consumer GPU reset unsupported.” NVIDIA’s nvidia-smi changelog says Windows support for
nvidia-smi -r / --gpu-reset
was added between the nvidia-smi
updates.More importantly, current NVIDIA documentation describes GPU reset on Windows as a driver restart that disables/re-enables the affected device and requires administrator privileges. That is a fairly different operation from freeing PyTorch cache.
So if these aren’t already separate internally, I would probably keep the cleanup path conceptually split something like:
Python/tensor lifetime cleanup
↓
PyTorch allocator cache cleanup
↓
external/stale process handling
↓
CUDA/GPU/device recovery
↓
Windows driver restart
That lets RESET
, Kill Zombie
, periodic cleanup, etc. keep their useful roles without making every “free VRAM” operation equivalent to a device reset.
For testing, I also think a very small amount of extra telemetry would make the 4 GB result much easier to interpret:
generation peak:
torch peak allocated
torch peak reserved
after generation:
allocated / reserved
after Neon cleanup:
allocated / reserved
OOM counters:
num_ooms
num_alloc_retries
Windows:
dedicated GPU memory
shared GPU memory
recovery:
after a real OOM, can the same Python process generate again?
That last one may be particularly useful for Neon: “OOM happened, cleanup ran, then the same process successfully generated again” is a stronger recovery test than only measuring a low post-cleanup idle number.
I did a small synthetic sanity check on a T4 with PyTorch’s caching allocator capped at approximately 4 GiB. It is not a Neon/SDXL benchmark and it does not reproduce Windows, but it produced one distinction that may be useful as a regression-test idea:
gc.collect()
torch.cuda.empty_cache()
ran;gc.collect()
empty_cache()
left that full That lines up with PyTorch’s documented distinction: empty_cache() releases unused cached allocator memory, but does not free memory occupied by live tensors, and PyTorch has also documented CUDA tensors being kept alive by
So if “zombie tensor” in Neon is intended to include still-live Python references, something like a retained-exception/traceback case could be a useful tiny regression test. If Kill Zombie
is instead about stale external GPU processes, then it is really a separate layer, which is fine too.
One design boundary I would be particularly careful about is GPU RESET
.
According to current NVIDIA documentation, on Windows it is not just a more aggressive equivalent of:
torch.cuda.empty_cache()
It is a driver restart. NVIDIA also says GPU reset is not guaranteed to work in every situation and is not currently recommended as a general production recovery mechanism.
So I would treat that as the last recovery tier, separately from normal cleanup and external-process handling, and perhaps probe whether the installed nvidia-smi
actually supports -r
before presenting the action.
That also avoids an odd upgrade trap: on the 591.86 setup shown here, the command is simply unrecognized; on a v595+ Windows stack the same call can become a real driver restart.
Overall, I think the useful thing Neon can expose is not just “VRAM went from X to Y.” It can make the different failure states visible:
live Python/CUDA object
allocator cache
fragmented/reserved allocator state
external GPU process
Windows shared-memory fallback
device/driver state
and then use the least invasive recovery path that matches the layer.
That would preserve the practical “make SDXL usable/recoverable on a 4 GB machine” goal while also making the 1650/3060/4090 tester results much easier to compare and interpret.