Repository navigation
Description
Is this a duplicate?
- I confirmed there appear to be no duplicate issues for this bug and that I agree to the Code of Conduct
Type of Bug
Runtime Error
Component
cuda.core
Describe the bug
Context.close() refuses to close a context that is current on the calling thread ("Cannot close a CUDA context while it is current"), but it does not notice when the context is current on another thread. Closing a green context from a second thread therefore destroys it (cuGreenCtxDestroy) while it is still current on the first thread.
Device.set_current(ctx) stores the caller's Context object (self._context = ctx in cuda_core/cuda/core/_device.pyx), not a reference of its own. close() resets that object's handle, so it also drops the reference the Device was holding. If no stream or other object holds the context, its last reference is gone and the green context is destroyed.
After that, the first thread is left with a destroyed current context:
- Operations that use the current context fail with
CUDA_ERROR_CONTEXT_IS_DESTROYED. Device.set_current(other)returns the previous context as an open, non-greenContextwrapping the destroyedCUcontext. The registry entry for the green context is gone, socreate_context_handle_ref()builds a new, non-owning box for it.- That
Contextcan be made current again. Using it fails withCUDA_ERROR_CONTEXT_IS_DESTROYED, and in about 1 of 4 runs the process aborts at exit withfree(): double free detected in tcache 2.
The stale Context from step 2 also triggers the green-context registry problem described in #3533. If the driver reuses the CUcontext address for a later green context, Device.create_context() returns this stale, non-green Context, and the new green context is destroyed as soon as create_context() returns. As far as I can tell, this cross-thread close() is the only way to reach that state using only cuda.core APIs.
How to Reproduce
import threading
from cuda.core import ContextOptions, Device, SMResourceOptions
dev = Device(0)
dev.set_current()
primary = dev.context
groups, _ = dev.resources.sm.split(SMResourceOptions(count=None))
ctx = dev.create_context(ContextOptions(resources=[groups[0]]))
dev.set_current(ctx)
# ctx.close() on this thread raises "Cannot close a CUDA context while it is current".
# On another thread it succeeds and destroys the green context.
t = threading.Thread(target=ctx.close)
t.start()
t.join()
# 1. The destroyed context is still current on this thread.
try:
dev.default_stream.sync()
except Exception as e:
print("1.", e)
# 2. Switching away returns an open, non-green Context for the destroyed context.
prev = dev.set_current(primary)
print(f"2. {prev} is_closed={prev.is_closed} is_green={prev.is_green}")
# 3. That Context can be made current again; using it fails, and the process
# may abort at exit with heap corruption.
dev.set_current(prev)
try:
dev.create_stream()
except Exception as e:
print("3.", e)Output:
1. CUDA_ERROR_CONTEXT_IS_DESTROYED: This error indicates that the context current to the calling thread has been destroyed using cuCtxDestroy, or is a primary context which has not yet been initialized.
2. <Context handle=0x2b3eba10 device=0> is_closed=False is_green=False
3. CUDA_ERROR_CONTEXT_IS_DESTROYED: This error indicates that the context current to the calling thread has been destroyed using cuCtxDestroy, or is a primary context which has not yet been initialized.
In 3 of 12 runs, the process then aborted at exit with free(): double free detected in tcache 2 (exit code 134). Steps 1 and 2 alone never aborted in my runs; the abort follows making the destroyed context current again in step 3.
Expected behavior
Closing a Context should not destroy a context that is still current on another thread. Either close() raises, as it does for the calling thread, or the context stays alive until no thread has it current.
One possible fix is for Device.set_current(ctx) to keep its own ContextHandle copy rather than the caller's Context object. Device instances are thread-local, so each thread that has the context current would then hold a reference, and close() would only drop the caller's reference. The green context would be destroyed when the last thread switches away from it.
Operating System
Ubuntu Linux (kernel 7.0), Python 3.12.3, cuda-core built from main at edf1b31 (1.3.1.dev54), cuda-bindings 13.5.0a3.dev30.
nvidia-smi output
NVIDIA RTX 6000 Ada Generation, driver 595.99.02, CUDA 13.2.
Activity