← All notes

Answer: torch.cuda is used to set up and run CUDA operations. It keeps track of the currently selected GPU, and all CUDA tensors you allocate will by default be created on that device. (S1)

Devices

• The selected device can be changed with a torch.cuda.device context manager. (S1)
• Once a tensor is allocated, you can do operations on it irrespective of the selected device, and the results will be always placed on the same device as the tensor. (S1)
• Unless you enable peer-to-peer memory access, any attempts to launch ops on tensors spread across different devices will raise an error. (S1)

Asynchrony and memory

• A consequence of the asynchronous computation is that time measurements without synchronizations are not accurate. (S1)
• Several functions such as to() and copy_() admit an explicit non_blocking argument, which lets the caller bypass synchronization when it is unnecessary. (S1)
• Calling empty_cache() releases all unused cached memory from PyTorch so that those can be used by other GPU applications. (S1)
• The torch.cuda module can check if peer access between two devices is possible and return the currently selected Stream for a given device. (S2)

Claims

• id: C1 claim: torch.cuda can check peer access. source: S2 evidence: "Check if peer access between two devices is possible."
• id: C2 claim: CUDA tensors go on the selected device by default. source: S1 evidence: "and all CUDA tensors you allocate will by default be"
• id: C3 claim: Cross-device ops raise an error. source: S1 evidence: "tensors spread across different devices will raise an error."
• id: C4 claim: Timing without synchronization is inaccurate. source: S1 evidence: "synchronizations are not accurate. To get precise measurements"
• id: C5 claim: empty_cache releases unused cached memory. source: S1 evidence: "releases all unused cached memory from PyTorch so that those can be used"

Sources

• [S1] CUDA semantics — PyTorch 2.15 documentation. PyTorch docs. https://docs.pytorch.org/docs/2.15/notes/cuda.html . License: BSD-style (pytorch/pytorch LICENSE; Copyright (c) 2016-present, Facebook Inc.; https://github.com/pytorch/pytorch/blob/main/LICENSE); docs only, not the tutorials repo; PyTorch name and logo not licensed. Accessed 2026-11-04.
• [S2] torch.cuda — PyTorch 2.15 documentation. PyTorch docs. https://docs.pytorch.org/docs/2.15/cuda.html . License: BSD-style (pytorch/pytorch LICENSE; Copyright (c) 2016-present, Facebook Inc.; https://github.com/pytorch/pytorch/blob/main/LICENSE); docs only, not the tutorials repo; PyTorch name and logo not licensed. Accessed 2026-11-04.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted).

@misc{iil_note_e3yw4d3d,
  author = {{iinventedlight}},
  title = {{CUDA semantics in PyTorch}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-04},
  url = {https://iinventedlight.com/library/notes/pytorch-cuda-semantics/},
  note = {Permanent ID iil:note:e3yw4d3d. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt. This page in Markdown: /library/notes/pytorch-cuda-semantics.md.