← All notes

Answer: Conceptually, autograd records a graph recording all of the operations that created the data as you execute operations, giving you a directed acyclic graph whose leaves are the input tensors and roots are the output tensors. By tracing this graph from roots to leaves, you can automatically compute the gradients using the chain rule. (S1)

Forward and backward

• When the forward pass is completed, we evaluate this graph in the backwards pass to compute the gradients. (S1)
• Some operations need intermediary results to be saved during the forward pass in order to execute the backward pass. (S1)
• When defining a custom Python Function, you can use save_for_backward() to save tensors during the forward pass and saved_tensors to retrieve them during the backward pass. (S1)

requires_grad and leaves

• requires_grad is a flag, defaulting to false unless wrapped in a nn.Parameter, that allows for fine-grained exclusion of subgraphs from gradient computation. (S1)
• torch.autograd provides classes and functions implementing automatic differentiation of arbitrary scalar valued functions. (S2)
• Autograd automatically supports Tensors with requires_grad set to True. (S2)
• All Tensors that have requires_grad which is False will be leaf Tensors by convention. (S2)

Claims

• id: C1 claim: Autograd records operations as a DAG. source: S1 evidence: "autograd records a graph recording all of the operations that created"
• id: C2 claim: Gradients are computed with the chain rule. source: S1 evidence: "compute the gradients using the chain rule."
• id: C3 claim: requires_grad excludes subgraphs. source: S1 evidence: "For more fine-grained exclusion of subgraphs from gradient computation"
• id: C4 claim: Tensors without requires_grad are leaves. source: S2 evidence: "All Tensors that have requires_grad which is False will be leaf Tensors by convention."

Sources

• [S1] Autograd mechanics — PyTorch 2.15 documentation. PyTorch docs. https://docs.pytorch.org/docs/2.15/notes/autograd.html . License: BSD-style (pytorch/pytorch LICENSE; Copyright (c) 2016-present, Facebook Inc.; https://github.com/pytorch/pytorch/blob/main/LICENSE); docs only, not the tutorials repo; PyTorch name and logo not licensed. Accessed 2026-11-04.
• [S2] Automatic differentiation package - torch.autograd — PyTorch 2.15 documentation. PyTorch docs. https://docs.pytorch.org/docs/2.15/autograd.html . License: BSD-style (pytorch/pytorch LICENSE; Copyright (c) 2016-present, Facebook Inc.; https://github.com/pytorch/pytorch/blob/main/LICENSE); docs only, not the tutorials repo; PyTorch name and logo not licensed. Accessed 2026-11-04.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted).

@misc{iil_note_6h053en6,
  author = {{iinventedlight}},
  title = {{How PyTorch autograd works}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-04},
  url = {https://iinventedlight.com/library/notes/pytorch-autograd/},
  note = {Permanent ID iil:note:6h053en6. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt. This page in Markdown: /library/notes/pytorch-autograd.md.