---
source: https://iinventedlight.com/library/notes/pytorch-autograd.md
title: "How PyTorch autograd works"
license: CC-BY-4.0
license_url: https://creativecommons.org/licenses/by/4.0/
source_license: "BSD-style (pytorch/pytorch LICENSE; Copyright (c) 2016-present, Facebook Inc.; https://github.com/pytorch/pytorch/blob/main/LICENSE); docs only, not the tutorials repo; PyTorch name and logo not licensed; prose restates or closely paraphrases cited source lines"
license_family: pytorch-bsd
rule_check: "needs rule check (family not in auto-publish rule; check.mjs auto-approves only usgov and arxiv-ccby)"
attribution: "Summary by iinventedlight (AI-assisted). Sources: PyTorch docs, Autograd mechanics — PyTorch 2.15 documentation, BSD-style (pytorch/pytorch LICENSE; Copyright (c) 2016-present, Facebook Inc.; https://github.com/pytorch/pytorch/blob/main/LICENSE); docs only, not the tutorials repo; PyTorch name and logo not licensed; PyTorch docs, Automatic differentiation package - torch.autograd — PyTorch 2.15 documentation, BSD-style (pytorch/pytorch LICENSE; Copyright (c) 2016-present, Facebook Inc.; https://github.com/pytorch/pytorch/blob/main/LICENSE); docs only, not the tutorials repo; PyTorch name and logo not licensed."
author: "iinventedlight (synthesis of cited sources); text drafted with AI assistance"
date: 2026-11-04
status: published
checked_on: 2026-11-04
as_of: 2026-11-04
confidence: medium
untested: "No code was run for this note; only the fetched documentation pages were read."
category: coding-references
topic_area: "code-ai-eng: PyTorch"
tags: ['pytorch', 'autograd', 'gradients']
sources:
  - id: S1
    title: "Autograd mechanics — PyTorch 2.15 documentation"
    publisher: PyTorch docs
    url: https://docs.pytorch.org/docs/2.15/notes/autograd.html
    accessed: 2026-11-04
    license: "BSD-style (pytorch/pytorch LICENSE; Copyright (c) 2016-present, Facebook Inc.; https://github.com/pytorch/pytorch/blob/main/LICENSE); docs only, not the tutorials repo; PyTorch name and logo not licensed"
  - id: S2
    title: "Automatic differentiation package - torch.autograd — PyTorch 2.15 documentation"
    publisher: PyTorch docs
    url: https://docs.pytorch.org/docs/2.15/autograd.html
    accessed: 2026-11-04
    license: "BSD-style (pytorch/pytorch LICENSE; Copyright (c) 2016-present, Facebook Inc.; https://github.com/pytorch/pytorch/blob/main/LICENSE); docs only, not the tutorials repo; PyTorch name and logo not licensed"
---
# How PyTorch autograd works

**Answer:** Conceptually, autograd records a graph recording all of the operations that created the data as you execute operations, giving you a directed acyclic graph whose leaves are the input tensors and roots are the output tensors. By tracing this graph from roots to leaves, you can automatically compute the gradients using the chain rule. (S1)

## Forward and backward

- When the forward pass is completed, we evaluate this graph in the backwards pass to compute the gradients. (S1)
- Some operations need intermediary results to be saved during the forward pass in order to execute the backward pass. (S1)
- When defining a custom Python Function, you can use save_for_backward() to save tensors during the forward pass and saved_tensors to retrieve them during the backward pass. (S1)

## requires_grad and leaves

- requires_grad is a flag, defaulting to false unless wrapped in a nn.Parameter, that allows for fine-grained exclusion of subgraphs from gradient computation. (S1)
- torch.autograd provides classes and functions implementing automatic differentiation of arbitrary scalar valued functions. (S2)
- Autograd automatically supports Tensors with requires_grad set to True. (S2)
- All Tensors that have requires_grad which is False will be leaf Tensors by convention. (S2)

## Claims
- id: C1
  claim: Autograd records operations as a DAG.
  source: S1
  evidence: "autograd records a graph recording all of the operations that created"
- id: C2
  claim: Gradients are computed with the chain rule.
  source: S1
  evidence: "compute the gradients using the chain rule."
- id: C3
  claim: requires_grad excludes subgraphs.
  source: S1
  evidence: "For more fine-grained exclusion of subgraphs from gradient computation"
- id: C4
  claim: Tensors without requires_grad are leaves.
  source: S2
  evidence: "All Tensors that have requires_grad which is False will be leaf Tensors by convention."

## Sources
- [S1] Autograd mechanics — PyTorch 2.15 documentation. PyTorch docs. https://docs.pytorch.org/docs/2.15/notes/autograd.html . License: BSD-style (pytorch/pytorch LICENSE; Copyright (c) 2016-present, Facebook Inc.; https://github.com/pytorch/pytorch/blob/main/LICENSE); docs only, not the tutorials repo; PyTorch name and logo not licensed. Accessed 2026-11-04.
- [S2] Automatic differentiation package - torch.autograd — PyTorch 2.15 documentation. PyTorch docs. https://docs.pytorch.org/docs/2.15/autograd.html . License: BSD-style (pytorch/pytorch LICENSE; Copyright (c) 2016-present, Facebook Inc.; https://github.com/pytorch/pytorch/blob/main/LICENSE); docs only, not the tutorials repo; PyTorch name and logo not licensed. Accessed 2026-11-04.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted).