---
source: https://iinventedlight.com/library/notes/hf-text-generation-strategies.md
title: "Text generation strategies in Transformers"
license: CC-BY-4.0
license_url: https://creativecommons.org/licenses/by/4.0/
source_license: "Apache-2.0 (huggingface docs repos); prose restates cited source lines"
attribution: "Summary by iinventedlight (AI-assisted). Sources: Hugging Face docs, Generation strategies · Hugging Face, Apache-2.0 (huggingface docs repos); Hugging Face docs, Text generation · Hugging Face, Apache-2.0 (huggingface docs repos)."
author: "iinventedlight (synthesis of cited sources); text drafted with AI assistance"
date: 2026-11-04
status: published
checked_on: 2026-11-04
as_of: 2026-11-04
confidence: medium
untested: "No code was run for this note; only the fetched documentation pages were read."
category: coding-references
topic_area: "code-ai-eng: Hugging Face"
tags: ['hugging-face', 'transformers', 'llm', 'decoding']
sources:
  - id: S1
    title: "Generation strategies · Hugging Face"
    publisher: Hugging Face docs
    url: https://huggingface.co/docs/transformers/generation_strategies
    accessed: 2026-11-04
    license: "Apache-2.0 (huggingface docs repos)"
  - id: S2
    title: "Text generation · Hugging Face"
    publisher: Hugging Face docs
    url: https://huggingface.co/docs/transformers/llm_tutorial
    accessed: 2026-11-04
    license: "Apache-2.0 (huggingface docs repos)"
---
# Text generation strategies in Transformers

**Answer:** 

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1", padding_side="left")
model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-v0.1", device_map="auto")

inputs = tok(["A list of colors: red, blue"], return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=50, do_sample=True, temperature=0.7)
print(tok.batch_decode(out, skip_special_tokens=True)[0])
```

## Strategies

|---|---|---|
| Greedy | default | short, factual output |
| Sampling | `do_sample=True` | creative or chat text |
| Beam search | `num_beams=4` | translation, summarization |

## Claims
- id: C1
  claim: Greedy search is the default decoding strategy.
  source: S1
  evidence: "Greedy search is the default decoding strategy."
- id: C2
  claim: Unless specified, greedy generates a maximum of 20 new tokens.
  source: S1
  evidence: "this strategy generates a maximum of 20 new tokens."
- id: C3
  claim: Greedy search begins to repeat itself on longer sequences.
  source: S1
  evidence: "it breaks down when generating longer sequences because it begins to repeat itself."
- id: C4
  claim: Beam search is good on input-grounded tasks.
  source: S2
  evidence: "Beam search is good on input-grounded tasks."
- id: C5
  claim: Set repetition_penalty above 1.0 if the model repeats itself.
  source: S2
  evidence: "repetition_penaltyfloatSet it to >1.0 if you’re seeing the model repeat itself often."

## Sources
- [S1] Generation strategies · Hugging Face. Hugging Face docs. https://huggingface.co/docs/transformers/generation_strategies . License: Apache-2.0 (huggingface docs repos). Accessed 2026-11-04.
- [S2] Text generation · Hugging Face. Hugging Face docs. https://huggingface.co/docs/transformers/llm_tutorial . License: Apache-2.0 (huggingface docs repos). Accessed 2026-11-04.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted).