← All notes

Answer: Hyper-parameters are parameters that are not directly learnt within estimators. It is possible and recommended to search the hyper-parameter space for the best cross validation score. (S1)

Two approaches

• For given values, GridSearchCV exhaustively considers all parameter combinations, while RandomizedSearchCV can sample a given number of candidates from a parameter space with a specified distribution. (S1)
• GridSearchCV exhaustively generates candidates from a grid of parameter values specified with the param_grid parameter. (S1)
• When fitting it on a dataset, all the possible combinations of parameter values are evaluated and the best combination is retained. (S1)
• HalvingGridSearchCV and HalvingRandomSearchCV can be much faster at finding a good parameter combination. (S1)

Tips

• Any parameter provided when constructing an estimator may be optimized in this manner. (S1)
• It is common that a small subset of parameters has a large impact on the predictive or computation performance of the model while others can be left to their default values. (S1)
• Using a pipeline for cross-validation and searching will largely keep you from the common pitfall of leaking test data. (S2)

Claims

• id: C1 claim: Hyper-parameters are not learnt within estimators. source: S1 evidence: "Hyper-parameters are parameters that are not directly learnt within estimators."
• id: C2 claim: GridSearchCV tries all combinations. source: S1 evidence: "for given values, GridSearchCV exhaustively considers"
• id: C3 claim: Any constructor parameter can be optimized. source: S1 evidence: "Any parameter provided when constructing an estimator may be optimized in this"
• id: C4 claim: Use a pipeline when searching. source: S2 evidence: "Using a pipeline for cross-validation and searching will largely keep"

Sources

• [S1] 3.2. Tuning the hyper-parameters of an estimator — scikit-learn 1.9.1 documentation. scikit-learn docs. https://scikit-learn.org/stable/modules/grid_search.html . License: BSD-3-Clause (scikit-learn/scikit-learn COPYING; Copyright (c) 2007-2026 The scikit-learn developers; https://github.com/scikit-learn/scikit-learn/blob/main/COPYING). Accessed 2026-11-04.
• [S2] Getting Started — scikit-learn 1.9.1 documentation. scikit-learn docs. https://scikit-learn.org/stable/getting_started.html . License: BSD-3-Clause (scikit-learn/scikit-learn COPYING; Copyright (c) 2007-2026 The scikit-learn developers; https://github.com/scikit-learn/scikit-learn/blob/main/COPYING). Accessed 2026-11-04.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted).

@misc{iil_note_r83m0cxa,
  author = {{iinventedlight}},
  title = {{Hyper-parameter search with GridSearchCV and RandomizedSearchCV}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-04},
  url = {https://iinventedlight.com/library/notes/sklearn-grid-search/},
  note = {Permanent ID iil:note:r83m0cxa. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt. This page in Markdown: /library/notes/sklearn-grid-search.md.