Hyperparameter Tuning: The Practical Guide to Not Guessing

A model’s parameters are learned during training. Its hyperparameters are set before training begins — learning rate, batch size, number of layers, regularization strength, dropout rate. These choices shape how training unfolds and often determine whether a model is mediocre or production-ready.

The problem is that the right hyperparameters depend on your specific data, task, and compute budget. There’s no universal answer. But there are systematic approaches that work far better than guessing.

Why hyperparameters matter more than you think

Consider learning rate alone. Set it too high, and training oscillates or diverges — the model never converges to a good solution. Set it too low, and training takes forever and may get stuck in a poor local minimum. The difference between a learning rate of 0.001 and 0.01 can be the difference between a model that works and one that doesn’t.

Now multiply that by every hyperparameter in your system. A typical deep learning model has 5-15 hyperparameters that meaningfully affect performance. The interaction effects between them make intuition unreliable. A batch size of 32 might work great with learning rate 0.001 but fail with 0.0001.

The tuning spectrum

There’s a spectrum from “no tuning” to “exhaustive search,” and the right position depends on your constraints.

Manual tuning (the educated guess)

Start with known defaults from papers or established codebases. For many architectures, the community has converged on reasonable starting points:

• Learning rate: 1e-3 for Adam, 1e-2 for SGD with momentum
• Batch size: 32-256 depending on GPU memory
• Weight decay: 1e-4 to 1e-2
• Dropout: 0.1-0.3 for transformers, 0.5 for older architectures

Manual tuning works when you have strong priors about your problem. If you’re fine-tuning a well-studied architecture on a standard task, defaults get you 80% of the way there.

Grid search (the brute force)

Define a grid of values for each hyperparameter and try every combination. If you have 3 learning rates, 3 batch sizes, and 3 dropout values, that’s 27 experiments.

Grid search is simple and parallelizable but scales exponentially. Adding a fourth hyperparameter with 3 values turns 27 experiments into 81. It also wastes budget testing combinations that are clearly bad — once you know a learning rate of 0.1 diverges, there’s no point testing it with every other parameter.

Random search (surprisingly effective)

Instead of a grid, sample hyperparameters randomly from defined ranges. Bergstra and Bengio showed in 2012 that random search finds good hyperparameters faster than grid search in most cases.

The reason is geometric: in high-dimensional spaces, grid search wastes many evaluations on unimportant dimensions. If learning rate matters a lot but batch size barely matters, grid search spends equal effort varying both. Random search naturally allocates more coverage to important dimensions.

Practical tip: Run 20-50 random trials. This is often enough to find a configuration within a few percent of optimal.

Bayesian optimization (the smart search)

Bayesian optimization builds a probabilistic model of the objective function (your validation metric as a function of hyperparameters) and uses it to choose which configuration to try next. It balances exploration (trying uncertain regions) with exploitation (refining promising regions).

Tools like Optuna, Ray Tune, and Weights & Biases Sweeps implement this. Bayesian optimization typically finds good configurations in fewer trials than random search — 10-30 trials can be sufficient for moderate hyperparameter spaces.

When it helps most: Expensive training runs where each evaluation takes hours or days. Bayesian optimization’s efficiency advantage grows when evaluations are costly.

Population-based training (the evolutionary approach)

PBT runs multiple training jobs in parallel, periodically copying weights from high-performing runs to low-performing ones while mutating their hyperparameters. It combines hyperparameter search with training, allowing hyperparameters to change during training.

This is particularly useful for hyperparameters that should change over time, like learning rate schedules. PBT can discover schedules that no fixed schedule would match.

What to tune first

Not all hyperparameters deserve equal attention. In practice, this ordering covers most scenarios:

1. Learning rate — Almost always the most impactful. Tune this first.
2. Learning rate schedule — Warmup steps, decay strategy, final learning rate.
3. Batch size — Interacts with learning rate. Larger batches often need larger learning rates.
4. Regularization — Weight decay, dropout, label smoothing. Tune when overfitting.
5. Architecture choices — Number of layers, hidden dimensions. Usually tune last, with bigger budget.

The learning rate finder

Before running a full hyperparameter search, use a learning rate range test. Start with a very small learning rate (1e-7) and increase it exponentially over one epoch, recording the loss at each step. Plot loss vs. learning rate.

The optimal learning rate is typically one order of magnitude below the point where loss starts increasing. This takes minutes and eliminates the most impactful hyperparameter from your search space.

Early stopping as a tuning strategy

Don’t run every experiment to completion. Use early stopping — monitor validation performance during training and halt runs that aren’t improving. This is especially valuable during hyperparameter search, where you might launch 50 runs but most will show within the first 20% of training whether they’ll be competitive.

Successive halving formalizes this: start many configurations with small budgets, periodically eliminate the worst performers, and allocate more budget to survivors. Tools like Hyperband implement this efficiently.

Common mistakes

Tuning on test data. Your test set should be touched exactly once, at the very end. Tune on a validation set. If your validation set is too small, use k-fold cross-validation.

Ignoring interactions. Learning rate and batch size interact strongly. Tuning them independently can miss the best combination.

Over-tuning. If your validation set is small, aggressive tuning can overfit to validation data. The more tuning iterations you run, the more likely you are to find a configuration that performs well on validation by chance.

Not tracking experiments. Log every run with its hyperparameters and results. You’ll want this history when you revisit the model in three months.

The practical framework

For most projects, this workflow balances quality with budget:

1. Start with established defaults for your architecture
2. Run a learning rate finder
3. Run 20-30 random search trials over your most important hyperparameters
4. If budget allows, run Bayesian optimization for 10-20 additional trials, seeded with results from step 3
5. Evaluate your best configuration on the held-out test set exactly once

This won’t find the absolute global optimum. It will find a configuration that’s close enough to matter — and it’ll do it in a budget that’s actually realistic for most teams.