$\mu$pscaling small models: Principled warm starts and hyperparameter transfer

Modern large-scale neural networks are often trained and released in multiple sizes to accommodate diverse inference budgets.

To improve efficiency, recent work has explored model upscaling: initializing larger models from trained smaller ones to accelerate convergence.

However, this method can be sensitive to hyperparameters that need to be tuned at the target upscaled model size, which is prohibitively costly to do directly.

It remains unclear whether tuning hyperparameters on smaller models and extrapolating via scaling laws is sound in this setting.

We address this with principled approaches to width-based upscaling and efficient hyperparameter tuning in this setting.

Motivated by $\mu$P and any-dimensional architectures, we introduce a general upscaling method that, like Net2Net, copies and perturbs weights, but uses theoretically grounded, width-dependent scalings for the perturbation noise and optimizer hyperparameters.

First, we prove that under zero perturbation, the upscaled model is functionally equivalent to the base model throughout training.

Second, we extend the $\mu$P theory to enable infinite-width limit analysis and establish hyperparameter transfer for upscaled models, greatly reducing the tuning cost.

We empirically demonstrate that this method is effective on realistic datasets and architectures.