A neural network contains many dense layers which perform matrix multiplication. The weight
matrices in these layers typically have full-rank. When adapting to a specific task, Aghajanyan et al.
(2020) shows that the pre-trained language models have a low “instrisic dimension” and can still
learn efficiently despite a random projection to a smaller subspace. Inspired by this, we hypothesize the updates to the weights also have a low “intrinsic rank” during adaptation. For a pre-trained
weight matrix W0 ∈ R
d×k
, we constrain its update by representing the latter with a low-rank decomposition W0 + ∆W = W0 + BA, where B ∈ R
d×r
, A ∈ R
r×k
, and the rank r  min(d, k).
During training, W0 is frozen and does not receive gradient updates, while A and B contain trainable
parameters. Note both W0 and ∆W = BA are multiplied with the same input, and their respective
output vectors are summed coordinate-wise. For h = W0x, our modified forward pass yields:
h = W0x + ∆W x = W0x + BAx (3)
We illustrate our reparametrization in Figure 1. We use a random Gaussian initialization for A and
zero for B, so ∆W = BA is zero at the beginning of training. We then scale ∆W x by α
r
, where α
is a constant in r. When optimizing with Adam, tuning α is roughly the same as tuning the learning
rate if we scale the initialization appropriately. As a result, we simply set α to the first r we try
and do not tune it. This scaling helps to reduce the need to retune hyperparameters when we vary
r (Yang & Hu, 2021).
A Generalization of Full Fine