LoRA: Fine-Tune a Giant Model by Training 1% of It

Fine-tuning a large model used to mean one painful thing: update every weight in it, keep a full copy per task, and pay for the GPUs to do it. A 7-billion-parameter model has 7 billion knobs. The Adam optimizer keeps two extra numbers per knob, so you're suddenly holding roughly three times the model in memory. Then, for every new task you tune, you store another 13 GB checkpoint. It works, but it's absurdly out of proportion to how much the model actually needs to change to, say, adopt a new…