Researchers from FAIR at Meta, Cornell University, and Carnegie Mellon University have demonstrated that large language models (LLMs) can learn to reason using a remarkably small number of trained parameters. The research team introduces TinyLoRA, a parameterization that can scale down to a single trainable parameter under extreme sharing settings. Using this method on a Qwen2.5-7B-Instruct backbone, the research team achieved 91.8% accuracy on the GSM8K benchmark with only 13 parameters, totaling just 26 bytes in bf16.
Overcoming the Constraints of Standard LoRA
Standard Low-Rank Adaptation (LoRA) adapts a frozen linear layer W ∈ Rdxk using trainable matrices A ∈ Rdxr and B ∈ Rrxk. The trainable parameter count in standard LoRA still scales with layer width and rank, which leaves a nontrivial lower bound even at rank 1. For a model like Llama3-8B, this minimum update size is approximately 3 million parameters.
TinyLoRA circumvents this by building upon LoRA-XS, which utilizes the truncated Singular Value Decomposition (SVD) of frozen weights. While LoRA-XS typically requires at least one parameter per adapted module, TinyLoRA replaces the trainable matrix with a low-dimensional trainable vector 𝜐 ∈ Ru projected through a fixed random tensor P ∈ Ruxrxr.
The update rule is defined as:
$$W’ = W + U\Sigma(\sum_{i=1}^{u}v_{i}P_{i})V^{\top}$$
By applying a weight tying factor (ntie), the total trainable parameters scale as O(nmu/ntie), allowing updates to scale down to a single parameter when all modules across all layers share the same vector.
Reinforcement Learning: The Catalyst for Tiny Updates
A core finding of the research is that Reinforcement Learning (RL) is fundamentally more efficient than Supervised Finetuning (SFT) at extremely low parameter counts. The research team reports that models trained via SFT require updates 100 to 1,000 times larger to reach the same performance as those trained with RL.
This gap is attributed to the ‘information density’ of the training signal. SFT forces a model to absorb many bits of information—including stylistic noise and irrelevant structures of human demonstrations—because its objective treats all tokens as equally informative. In contrast, RL (specifically Group Relative Policy Optimization or GRPO) provides a sparser but cleaner signal. Because rewards are binary (e.g., exact match for a math answer), reward-relevant features correlate with the signal while irrelevant variations cancel out through resampling.
Optimization Guidelines for Devs
The research team isolated several strategies to maximize the efficiency of tiny updates:
- Optimal Frozen Rank (r): Analysis showed that a frozen SVD rank of r=2 was optimal. Higher ranks introduced too many degrees of freedom, complicating the optimization of the small trainable vector.
- Tiling vs. Structured Sharing: The research team compared ‘structured’ sharing (modules of the same type share parameters) with ’tiling‘ (nearby modules of similar depth share parameters). Surprisingly, tiling was more effective, showing no inherent benefit to forcing parameter sharing exclusively between specific projections like Query or Key modules.
- Precision: In bit-constrained regimes, storing parameters in fp32 proved most performant bit-for-bit, even when accounting for its larger footprint compared to bf16 or fp16.
Benchmark Performance
The research team reports that Qwen-2.5 models often needed around 10x fewer updated parameters than LLaMA-3 to reach similar performance in their setup.
On harder benchmarks like MATH500 and AIME24, 196-parameter updates for Qwen2.5-7B-Instruct retained 87% of the absolute performance improvement of full finetuning across six difficult math benchmarks.
Key Takeaways
- Extreme Parameter Efficiency: It is possible to train a Qwen2.5-7B-Instruct model to achieve 91.8% accuracy on the GSM8K math benchmark using only 13 parameters (26 total bytes).
- The RL Advantage: Reinforcement Learning (RL) is fundamentally more efficient than Supervised Finetuning (SFT) in low-capacity regimes; SFT requires 100–1000x larger updates to reach the same performance level as RL.
- TinyLoRA Framework: The research team developed TinyLoRA, a new parameterization that uses weight tying and random projections to scale low-rank adapters down to a single trainable parameter.
- Optimizing the “Micro-Update”: For these tiny updates, fp32 precision is more bit-efficient than half-precision formats , and “tiling” (sharing parameters by model depth) outperforms structured sharing by module type.
- Scaling Trends: As models grow larger, they become more ‘programmable’ with fewer absolute parameters, suggesting that trillion-scale models could potentially be tuned for complex tasks using just a handful of bytes.
Check out the Paper. Also, feel free to follow us on Twitter and don’t forget to join our 120k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
