Together AI Releases Together Compiler: Agents Make Inference Up to 2.6x Faster

Together AI has introduced Together Compiler (TKCC), an agent-driven system that keeps re-optimizing inference as production traffic changes. Together Compiler already runs on several of Together’s serverless endpoints. The core claim for developers: an engine tuned once goes stale, so tuning should never stop.

TL;DR

System: It is an optimization system, tested on Kimi K3, GLM 5.3, MiniMax M3.1 Flash, DeepSeek v4.1 Flash and Llama 1B.
Runs on: NVIDIA B200, B300, GB300, H100, AMD MI355X and AWS Trainium2. Each run used at most 2 nodes per model.
Performance: Up to 2.6x higher TPS than the best published InferenceX configurations, in under 24 hours per setting.
Best: 1,111 P90 TPS per user on MiniMax M3.1 Flash (B200, batch size 1), 1.8x over the 633 TPS baseline.
Worst: GLM 5.3 on MI355X moved only from 463 to 480 TPS (about 4%).

What is Together Compiler?

Together Compiler is a continual compilation loop from Together AI that uses coding agents to rewrite a model’s forward pass. It takes a server launch command, a traffic profile and a hardware configuration. Agents propose faster forward passes, and a persistent engine measures each one. The winning code ships into the serving engine. When traffic drifts, the loop runs again.

Why does inference need continual compilation?

Together AI team reports that its traffic shape never sits still. Prompt lengths in one country ran about 97K tokens versus 70K elsewhere. One customer’s P90 input length grew from 18K to 33K tokens in 3 days.

Kernels tuned on random data can also mislead. Random inputs activated 13 to 38% more MoE experts than real traffic. One agent-written top-k kernel took 20 µs on random data. On real data at 70K context, it took up to 62 ms.

How does the persistent engine work?

Most agent loops restart vLLM, SGLang or TensorRT-LLM for every experiment. The research team measures 761 seconds of cold startup for MiniMax M3.1 Flash on TensorRT-LLM. Together Compiler starts the engine once. Agents submit a candidate forward pass over RPC. The server swaps the forward function pointer and re-captures CUDA graphs.

Two measurement tiers balance speed and fidelity. A fast tier serves 12 layers for quick screening. A slow tier serves 60 layers to confirm gains on real distributions. Two tiers tested 8.1x more candidates than one slow tier (1,354 vs 168). They also found a solution 7.2% faster. Together AI team reports 252x higher agent experiment throughput than an engine-restart baseline.

How does it choose a parallelism layout?

A cost model ranks tensor, expert, data and context parallel layouts before agents start. It mixes spec-sheet numbers, microbenchmarks and measured forward passes. It also uses measured MoE expert imbalance and uneven request lengths. On Kimi K3, it found a production layout with up to 24% higher TPS.

What did agents need from human experts?

An expert-curated knowledge base produced 25 to 31% lower speedup after 12 hours. Naive agents told to make the forward pass “as fast as possible” beat expert strategies. The ensemble scores were 6.2x versus 4.9x after 10 trials. Custom logging structures ate up to 43% of tool calls. Together AI now gives agents raw engine source code, not an intermediate representation.

Benchmarks: how fast is Together Compiler?

On SemiAnalysis InferenceX‘s AgentX workload, Together Compiler beat the best open configurations by up to 2.6x TPS. Each setting ran under 24 hours on at most 2 nodes.

Batch size 1, P90 TPS per user (Together Compiler vs InferenceX best, self-reported):

MiniMax M3.1 Flash (B200): 1,111 vs 633
DeepSeek v4.1 Flash (GB300): 1,205 vs 943
GLM 5.3 (B200): 417 vs 329
Kimi K3 (GB300): 450 vs 423
GLM 5.3 (MI355X): 480 vs 463

For MiniMax, the cost model picked TP8 over the baseline’s TP4. That halves each GPU’s weight traffic in memory-bound decode.

On Llama 1B at batch size 1, it beat Hazy Research megakernels, Mirage and CUDA-graph baselines by up to 1.7x. Beyond LLMs, it sped up 8 scientific models by 1.5x to 6.4x with bit-exact outputs. Claude Code with /goal reached 1.2x to 4.4x and failed on AIFS-ENS 2.0.

<div class="panel on" data-p="0">
<div class="flow">
<div class="node on" data-s="0"><span class="n">01</span><b>Traffic drifts</b>prompt lengths shift</div>
<div class="node" data-s="1"><span class="n">02</span><b>Cost model</b>ranks TP / EP / DP / CP</div>
<div class="node" data-s="2"><span class="n">03</span><b>Parallel agents</b>rewrite forward pass</div>
<div class="node" data-s="3"><span class="n">04</span><b>2-tier engine</b>hot reload + measure</div>
<div class="node" data-s="4"><span class="n">05</span><b>Deploy</b>best code ships</div>
</div>
<div class="detail" id="tk-d"></div>
<div class="ctrl"><button class="btn ghost" id="tk-prev">← Back</button><button class="btn" id="tk-next">Next step →</button><div class="prog"><i id="tk-pg"></i></div><button class="btn ghost" id="tk-auto"> Auto-play</button></div>
</div>

<div class="panel" data-p="1">
<div class="legend">Time before a candidate can be measured (MiniMax M3.1 Flash, TensorRT-LLM, paper Table 1)</div>
<div class="row"><span class="lb">Cold engine start (status quo)</span><div class="track"><div class="bar b0" data-w="100"></div></div><span class="vv">761 s</span></div>
<div class="row"><span class="lb">Hot reload, slow tier (60 layers)</span><div class="track"><div class="bar b2" data-w="5.9"></div></div><span class="vv">45 s</span></div>
<div class="row"><span class="lb">Hot reload, fast tier (12 layers)</span><div class="track"><div class="bar b1" data-w="2.2"></div></div><span class="vv">17 s</span></div>
<div class="grid2" style="margin-top:12px">
<div class="stat"><div class="v">252x</div><div class="k">more agent experiments than an engine-restart baseline</div></div>
<div class="stat"><div class="v">8.1x</div><div class="k">more candidates with 2 tiers vs 1 slow tier (1,354 vs 168)</div></div>
<div class="stat"><div class="v">7.2%</div><div class="k">faster final solution from the two-tier setup</div></div>
<div class="stat"><div class="v">11x</div><div class="k">lower idle time per agent</div></div>
</div>
</div>

<div class="panel" data-p="2">
<div class="sel"><button class="tab on" data-m="p90">P90 TPS per user, batch size 1</button><button class="tab" data-m="llama">Llama 1B, batch size 1 (B200)</button></div>
<div class="legend" id="tk-lg"></div>
<div id="tk-bars"></div>
<p class="note">Self-reported by Together AI. InferenceX baselines as of Oct 27, 2026. Under 24 hours, at most 2 nodes per model.</p>
</div>

<div class="panel" data-p="3">
<div class="grid2">
<div class="card"><h4>Knowledge base <span>+</span></h4><div class="more">An expert-curated knowledge base gave <span class="big">25 to 31%</span> lower speedup after 12 hours. Agents wasted time searching it.</div></div>
<div class="card"><h4>Expert strategies <span>+</span></h4><div class="more">Naive agents ("make it as fast as possible") beat expert-assigned strategies: <span class="big">6.2x vs 4.9x</span> after 10 trials.</div></div>
<div class="card"><h4>Custom logging <span>+</span></h4><div class="more">Human-designed research logs consumed up to <span class="big">43%</span> of tool calls per trial.</div></div>
<div class="card"><h4>No intermediate representation <span>+</span></h4><div class="more">Raw engine source let agents rewrite Kimi K3’s MLA metadata kernel for a <span class="big">1.7 to 2.5x</span> kernel speedup.</div></div>
<div class="card"><h4>Megakernels on GPUs <span>+</span></h4><div class="more">Rarely paid off: only <span class="big">0.22%</span> faster on AMD GLM 5.3. Trainium2 was the exception.</div></div>
<div class="card"><h4>Still the bottleneck <span>+</span></h4><div class="more">Agents spend over <span class="big">50%</span> of their time waiting on measurement gates.</div></div>
</div>
</div>

<div class="foot"><span>Sources: Together AI paper and blog</span><span class="mtp">© Marktechpost</span></div>
</div>
<script>
(function(){
var R=document.getElementById(‘tkcc-x’);
function rs(){try{parent.postMessage({tkccH:R.offsetHeight+40},’*’)}catch(e){}}
var steps=[
‘Together reports traffic keeps changing. One customer\’s P90 input length grew from <em>18K to 33K</em> tokens in 3 days. Random test data activates <em>13 to 38%</em> more MoE experts than real traffic.’,
‘A cost model ranks parallelism layouts before agents start. It blends spec sheets, microbenchmarks and measured passes. On Kimi K3 it found a layout with up to <em>24%</em> higher TPS.’,
‘Multiple coding agents work in parallel and share findings on a message board. High-frequency communication made search <em>4.7x</em> faster in ablations.’,
‘The engine starts once. Each candidate is hot-reloaded: swap the forward pointer, re-capture CUDA graphs. A <em>12-layer</em> fast tier screens; a <em>60-layer</em> slow tier confirms.’,
‘After the budget runs out, verification tests check correctness and KV cache safety. The best config and forward pass go to production. Result: up to <em>2.6x</em> TPS over InferenceX best.’];
var cur=0,auto=null,nodes=R.querySelectorAll(‘.node’);
function show(i){cur=(i+5)%5;nodes.forEach(function(n,k){n.classList.toggle(‘on’,k===cur)});R.querySelector(‘#tk-d’).innerHTML=steps[cur];R.querySelector(‘#tk-pg’).style.width=((cur+1)*20)+’%’;}
nodes.forEach(function(n,k){n.onclick=function(){show(k)}});
R.querySelector(‘#tk-next’).onclick=function(){show(cur+1)};
R.querySelector(‘#tk-prev’).onclick=function(){show(cur-1)};
R.querySelector(‘#tk-auto’).onclick=function(){var b=this;if(auto){clearInterval(auto);auto=null;b.textContent=” Auto-play”;b.classList.remove(‘pulse’)}else{auto=setInterval(function(){show(cur+1)},2600);b.textContent=”❚❚ Pause”;b.classList.add(‘pulse’)}};
show(0);
var data={p90:{lg:'<i style="background:linear-gradient(90deg,#5b7cff,#ff4f8b)"></i>Together Compiler<i style="background:#3a3d6e"></i>InferenceX best’,max:1300,rows:[[‘MiniMax M3.1 Flash · B200’,1111,633],[‘DeepSeek v4.1 Flash · GB300’,1205,943],[‘DeepSeek v4.1 Flash · B200’,813,671],[‘Kimi K3 · GB300’,450,423],[‘GLM 5.3 · MI355X’,480,463],[‘GLM 5.3 · B200′,417,329]]},
llama:{lg:'<i style="background:linear-gradient(90deg,#5b7cff,#ff4f8b)"></i>Together Compiler<i style="background:#3a3d6e"></i>Best baseline per length (megakernel or CUDA graph, output tokens/s)’,max:2300,rows:[[‘OSL 256’,2118,1649],[‘OSL 1,024’,2011,1569],[‘OSL 4,096’,1945,1414],[‘OSL 16,384’,1871,1123],[‘OSL 65,536′,1482,892]]}};
function bars(m){var d=data[m],h=””;d.rows.forEach(function(r){h+='<div class="row"><span class="lb">’+r[0]+'</span><div style="flex:1"><div class="track" style="height:20px;margin-bottom:3px"><div class="bar b1" data-w="’+(r[1]/d.max*100)+’">’+r[1].toLocaleString()+'</div></div><div class="track" style="height:20px"><div class="bar b0" data-w="’+(r[2]/d.max*100)+’">’+r[2].toLocaleString()+'</div></div></div></div>’});
R.querySelector(‘#tk-lg’).innerHTML=d.lg;R.querySelector(‘#tk-bars’).innerHTML=h;grow(R.querySelector(‘#tk-bars’));}
function grow(el){var b=el.querySelectorAll(‘.bar’);b.forEach(function(x){x.style.width=”0″});setTimeout(function(){b.forEach(function(x){x.style.width=(x.textContent?Math.max(x.dataset.w,9):Math.max(x.dataset.w,1.5))+’%’})},60);setTimeout(rs,1200);}
R.querySelectorAll(‘[data-m]’).forEach(function(t){t.onclick=function(){R.querySelectorAll(‘[data-m]’).forEach(function(z){z.classList.remove(‘on’)});t.classList.add(‘on’);bars(t.dataset.m)}});
R.querySelectorAll(‘.tabs .tab’).forEach(function(t){t.onclick=function(){var i=t.dataset.t;R.querySelectorAll(‘.tabs .tab’).forEach(function(z){z.classList.toggle(‘on’,z===t)});R.querySelectorAll(‘.panel’).forEach(function(p){p.classList.toggle(‘on’,p.dataset.p===i)});if(i===’1′)grow(R.querySelector(‘[data-p="1"]’));if(i===’2′)bars(R.querySelector(‘[data-m].on’).dataset.m);rs();}});
R.querySelectorAll(‘.card’).forEach(function(c){c.onclick=function(){c.classList.toggle(‘open’);c.querySelector(‘h4 span’).textContent=c.classList.contains(‘open’)?’−’:’+’;setTimeout(rs,400)}});
window.addEventListener(‘load’,rs);window.addEventListener(‘resize’,rs);setTimeout(rs,300);
})();
</script>
</body></html>
” style=”width:100%;max-width:880px;height:600px;border:0;display:block;margin:0 auto;” loading=”lazy” title=”Together Compiler interactive explainer”>
window.addEventListener(“message”,function(e){if(e.data&&e.data.tkccH){var f=document.getElementById(“tkcc-frame”);if(f)f.style.height=e.data.tkccH+”px”;}});

How does it compare with other agentic optimization systems?

MetricsTogether Compiler Cursor + NVIDIA multi-agent AMD GEAK Apple RSI-Inference BenchTypeProduction optimization loopMulti-agent kernel optimizerTriton kernel agent frameworkBenchmark for coding agentsScopeFull forward pass inside the serving engine235 standalone CUDA kernels (SOL-ExecBench)Individual Triton kernels37 tasks editing vLLM or SGLangMeasurementPersistent engine, hot reload, 2 tiers, real trafficSOL-ExecBench on 27 B200 GPUsTritonBench-revised and ROCm benchmarkVerifier builds the engine and serves the modelHardwareB200, B300, GB300, H100, MI355X, Trainium2B200AMD Instinct MI300XH100, B200Main resultUp to 2.6x TPS vs InferenceX best1.38x geomean, wins on 149 of 235Up to 2.59x speedup, up to 63.33% correctnessScores agents; no single system resultTime budgetUnder 24 hours per setting3 weeksNot disclosedUp to 8 hours per taskAvailabilityLimited beta for Together customersResults in public repoOpen sourceApple Sample Code License

Key Takeaways

Together Compiler re-optimizes inference continually as traffic shifts.
Persistent engines with hot reload deliver 252x more agent experiments.
Up to 2.6x TPS over the best InferenceX configs within 24 hours.
Naive agents beat expert knowledge bases and curated strategies.
Self-reported; gains range from about 4% to 2.6x across settings.

Check out the paper, the Together Compiler blog and the Polymorphic Inference post. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us

The post Together AI Releases Together Compiler: Agents Make Inference Up to 2.6x Faster appeared first on MarkTechPost.

Source link

Together AI Releases Together Compiler: Agents Make Inference Up to 2.6x Faster

China Speeds Up a Comprehensive AI Law as Users Pass 700m

Powering up for a pivotal moment | MIT News

Meet Factory Droid 1.0: A Model-Independent Coding Agent Harness Built on Script Mode

OpenAI’s funding round shrinks, plus 4 more

MIT affiliates named 2026 Schmidt Sciences AI2050 Fellows | MIT News

Edge0 Releases Audio8-TTS-Realistic, a 0.6B Voice Cloning TTS Model

Beginner’s Guide to Dropshipping in 2027 (Full Course)

Trump-Linked Tokens Crash as Democrats Win Midterms

Bitcoin and gold surge after US elections, but $378 million short squeeze casts doubt on debasement trade

Bitcoin Defies Selling Pressure as $75K Holds the Line

Justin Sun Seeks $264K Sanctions Against World Liberty

Top Insights

Binance’s Spot Market Share Drops to 42% After $6.3B Hack as OKX Gains

Higher Rates Are Back: Can Brookfield Infrastructure Keep Growing Its Payout?

Together AI Releases Together Compiler: Agents Make Inference Up to 2.6x Faster

TL;DR

What is Together Compiler?

Why does inference need continual compilation?

How does the persistent engine work?

How does it choose a parallelism layout?

What did agents need from human experts?

Benchmarks: how fast is Together Compiler?

How does it compare with other agentic optimization systems?

Key Takeaways

Related Posts