Put a fast model behind the wrong gateway and your user stares at an empty chat bubble for a second and a half. The model already sent its first token; the gateway is holding it. We measured exactly that happening.
If you run models in production, you end up with a gateway eventually. A good one routes every provider through a single URL, keeps API keys off developer laptops, meters spend per team, and scans traffic before it leaves. The catch is that it sits inside every request your users make. So before you pick one, you want to know what it does to the moments they actually feel.
In Part 1 we measured raw gateway cost against an instant backend. Nobody ships an instant backend, so this time it behaves like a real model: first token around 300 ms, a two-second turn, tokens streamed one at a time. In front of it: Highflame (Rust), Bifrost (Go), and LiteLLM (Python).
Three moments decide the experience:
- The first token. Does the app still feel instant?
- Peak hour. When everyone talks at once, do calls come back?
- The tool call. What does an agent pay per action?
Moment one: the first token
The silence before the first token is where a user decides whether the app hung. The model owns almost all of that silence: with no gateway anywhere in sight, it takes about 300 ms to produce its first token. The only question for a gateway is what it adds on top of that.
Highflame adds about 2 ms. That is the whole toll, measured at ten chats where noise is smallest, and it is paid once at the front: the completed answer lands within single-digit milliseconds of the direct stream at every concurrency we ran, so no cost accumulates per token. At a hundred simultaneous chats the first token through Highflame lands in 307 ms, indistinguishable from hitting the model with no gateway at all.
The failures are worth naming, because you would never catch them on a spec sheet:
- Bifrost buffers. In its stock configuration it forwards the stream but holds every byte until the model finishes, so the first token shows up together with the last one. That is 1.3 seconds of gateway-added wait on every turn, and the UI reads as frozen even though tokens were flowing the whole time.
- LiteLLM compounds. It streams properly at ten chats, but its per-token work stacks up under load. At fifty concurrent chats it adds about 350 ms to the first token; at a hundred, the median first token slips past six seconds, and the streamed answer that takes 1.6 seconds direct takes 7.7 through it.
Moment two: everyone talks at once
Real conversations hold connections open. Each request waits about two seconds for its turn to finish, so a busy afternoon (agent fleets fanning out, a CI run, the whole company inside the same hour) turns into thousands of sockets held open at once.
We held 5,000 conversations open simultaneously for five minutes, every one waiting on a two-second model turn. Highflame answered all 572,563 requests it was offered, a true 100%, with a p99 of 3.3 seconds, and it carried the whole rush in 733 MB of memory. Bifrost answered everything too, but needed twice the memory to do it. Holding the rush is what a compiled gateway should do; holding it on half the machine is the difference.
The rush separates the bottom of the field. LiteLLM answered 30% of its requests and made the survivors wait a median of 6.2 seconds, with a p99 of 26.7 seconds. The missing seven in ten calls simply never came back, which in production is a user re-sending the prompt into a system that is already drowning.
The top separates the moment the model’s two seconds stop hiding the gateways. In Part 1 we rushed the same 5,000 connections at instant answers: Highflame answered at a p99 of 0.83 seconds to Bifrost’s 0.99, both at 100%. Held at full throttle for five minutes, Highflame moved 14,331 requests a second with zero failures across 4.3 million requests, a little over twice Bifrost’s rate.
The trap for a buyer is that at 100 held conversations all three gateways still answer everything: two-second responses, 100% success. At 500, LiteLLM’s median is already two full seconds over the model’s own turn while the other two sit flat. The differences live at the scale you are planning to grow into, which is exactly the scale you will not test by hand before picking one.
Moment three: the tool call
Agents spend their time calling tools, and MCP is how those calls travel. Here the field thins out before the race starts: Highflame is the only one of the three we could place transparently between an MCP client and an MCP server, with the client’s own session passing straight through. Bifrost’s MCP support fronts tools to LLM calls rather than to a standalone MCP client, and LiteLLM’s MCP gateway would not give us an equivalent transparent pass-through to measure.
So the bar is the tool itself, and the only number that matters is what the proxy adds over it. Highflame adds 17 ms at ten concurrent sessions. At a hundred sessions it adds 181 ms, most of that sessions queueing for the same tool rather than gateway work, while delivering 260 calls a second with zero failures.
A hundred sessions is also nowhere near the ceiling. Pushed with a dedicated load rig against an instant tool, a single Highflame instance on four cores peaks at 8,200 tool calls a second and holds around 6,800 with a thousand concurrent agent sessions on it. And the part a capacity planner should care about: the same tool server hit directly, with no gateway at all, buckled under that thousand-session fan-in.
That headroom matters because the MCP proxy is where Highflame does its governance work: it authenticates the caller, and it is the enforcement point where Cedar policy checks and Shield’s scanners run when you turn them on.
It fits where your agents run
Highflame boots at 25 MB and walked into the 5,000-conversation rush carrying 66 MB; the rush itself peaked at 733 MB and settled back down after. LiteLLM entered the same rush already holding 486 MB, seven times Highflame’s working set, and answered 30% of it.
A footprint like that is what lets you run the gateway as a sidecar next to the workload, or one per agent sandbox, instead of as one central box everything must reach.
Picking your first gateway
If you have not put a gateway in front of your models yet, three questions matter, in the order your users feel them. Highflame’s answers:
- Does the first token still feel instant? Yes. The model takes its ~300 ms; Highflame adds about 2 ms on top, and at a hundred concurrent chats the two are indistinguishable.
- Does peak hour drop calls? No. All 572,563 requests answered across five minutes of 5,000 held conversations, in 733 MB of memory.
- Do tool calls pay a toll? Highflame adds 17 ms to a 200 ms tool call, and none fail.
What Highflame actually inspects on that hot path is on the platform page.
The setup, for the record
The rig is two AWS hosts: the gateway and the simulated model share a 4-core c7i.xlarge, the load generator runs on its own 8-core box, and every request crosses a real network. The model times like a real one (~300 ms to first token, two-second turns, streamed tokens). Every gateway ran its stock build with its default config, guardrails off, from a fresh boot, measured on 2026-07-22; every five-minute number is five unbroken minutes. The MCP ceiling came from a separate three-host rig. Part 1’s full dataset lives on the benchmarks page.
