The morning your whole team opens Claude Code at once, a tool call that should come back in a quarter of a second can take thirty-seven. The model is fine; the wait lives in the gateway in front of it.
Most of the day that gateway is invisible. A hundred developers doing normal work average about one request every twenty seconds, and any gateway ever written can serve that. The one you choose only starts to matter in the few seconds when everyone lands together: the morning rush, a CI run fanning out, three agent workflows firing at once.
So we measured exactly those seconds. Five gateways, the same AWS hardware, the same burst.
Forget the average
Real developer traffic is spiky. Everyone starts their morning inside the same half hour. A CI run fans out a dozen tool calls at once. An agentic workflow chains six calls back to back inside a single session, and three of those workflows fire together. For a few seconds at a time, “one request every twenty seconds” turns into hundreds of requests in flight, and then it falls back to nothing.
Those spikes are the only moments your gateway is under real pressure, and they are exactly the moments a developer is watching a tool call, waiting for it to come back.
How we ran it
We rebuilt the benchmark Maxim publishes for their Bifrost gateway, on the same class of AWS hardware they used (a t3.medium and a t3.xlarge in us-east-1), so the result could not be waved off as a different setup. Five gateways went through it: our own, Highflame, plus Bifrost, Portkey, Kong, and LiteLLM. Every number below is the median of three runs on real cloud hardware, not a laptop.
Under steady load, Highflame barely moves
Before the burst, the steady-state version. Can the gateway hold a heavy, constant rate without falling behind? We aimed 5,000 requests a second at each one for a minute.
| Gateway | Sustained rps (target 5,000) | p50 | p99 | Success |
|---|---|---|---|---|
| Highflame | 4,999.9 | 1.14 ms | 13.29 ms | 100.00% |
| Bifrost | 4,998.4 | 6.98 ms | 30.07 ms | 99.99% |
| Kong | 1,546.3 | 28.37 ms | 65.34 ms | 99.95% |
| Portkey | 306.0 | 155.06 ms | 271.22 ms | 99.73% |
| LiteLLM | 95.5 | 490.54 ms | 978.97 ms | 99.13% |
Only Highflame held the full 5,000 a second at 100.00% success, with a 1.14ms median. Bifrost matched the throughput but ran about six times slower at that median, and a little over twice as slow at p99. The other three never came close: Kong topped out near 1,500, Portkey near 300, LiteLLM near 95. Neither front-runner was straining here, so the gap between the two is a real latency difference and not one of them drowning, which makes this the cleanest comparison in the whole exercise.
Headroom for when you scale
Maxim’s test stops at 5,000, where Highflame and Bifrost both cruise. So we kept pushing: 10,000, then 20,000 requests a second at the same box, to find where each one actually tops out.
Highflame tops out near 8,200 requests a second to Bifrost’s 5,770: on identical hardware, roughly 42% more headroom before it caps. And it does not buy that headroom with latency. At the very edge Highflame holds a p99 around 12ms while Bifrost sits near 27ms. Your team will never sit at 8,000 requests a second. It will spike toward the ceiling now and then, and when it does you want the gateway to flatten out, not fall over.
Now everyone hits at once
Here is the version that matches the morning-standup moment: 500 requests in flight at once, the momentary crush a bursty team produces. This is the response time a developer actually feels, at p99.
Highflame answers the burst in 0.23 seconds, the quickest of the five, with Bifrost a hair behind at 0.25. Then the floor drops out. The same tool call, on the same hardware, takes nearly seven seconds through Portkey and almost forty through LiteLLM, which also began dropping requests, its success rate sliding to about 93%. That is not a slightly slower response. That is a developer watching a spinner, giving up, and re-running the command.
The pattern comes down to what actually runs the request.
The two natively concurrent compiled gateways, Highflame on Rust and Bifrost on Go, stay in the low hundreds of milliseconds even under the crush. The two built on an interpreted or single-threaded managed runtime fall off a cliff, into whole seconds. Kong, an nginx-based proxy, lands in between: a real cost, no collapse. We have a theory about why Portkey’s tail explodes the way it does, but we have not put a profiler on it, so we will let the number stand and save the diagnosis for another day.
What this buys your team
Put Highflame on the hot path and the burst your team makes every morning never turns into a support ticket. Tool calls come back in about the time it takes to notice they left, at the rush and at the ceiling both, so nobody sits on a spinner or re-runs a command to check whether the agent hung. And because it holds latency flat as load climbs, the gateway becomes one thing you do not have to revisit the next time headcount doubles.
Caveats
The load generator ran on the same box as each gateway, the same way for all five, so the comparison between them is fair; whether that matches every detail of Maxim’s own rig, we cannot say, so treat our numbers against theirs with that in mind. And the response-time-under-burst figures are what a person feels when a system is briefly overloaded, not a clean “gateway overhead” number. The clean overhead comparison is the 5,000-a-second table up top, and the full five-gateway breakdown lives on our benchmarks page.
Why this is the gateway to run
Speed is only half of it. Highflame does more than move bytes between a developer and a model: it inspects every LLM and MCP call inline, checks it against your policy, and only then forwards it. Inspection that costs a developer real time is inspection a team quietly switches off. Leading that tail-latency chart is what lets Highflame run that detection on the hot path without ever being the reason your developers wait.
The speed is the easy part to measure. What Highflame looks for in each of those calls is where it gets interesting. See what runs on the hot path.
