Busbar

Comparison

Busbar vs LiteLLM

Everybody evaluating Busbar knows LiteLLM, so this page is specific about what's actually different. It isn't a feature list. LiteLLM is genuinely good at what it is: an AI gateway, a Python-native router with huge provider coverage. Busbar is a different kind of tool: an AI control plane in a single Rust binary. It covers more than LLMs (chat, embeddings, images, audio, moderations, and rerank across every provider) and does more than route (failover, breakers, governance, observability). The difference shows up in three places: how protocols are handled, what happens when a provider fails, and what the layer itself costs you at runtime. Every claim about LiteLLM below is cited to their own documentation, and every claim about Busbar to ours.

The short version

BusbarLiteLLM
Core format Superset core; all six protocols first-class, no privileged one OpenAI-shaped core [7]; native ingress built endpoint by endpoint [4]
Native ingress + load balancing Every protocol routes through pools with failover /v1/messages routes cross-provider [4]; Gemini/Bedrock native routes stay within their provider [9]; raw pass-throughs bypass fallbacks [5]
Bedrock SDK ingress A signed boto3 client, load-balanced across non-Bedrock backends, response re-encoded as valid binary eventstream boto3 /converse and /invoke load-balance across Bedrock deployments only; no cross-provider egress [9]
Failure classification Fault-attributed: provider outage, auth/billing, client fault, and context-length each handled differently, per (pool, lane) Exception hierarchy feeding retry counts and a cooldown timer [3]
Recovery Single-flight half-open probe, exponential backoff with jitter, Retry-After honored as a floor Deployment returns when cooldown_time expires [3]
Failover boundary In-flight, before the first byte, across protocol families Retry after the failed call returns [3]
Runtime One static Rust binary; nothing else to run Python/Uvicorn workers + PostgreSQL [2]; Redis for multi-instance state [3]
Added latency 0.038 ms median, 0.045 ms p99 (small call); 0.084 ms median, 0.158 ms p99 translating a 12k-token body cross-protocol, from our published benchmark [6] 2 ms median, 8 ms p95, 13 ms p99, from their published benchmark [1]
Memory ~5.6 MB idle (v1.3.2), from our published numbers [8] ~359 MB peak (Python), ~31.7 MB (Rust core, beta), from their published numbers [7]
Throughput 19,505 req/s on a single 2-vCPU machine (~9,750 req/s per vCPU), from our published benchmark [6] ~1,170 req/s from a 4-machine deployment, 4 vCPUs each (16 total, ~73 req/s per vCPU), from their published benchmark [1]
Provider coverage 113 verified preconfigured, plus any provider on the six protocols via your own providers.yaml 100+ preconfigured out of the box
Observability Prometheus, OTLP, admin API; bring the Grafana you already run Bundled admin UI

In plain terms: our worst published latency case beats their best by more than 20×, and one Busbar vCPU handles about as many requests per second as 130 LiteLLM vCPUs (9,750 req/s against ~73 req/s). Both ratios measure the same thing: how much CPU a request costs.

Benchmark figures are each project's own published numbers on its own hardware, so read the latency and throughput rows as orders of magnitude, not a stopwatch. Busbar's harness is checked into the repo under bench/, and the opt-in Server-Timing header lets you measure Busbar's overhead on your own traffic instead of trusting either table. Full methodology.

No privileged protocol

LiteLLM's core normalizes everything into the OpenAI format ("one OpenAI-compatible API," in their own words [7]), and its native-format ingress is built endpoint by endpoint on top of that core. Where an endpoint has been built, it routes: Anthropic /v1/messages reaches other providers [4], while the Gemini and Bedrock native endpoints load-balance within their own provider [9]. The raw pass-through routes forward without fallbacks [5]. What you don't get, on any of them, is what the six-protocol matrix gives by construction: any native ingress reaching any backend.

Busbar doesn't make you pick, on any protocol. All six wire protocols (OpenAI, Anthropic, Gemini, Bedrock, Cohere, Responses) are first-class in both directions, through one superset intermediate representation: an internal format rich enough to hold every protocol's features, which each protocol maps into and out of, so nothing is flattened to any vendor's shape. When ingress and egress match, the body passes through byte-for-byte. An OpenAI SDK and a signed Bedrock SDK can hit the same pool at the same time, each gets its own native response back, and failover works underneath both.

That Bedrock case is worth dwelling on, because it's the hardest one: Busbar verifies the inbound AWS SigV4 signature, translates the Converse request through the IR to whichever backend the pool selects, then re-encodes the streamed response as binary application/vnd.amazon.eventstream frames with valid CRC32 checksums, so boto3 parses it as if Bedrock itself had answered. We haven't found this anywhere else. How translation works.

What that means in practice: Busbar is valuable before the second provider exists. A team locked to Bedrock today (a BAA that covers AWS but not yet Anthropic, say) points boto3's endpoint_url at Busbar and changes nothing else; same-protocol traffic passes through byte-for-byte, and no new entity enters the data path, because Busbar runs in your infrastructure. The day a direct Anthropic BAA lands, an Anthropic lane is one config edit, and the application code never learns it happened. With LiteLLM, Bedrock-SDK ingress load-balances across Bedrock deployments only [9], so that same migration starts with rewriting your integrations.

100+ providers vs 6 protocols is not the same metric

The counts look lopsided until you notice they measure different things. A provider count measures countries; protocols are languages. LiteLLM lists 100+ countries. Busbar speaks six languages, and one of them, OpenAI-compatible, is the lingua franca that nearly every hosted and local provider exposes: Groq, Mistral, Together, DeepSeek, Fireworks, xAI, Ollama, vLLM, and most of the rest of the list. Any provider speaking one of the six is a few lines in your own providers.yaml: a name, a base_url, the env var holding its key. Adding a provider takes about a minute, and you don't wait on us to do it.

So the genuine difference is narrower than the counts suggest. LiteLLM's catalog has the entry typed for you; with Busbar you may type it yourself. The verified catalog Busbar ships is deliberately curated rather than scraped, because each entry's error-code mappings feed the breaker's fault attribution. The one real gap is a provider whose wire format is none of the six and compatible with none: that needs new translator code, and it's the single case where you'd wait on anyone.

A real circuit breaker, not a cooldown timer

LiteLLM does classify errors: it maps provider failures into an exception hierarchy with per-exception retry policies [3]. The difference is what classification drives. There, it feeds retry counts and a cooldown timer; a cooled-down deployment returns when the timer expires. Here, it feeds a state machine with distinct dispositions:

Breaker state is per pool, per lane, so a lane rate-limiting in one pool can still serve another where it's healthy. Recovery is a single half-open probe request, not a thundering herd. And a failing request reroutes to the next lane before your client sees a single byte, even on a streaming request. No Redis, no coordination layer, it's all in the binary; LiteLLM uses Redis to share cooldown state across multiple instances [3]. The full state machine.

The control plane itself is basically free

One static Rust binary, no interpreter, no garbage collector in the request path. Median added latency is in the tens of microseconds, and stays under 100 µs even translating a 12k-token body between protocols in both directions [6]. LiteLLM's own published benchmark reports 2 ms median and 13 ms p99 of proxy overhead [1]: different harness, different hardware, but a different order of magnitude too.

And you don't have to take either number on faith. Set one config flag (observability.emit_server_timing: true) and Busbar reports its own overhead on every response in a standard Server-Timing header, so you can check the number on your own traffic instead of trusting our benchmark. Benchmark and methodology.

About LiteLLM's Rust rewrite

LiteLLM is rebuilding its data plane in Rust, and their launch post says why in plain terms: "Under real load, CPU and memory climb with concurrency, and pods get OOM-killed at the worst time" [7]. We think they're right, and the move validates the architecture Busbar has shipped since day one: a compiled, no-GC data plane is what this job requires.

The differences that remain are worth stating precisely. Their Rust core is in early beta, with a staged rollout planned through December 2026 [7]; Busbar has been a single Rust binary since its first release, so there is no migration to wait for or live through. Their plan keeps customer plugins (auth, guardrails, callbacks) in an optional Python sidecar and long-tail providers on Python [7]; Busbar's request path is one process end to end. And a Rust rewrite of an OpenAI-normalized gateway is still an OpenAI-normalized gateway: the hub format, the endpoint-by-endpoint native ingress, and the pass-through split don't change with the implementation language. On the numbers, their published Rust-core overhead (~0.05 ms) lands in the same class as ours; memory (their 31.7 MB peak vs our ~4.5 MB resident) and the single-binary deployment story remain architectural, not implementation, differences [7].

Where LiteLLM is the right call

If that's your shape, use LiteLLM. It's a good tool and we say so without hedging.

If this layer is production infrastructure carrying all of your AI traffic, you need a control plane, not just a gateway, and that's the problem Busbar was built for. Get started in five minutes, or read why Busbar exists.

References

Facts about LiteLLM checked against docs.litellm.ai on July 9, 2026. If something here is out of date, tell us and we'll fix it.

  1. LiteLLM benchmarks: 2 ms median / 8 ms p95 / 13 ms p99 "LiteLLM Overhead Duration"; ~1,170 RPS across 4 instances of 4 vCPU / 8 GB.
  2. LiteLLM production guide: Uvicorn workers, PostgreSQL, deployment guidance, Redis version requirements.
  3. LiteLLM routing and fallbacks: retry-based fallbacks, allowed_fails, cooldown_time, per-exception retry policies, and Redis for tracking cooldown and usage state across instances.
  4. LiteLLM /v1/messages: Anthropic-format ingress routed through the LiteLLM router.
  5. LiteLLM pass-through endpoints: raw native-format routes forwarded to their provider, without fallbacks.
  6. Busbar benchmark: methodology, per-protocol numbers, and the reproduction steps; harness in the repo under bench/.
  7. LiteLLM Rust launch post: the Python problems in their words, the Rust-core numbers, the beta status and staged rollout through December 2026, and the Python sidecar for plugins.
  8. Busbar, in numbers: memory, latency, and throughput measurements on the released binary, and how to reproduce them.
  9. LiteLLM Bedrock Converse endpoint and generateContent endpoint: routed native ingress that load-balances within its own provider's deployments.