Making KrakenD the AI gateway for every API, every model and every agent

Last month we shipped KrakenD 3.0, the first release of Series 3 and, in the team's own words, the biggest jump in the gateway's capabilities since it launched in 2016. KrakenD is our AI API gateway: an open-source gateway that enterprises in government, finance, telecom and healthcare, including Fortune 500 companies, put in front of their services to aggregate, secure and deliver traffic. When it joined the group in September 2025, we said we were laying the foundation for the next generation of AI-powered API gateways. Thirteen months later, this is what that looks like.

Every AI call is an API call

The idea behind 3.0 is that simple. A product feature asking for a completion, an agent calling a backend, a prompt leaving for a provider: all of it is traffic, and governing traffic is what a gateway is for.

Until now, teams plugged AI in their own way. Keys scattered across every service, so nobody could say which app talked to which provider. A token bill discovered at the end of the month, with no way to tie it back to a team. Agents calling backends with no auth, no limits and no logs. A gateway is the natural place to fix all of that, but AI traffic comes with a catch. A model’s answer arrives word by word, and most gateways can’t act on it while it’s moving: they either let it through untouched or hold it until the whole answer is ready, which kills the real-time experience users expect. KrakenD 3.0 changes that. It reads and controls the answer as it flows, so budgets and rules act on every response while it’s still reaching the user. AI governance becomes API governance, run by the gateway you already operate.

How it works

3.0 makes streaming part of the gateway core. Three new encodings, sse, ndjson and eventstream, parse a stream message by message as it passes through, so the quota component, the JMESPath modifier and the response-body modifier act on each message while the client is still receiving it. Nothing buffers. The ai/llm component streams from every supported provider and normalises the stream to the OpenAI or Anthropic wire format with a format option, so a client built against one API can be served by another provider without knowing; KrakenD's own example is pointing Claude Code at the gateway and routing it to Amazon Bedrock.

With streams governable, the rest of the AI layer sits at the point where a request enters. Five components make up that layer.

Prompt Guard inspects the prompt before it leaves for a provider, with three kinds of guard that stack: regular expressions for known injection patterns, CEL policies for rules about the request, and an external classifier that scores each prompt. Each guard carries a severity, and the operator decides whether a request passes or is blocked when the classifier is unreachable. Every check is counted in a krakend.promptguard.check metric through OpenTelemetry, labelled with the guard that blocked it, and written into the trace. That is how false positives get tuned in production rather than guessed at.

The Semantic Cache answers before a provider is called when a new prompt means the same thing as one already answered. "How do I reset my password?" and "I forgot my password, what do I do?" get one answer and one model call. Embeddings come from an ONNX model running inside the gateway, so computing the cache key costs no API call, and Redis holds the vectors. It is a decision per endpoint, because a semantic hit is a bet that two prompts mean the same thing: a good bet for FAQ-shaped traffic, a bad one for prompts carrying a single user's context.

The AI Router chooses the model per request rather than per endpoint. Expression gates match request headers or CEL expressions, the same policy language KrakenD uses for security rules, so a routing decision is deterministic and reviewable like an authorisation rule. A prompt classifier, an integration with Not Diamond, reads the prompt and picks the model it judges best, so a short lookup goes to a cheap, fast model and a long reasoning task to a frontier one. When no gate matches or the classifier cannot decide, a failover provider takes the request. Combined with the new fallback proxy strategy, which tries backends in order and returns the first success, and with quota fallbacks, a cost ceiling becomes configuration: this tenant's budget is spent, send them to the local model.

Underneath sits the LLM Provider Inventory. Each provider is declared once at service level under ai/providers, credentials included, and referenced by name from any endpoint, so swapping a provider is a configuration change rather than a code change in every application. Every LLM metric now carries the model that served the request, so spend is legible per model.

The MCP server, which exposes gateway endpoints as tools for agents, gains the two things production needed. Authentication follows the MCP authorisation specification: the gateway publishes protected resource metadata on its well-known endpoint, a compliant client discovers the authorisation server from it and returns with a token the gateway validates. Tool filtering by audience means each client sees only the tools meant for it, a security control and a cost control at once, since every tool definition a server advertises is sent to the model on every call.

Take a bank's KYC assistant asking whether a counterparty is clean. It calls one tool, get_kyc_dossier, and the gateway assembles the answer from an entity registry over REST, an ownership chain over GraphQL and a sanctions list on a legacy SOAP service, with the tool visible only to the roles allowed to call it, every call on a team budget and every call in OpenTelemetry. One tool per use case, not one per endpoint, and a legacy system becomes something an agent can act on without being rewritten.

The human stays at the configuration layer. Nothing routes, caches or blocks unless an operator declared it in configuration that is versioned and reviewed like code, and the krakend report command packages that configuration, sensitive fields redacted, into one file for support.

What shipped, in numbers

The open-source edition gets the housekeeping a major version is for: wildcard paths in JWT claims, the HTTP method in endpoint logs and the leaner binary. Not all of it is addition: plugins leave the Community Edition, configuration files move to "version": 4 or the gateway refuses to start, and a component declared under a legacy namespace is no longer accepted. Most 2.x setups upgrade with a handful of edits; the upgrade guide lists every removal and its replacement.

Across the group, the implication is the point of this series. KrakenD is one of 18 acquisitions in a group whose software serves 200,000+ businesses, and every product in it that puts a model in front of a customer will need the same things: a provider abstraction, failover, caching, prompt filtering, budget enforcement and a safe way to expose its API to agents. Those now exist, built once, inside KrakenD. Any product that chooses to sit behind the gateway gets them by configuration instead of writing them again.

The Circeus thesis

This is what acquire and evolve means in practice. A ten-year-old gateway built for speed is thirteen months into the group and now built for AI too: it governs every API, every model and every agent that passes through it. That is what a central AI capability at the infrastructure layer looks like: the systems of record keep their data, and the layer in front of them is where they become systems of action.