Agent Control Planes and Observability: The Infrastructure Layer Nobody's Built Yet

The Missing Layer Between Agent Demos and Agent Production

Every enterprise I work with has built agent demos. Most have a handful of agents in production — customer service bots, internal knowledge assistants, pipeline automation workflows. A few are running multi-agent orchestration at meaningful scale. But almost none of them have built the infrastructure layer that sits between the agents and the enterprise: the control plane.

This is the layer that answers the questions every CTO, CISO, and enterprise architect starts asking once agents move beyond pilot: Who authorized this agent? What can it access? How much is it spending? Why did it make that decision? Who did it delegate to? What happened when it failed? These are not agent capability questions. They are infrastructure governance questions — and the tooling to answer them at enterprise scale does not exist yet in most organizations.

The agent ecosystem has been so focused on building better agents — smarter reasoning, better tool use, more capable models — that the infrastructure layer has been left as an exercise for the reader. We are at the same point the microservices ecosystem was in 2015: services were proliferating, but service meshes, API gateways, and observability platforms were still nascent. The agent infrastructure gap is real, it is growing, and it is the single biggest blocker to enterprise-scale agent adoption.

In this post, I want to lay out what the agent control plane pattern looks like, why existing infrastructure (API gateways, HTTP proxies, APM tools) is insufficient, and what architects should be building now to close the gap.

The Agent Control Plane Pattern

A control plane, in infrastructure terms, is the layer that manages, configures, and monitors the data plane — where actual traffic flows. In Kubernetes, the control plane (API server, scheduler, controller manager) manages the data plane (pods, services, network). In a service mesh like Istio, the control plane (Istiod) configures the data plane (Envoy sidecars) that handle service-to-service traffic.

The agent control plane applies the same separation to agent systems. The data plane is where agents execute — calling tools, querying models, exchanging messages via MCP and A2A, producing outputs. The control plane is where the enterprise governs that execution:

• Agent Registry: a centralized catalog of every agent, its capabilities, its permissions, its owner, and its operational status. When a new agent is deployed, it registers. When an orchestrator needs to discover agents, it queries the registry. When an auditor asks "what agents do we have in production?", the registry answers.
• MCP Server Catalog: a registry of every MCP server — the tools, resources, and prompts each exposes — along with access policies, version information, and health status. This is the agent equivalent of an API catalog. Without it, agents discover tools ad hoc, and the enterprise has no visibility into what integrations exist.
• Model Registry: a catalog of available models with their capabilities, cost profiles, rate limits, and routing policies. The control plane uses this registry to enforce model selection policies — ensuring that expensive frontier models are reserved for tasks that justify their cost, and that sensitive data is routed only to models that meet compliance requirements.
• Policy Engine: runtime policy enforcement for agent actions. What data can this agent access? What tools can it call? What spend limits apply? What approval workflows must it trigger before taking high-impact actions? Policies are defined declaratively and enforced at the control plane, not embedded in agent code.
• Spend Management: token metering, cost attribution, and budget enforcement at the agent, team, and organization level. Every model call, every tool invocation, every resource access is metered and attributed to the responsible agent and its owning team. Budget limits trigger alerts or hard stops before runaway costs accumulate.

The control plane pattern works because it separates what agents can do (capabilities, defined by agent developers) from what agents are allowed to do (policies, defined by the enterprise). Agent developers focus on building effective agents. The control plane ensures those agents operate within enterprise-defined boundaries. This separation is not optional at scale — it is the only way to maintain governance without bottlenecking every agent development team through a central approval process.

Why API Gateways and HTTP Proxies Are Not Enough

The first reaction most infrastructure teams have to the agent governance problem is "we already have API gateways — can we just put agents behind them?" The answer is no, and the reasons are instructive.

Traditional API gateways operate at the HTTP request level. They see URLs, headers, query parameters, and request bodies. They can enforce rate limits, authenticate callers, route traffic, and log requests. This works well for REST APIs because the semantics of a REST call are visible at the HTTP level — the URL tells you what resource is being accessed, the method tells you what operation is being performed.

MCP traffic is different. An MCP interaction typically flows over a single HTTP endpoint (the Streamable HTTP transport). The actual semantics — tools/list, tools/call, resources/read, completion/complete — are encoded in the JSON-RPC body, not in the URL or HTTP method. An HTTP proxy sees a stream of POST requests to the same endpoint and has no visibility into what tools are being called, what resources are being read, or what arguments are being passed.

This means that API gateways cannot enforce MCP-level policies without deep packet inspection of the JSON-RPC body — something they were not designed for. You cannot rate-limit tools/call independently from resources/read. You cannot enforce that Agent X is allowed to call database_query but not database_write. You cannot log which specific tools are being invoked or measure latency per tool call. The gateway sees HTTP traffic; it does not see agent behavior.

The same limitation applies to traditional APM (Application Performance Monitoring) tools. They instrument HTTP calls, database queries, and service-to-service communication. But they do not understand MCP method semantics, A2A agent delegation, or the causal chain that connects a user request to a sequence of agent actions across multiple services. When Agent A delegates to Agent B, which calls an MCP tool on Server C, which queries a database — traditional APM sees three unrelated HTTP calls. It does not see the delegation chain that connects them.

Identity-Aware Tracing for Multi-Agent Delegation Chains

One of the hardest observability problems in multi-agent systems is tracing delegation chains. When a user makes a request that triggers Agent A, which delegates to Agent B, which calls Agent C, which invokes an MCP tool — the enterprise needs to trace this entire chain to understand what happened, why, and on whose behalf.

Traditional distributed tracing (OpenTelemetry, Jaeger, Zipkin) handles the mechanics of trace propagation well. But agent delegation introduces a dimension that service-to-service calls do not have: identity delegation. When Agent B acts on behalf of Agent A, which acts on behalf of User X, every action in the chain needs to carry the full delegation context — who initiated the request, who delegated to whom, and what permissions apply at each hop.

This is where RFC 8693 Token Exchange with act (actor) claims becomes essential. RFC 8693 defines a mechanism for exchanging one security token for another, with the act claim explicitly recording the delegation chain. When Agent A delegates to Agent B, Agent A exchanges its token for a new token that says "Agent B acting on behalf of Agent A acting on behalf of User X." Each link in the delegation chain is recorded in the token, and every downstream service can inspect the full chain to make authorization decisions.

The act claim structure is recursive, which is exactly what multi-agent delegation requires:

• User X authenticates and receives a token with their identity.
• Agent A receives a delegated token: { "sub": "agent-a", "act": { "sub": "user-x" } }
• Agent B receives a further-delegated token: { "sub": "agent-b", "act": { "sub": "agent-a", "act": { "sub": "user-x" } } }
• Any service in the chain can unwind the act claims to see the full delegation path and apply policies accordingly.

Combining RFC 8693 token exchange with OpenTelemetry trace context creates a powerful observability model: every span in a trace carries both the technical context (trace ID, span ID, parent span) and the identity context (who is acting, on whose behalf, through what delegation chain). This enables queries that traditional tracing cannot answer: "Show me every action taken on behalf of User X across all agents" or "Show me every tool call made by Agent B when delegated by Agent A."

Building this requires instrumentation at the agent level, not just the infrastructure level. Each agent must propagate both trace context and delegation tokens when making downstream calls. MCP clients and A2A clients must be instrumented to include this context in every request. This is new plumbing that most agent frameworks do not yet provide — but it is essential plumbing for production multi-agent systems.

Agent Gateways: A New Infrastructure Primitive

If API gateways cannot see agent-level semantics, the answer is not to force agents behind API gateways — it is to build a new infrastructure primitive: the agent gateway.

An agent gateway sits at the boundary of the agent system and understands agent protocols natively. It speaks MCP, A2A, and the model inference APIs. It can inspect, route, and policy-enforce at the semantic level — understanding which tools are being called, which agents are delegating to which other agents, what model is being invoked, and what data is flowing through the system.

The capabilities of an agent gateway go beyond what API gateways provide:

• Token metering: the gateway counts tokens per model call, per agent, per user, and per organization. It enforces token budgets and generates cost attribution data. This is not HTTP byte counting — it is LLM-native metering that understands the difference between input tokens, output tokens, and cached tokens, each with different cost profiles.
• Prompt sanitization: the gateway inspects prompts flowing to models and applies sanitization policies — stripping PII, blocking prompt injection patterns, enforcing content policies. This is defense-in-depth: even if an agent's code does not sanitize its prompts (and most do not), the gateway catches policy violations before they reach the model.
• Agent identity and authentication: the gateway authenticates agents using their identity tokens, validates delegation chains, and enforces that only authorized agents can access specific MCP servers, models, or downstream agents. This is capability-based authorization at the agent level — not just "is this caller authenticated?" but "is this agent authorized to call this specific tool with these specific arguments in this specific delegation context?"
• Capability-based authorization: the gateway enforces fine-grained access control based on agent capabilities. Agent X can call tools/call for read_customer but not delete_customer. Agent Y can access Model A but not Model B. These policies are defined in the control plane and enforced at the gateway, not embedded in agent code.
• MCP method-level observability: the gateway logs every MCP method call — tools/list, tools/call, resources/read — with full arguments, responses, latencies, and error codes. This provides the visibility that HTTP-level logging cannot: a complete record of what every agent did, at the tool-call level of granularity.
• Traffic shaping and rate limiting: the gateway enforces rate limits at the MCP method level, the model level, and the agent level. It can throttle a runaway agent that is making too many tool calls, route traffic away from an overloaded MCP server, or shed load during capacity crunches — all with agent-level awareness that HTTP rate limiting cannot provide.

Agent gateways are still an emerging category. A few vendors are building them — most notably Axway with their AI Control Plane — but the category is nascent. Most enterprises that need agent gateway capabilities today are building them in-house, typically as custom proxy layers that understand their specific agent protocols and policies. This is the frontier of agent infrastructure, and it represents a significant opportunity for platform vendors and open-source projects.

AAIF Working Groups and Emerging Standards

The good news is that the industry recognizes the agent infrastructure gap, and standards work is underway. The Agentic AI Foundation (AAIF), which brought MCP and A2A under a shared governance umbrella in August 2026, has established working groups that directly address the control plane and observability challenges.

Three AAIF working groups are particularly relevant to agent infrastructure:

Identity & Trust

This working group is defining standards for agent identity, authentication, and delegation. Key questions include: How do agents prove their identity to other agents and services? How are delegation chains represented and validated? How do policies reference agent identities? The work here directly enables the RFC 8693-based delegation tracing pattern described earlier — but aims to standardize it so that agents from different vendors can participate in shared delegation chains with interoperable identity tokens.

Observability & Traceability

This working group is defining standards for agent-level observability — extending OpenTelemetry with agent-aware semantic conventions. The goal is to enable cross-vendor, cross-framework observability for multi-agent systems: a trace that starts in one vendor's agent framework, passes through an MCP server, delegates to another vendor's agent, and calls a third-party model should be traceable end-to-end with consistent semantics. This requires defining standard span attributes for agent actions, tool calls, model invocations, and delegation events.

Security & Privacy

This working group addresses the security primitives for agent systems — capability-based authorization, prompt sanitization standards, data flow policies, and audit requirements. The intersection with the control plane pattern is direct: the security working group is defining what policies mean, and the control plane enforces those policies at runtime.

These standards are in early development, and it will likely be 12-18 months before they are mature enough for production adoption. But architects should be tracking them now, because the patterns they standardize will become the expected infrastructure for enterprise agent deployments. Building agent infrastructure today without awareness of these emerging standards risks significant rework when the standards arrive.

The Vendor Landscape: Who Is Building What

The agent control plane and gateway space is early, but several vendors are staking out positions:

• Axway AI Control Plane: Axway has been the most explicit about positioning their platform as an agent control plane — combining API management heritage with agent-native capabilities including MCP gateway functionality, agent registry, and policy enforcement. Their approach extends traditional API management patterns into the agent domain, which gives them a head start on the gateway capabilities but requires adaptation for agent-specific patterns like delegation chains and multi-agent orchestration.
• Salesforce Agent Optimizer: within the Salesforce ecosystem, Agent Optimizer provides lifecycle management and observability for Agentforce agents. It is not a general-purpose agent control plane — it is scoped to the Salesforce platform — but it demonstrates the pattern: centralized management, performance monitoring, version control, and optimization recommendations for production agents.
• Cloud provider offerings: AWS, Google Cloud, and Azure are all building agent infrastructure capabilities into their platforms. AWS Bedrock has agent management features. Google's Vertex AI has agent observability. Azure AI Foundry has agent deployment and monitoring. These are evolving rapidly but remain cloud-specific rather than multi-cloud or protocol-native.
• Open-source and startup ecosystem: several startups are building agent observability platforms (extending OpenTelemetry for agent semantics), agent gateway proxies (MCP-aware routing and policy enforcement), and agent registry services. The space is fragmented and early, but the number of entrants signals strong demand for these capabilities.

The vendor landscape reinforces a key architectural principle: do not wait for a single vendor to build your entire agent control plane. The space is too early and too fragmented for any single vendor to provide a complete solution. Instead, design your control plane as a composition of capabilities — registry, gateway, observability, policy engine — that can be sourced from different vendors or built in-house as the market matures.

What Architects Should Build Now

Given the early state of agent infrastructure tooling, what should enterprise architects be building today? Here is my recommended priority stack:

1. Agent Registry

Start with a simple registry of every agent in your organization: name, owner, capabilities, permissions, deployment status, and dependencies. This can start as a spreadsheet or a YAML file in a Git repository — the format matters less than the discipline of maintaining it. Every new agent deployment should include registration. Every agent retirement should include deregistration. The registry becomes the foundation for everything else: you cannot govern what you cannot see.

2. MCP Server Catalog

Catalog every MCP server your organization operates or connects to: what tools it exposes, what data it accesses, what authentication it requires, and what policies apply. As MCP adoption grows, MCP servers will proliferate the way APIs did — and without a catalog, you will face the same discovery and governance challenges that drove the adoption of API catalogs a decade ago. The Salesforce Catalog, announced at Dreamforce 2026, demonstrates this pattern: an MCP-enabled index of how your org is built, so agents can discover capabilities before acting.

3. Delegation-Chain Tracing

Implement identity-aware tracing for multi-agent interactions. Instrument your agents to propagate both OpenTelemetry trace context and RFC 8693 delegation tokens. Log every delegation event with the full chain. This is the observability capability that will matter most when debugging production multi-agent issues — and it is the hardest to retrofit after the fact. Build it in from the beginning.

4. Policy-as-Code for Agent Permissions

Define agent access policies in code, not in configuration UIs or tribal knowledge. Use a policy-as-code framework (OPA/Rego, Cedar, or even simple YAML policy files) to declare what each agent can access, what tools it can call, what models it can use, and what spend limits apply. Store policies in version control. Review policy changes like code changes. Enforce policies at runtime through your gateway or middleware layer. Policy-as-code ensures that agent governance is auditable, reproducible, and scalable — three things that manual governance processes are not.

5. Cost Attribution and Metering

Implement per-agent, per-model cost tracking from day one. Every model call should be attributed to the calling agent, the owning team, and the originating user or workflow. Aggregate costs by agent, by team, by use case, and by model. Set budget alerts. Publish cost dashboards. Cost visibility is not just a finance concern — it is an architectural concern that drives model selection, caching strategy, and routing decisions. The organizations that build cost attribution early make better architectural decisions than those that discover their costs after the fact.

The Service Mesh Analogy: Where Agent Infrastructure Is Heading

The trajectory of agent infrastructure mirrors the evolution of service infrastructure over the past decade. In the early days of microservices (2012-2015), services proliferated without centralized management. Service discovery was ad hoc. Load balancing was configured manually. Security between services was inconsistent. Observability was per-service, not cross-service.

Then the infrastructure layer matured. Kubernetes provided a control plane for container orchestration — scheduling, scaling, health checking, service discovery. Istio and other service meshes provided a data plane for service-to-service communication — mutual TLS, traffic management, observability, policy enforcement. API gateways provided ingress management — authentication, rate limiting, routing. OpenTelemetry provided cross-service observability standards. Together, these layers transformed microservices from an architectural pattern into a manageable operational reality.

Agent infrastructure is at the same inflection point. Agents are proliferating without centralized management. Discovery is ad hoc. Security between agents is inconsistent. Observability is per-agent, not cross-agent. The patterns that solved these problems for microservices — registries, gateways, control planes, observability standards — are the patterns that will solve them for agents. The specific implementations will differ (MCP-aware gateways instead of HTTP-aware gateways, delegation-chain tracing instead of service-call tracing, capability-based authorization instead of URL-based authorization), but the architectural principles are the same.

We are in the 2015-equivalent moment for agent infrastructure. The organizations that invest in the agent control plane, agent gateway, and agent observability layers now — even in their simplest forms — will be the organizations that can scale agent adoption to production without governance crises. The organizations that wait for the tooling to mature before investing will find themselves with hundreds of ungoverned agents and no way to manage them.

The Infrastructure Imperative

The agent ecosystem has been focused on making agents smarter. That is necessary but insufficient. Smart agents without infrastructure governance are like powerful microservices without Kubernetes — they work in demos, they work in small deployments, and they collapse under the weight of their own success at enterprise scale.

The next phase of enterprise agent adoption is not about better models or better prompts. It is about the infrastructure layer that makes agent systems manageable, observable, governable, and economically sustainable at scale. It is about control planes that know what agents exist and what they are allowed to do. It is about gateways that understand agent protocols and enforce policies at the semantic level. It is about observability that traces delegation chains across agents, services, and models. It is about identity frameworks that enable secure delegation without losing accountability.

This infrastructure layer does not exist in complete form today. Pieces are emerging — from AAIF standards work, from vendors like Axway, from cloud provider agent platforms, from open-source projects. But the complete picture will take 2-3 years to fully materialize. Enterprise architects who understand the patterns now and build incrementally toward them will be ready when the tooling arrives. Those who wait will be playing catch-up.

Build the registry. Instrument the traces. Define the policies. Meter the costs. The agents are here. The infrastructure to govern them is the next frontier.