Scaling agent workflows with stateless tool servers and header-based routing
Agent systems become difficult to scale when every tool request depends on the same backend instance that handled an earlier request. Stateless tool servers address that constraint by allowing MCP traffic to reach any healthy instance, while er-based routing gives gateways a protocol-aware way to dispatch requests without inspecting tool payloads.
This design is especially relevant for production agent workflows, where models may chain tool calls, execute work in parallel, run code, and coordinate across remote services. The goal is not to eliminate state everywhere. It is to keep protocol transport state out of the tool-serving tier when a tool does not need it, then place durable application state, authorization, and orchestration where those concerns can be operated deliberately.
How stateless tool servers scale agent workflows
A stateless tool server does not require session affinity between a client and a particular server instance. In practical terms, a request can arrive at one instance and the next request can arrive at another, provided each instance has the information required to process the individual request.
That matters because agent workflows are increasingly operational systems rather than isolated demos. OpenAI’s recent guidance describes agents that connect to tools, work across files, execute code in sandboxed environments, and orchestrate multi-step work. Its programmatic tool-calling guidance also emphasizes chaining related calls, parallelizing calls, and filtering or combining results in code. A tool tier that can distribute requests across instances is a better architectural match for this kind of variable, concurrent workload.
Direct answer: Scale agent workflows with stateless MCP tool servers by making each request independently serviceable, placing instances behind a standard load balancer, and routing by the
Mcp-Methoder rather than maintaining protocol-level sticky sessions. Keep durable workflow, user, and approval state in the application systems that own it.
The MCP protocol’s 2026-07-28 release candidate moved remote server deployments toward this model. The protocol blog states that servers which previously needed sticky sessions, a shared session store, and deep packet inspection can run behind a plain round-robin load balancer. It also introduces routing on an Mcp-Method er and permits caching of tools/list responses for as long as ttlMs allows.
The important distinction is between stateless transport and an entirely stateless business system. A tool that retrieves a record from a database, submits a request to an internal API, or evaluates a policy can be transport-stateless even when the broader application has durable data. The server simply does not rely on a private, in-memory protocol session being pinned to a single replica.
Transport state:
connection or session-specific protocol information held by a tool server.
Application state:
durable business data such as accounts, workflows, approvals, records, and audit history.
Agent state:
conversation context, run progress, and orchestration information managed by an application, SDK, or hosted harness.
Separating these concerns gives teams clearer scaling boundaries. The tool-serving tier can scale with request demand, while the systems that own durable state can use the storage, consistency, and governance approach appropriate to their data.
Why MCP er-based routing changes the infrastructure design
Header-based routing makes protocol intent visible to routing infrastructure without requiring the infrastructure to inspect the request. In the release candidate, the relevant request metadata is carried in the Mcp-Method er. The protocol blog specifically identifies this as a way to route traffic while removing the old need for deep packet inspection.
This is a meaningful simplification. A gateway can make a routing decision from a standard request er, while tool implementation details remain inside the service. As an inference from the MCP release description, that reduces coupling between the gateway configuration and the internal payload structure used by a particular tool or server implementation.
What a gateway can do with method metadata
The exact policies remain a deployment decision, but method metadata gives teams a clean routing surface for protocol-aware handling. For example, infrastructure can distinguish discovery traffic from tool invocation traffic without parsing a.
Send MCP requests through a standard load-balancing path when all instances can handle them.
Apply a caching policy to
tools/list
only for the duration permitted by
ttlMs
.
Keep method-aware gateway rules separate from the implementation details of individual tools.
Observe traffic categories at the edge without making the gateway responsible for interpreting tool arguments.
Header routing should not be confused with a complete authorization design. The presence of a method er does not determine whether a caller may use a tool, access a tenant’s data, or approve a consequential action. Those checks still belong in the identity, policy, and application layers that own them.
It also does not mean that every method needs separate infrastructure treatment. For many teams, the immediate benefit is simpler: use the er to support the protocol’s stateless routing model, then add more specific policies only when there is a clear operational reason.
Choose stateless MCP transport when tool interactions are request-driven
Stateless transport is a strong fit for simple tool servers whose work is initiated by the client and completed through request-response interactions. The Ruby MCP SDK documents this directly: it recommends stateless: true for simple tool servers that do not need server-initiated requests.
The C# SDK likewise documents stateless servers as a supported mode. Its documentation notes that earlier versions defaulted to stateful behavior, while stateless servers can still respond to older clients. The trade-off is explicit: session-dependent features are dropped, including unsolicited notifications, resource subscriptions, and per-client isolation.
Good candidates for stateless tool serving
Look for tools where the request contains, or can safely resolve, the context needed for execution. The tool may access durable systems, but its server does not need to retain a private protocol session for the caller.
Read-oriented tools:
tools that fetch approved information from a source of record.
Action tools with explicit inputs:
tools that submit a clearly defined operation and rely on durable systems to record its outcome.
Transformation tools:
tools that validate, classify, format, or process supplied content as part of a single request.
Discovery endpoints:
tool metadata operations such as
tools/list
, subject to the protocol’s
ttlMs
caching allowance.
When stateful behavior may still be necessary
Do not force a stateful interaction into a stateless server merely to obtain simpler routing. A server that relies on unsolicited notifications, resource subscriptions, or strict per-client isolation is using capabilities that the documented stateless mode does not preserve.
There is an important nuance for interactive workflows: the MCP stateless model still supports client-requested elicitation prompts. According to the protocol blog, this preserves mid-call interaction without requiring sticky server state. That makes stateless transport more capable than a simplistic “one request, no interaction” model, but it does not restore every server-initiated or session-specific feature.
The design question is therefore not “Does the overall workflow have state?” Most useful workflows do. The question is “Must this MCP server maintain protocol session state to deliver its capability?” If the answer is no, a stateless deployment removes an infrastructure dependency that otherwise follows every replica and failover decision.
Separate agent orchestration from the stateless tool tier
Stateless tool servers work best when responsibilities are explicit. OpenAI’s Agents SDK guidance draws a useful line: use the SDK track when your server owns deployment, tool implementations, state storage, and approval decisions, while the SDK handles the agent loop and invokes the tools.
OpenAI’s Agents API architecture describes a complementary arrangement in which a hosted harness runs the model and tool loop, while an application can connect to remote MCP servers. The platform also states that agents can connect to MCP servers directly from the hosted sandboxed session, alongside capabilities such as code execution, file editing, and artifact production.
These models do not require identical deployments, but they point to the same architectural discipline: the component running the agent loop does not need to be the component that implements every business tool, and a remote tool server does not need to become the owner of the agent’s long-lived workflow state.
Assign ownership before choosing a scaling pattern
Tool server:
validates a tool request, enforces the tool’s applicable policy, calls owned dependencies, and returns a result.
Application server:
owns application-specific state, business workflows, approvals, and integrations that belong to the product.
Agent harness or SDK:
manages the model and tool loop according to the selected deployment model.
Durable state systems:
retain records that must outlive an individual request or replica.
OpenAI’s guidance on running agents further clarifies why this separation matters. It describes session as your storage plus the SDK, with persistent chat state and resumable runs under your control. It describes conversationId as shared server-managed state across workers or services. Both patterns show that agent continuity can be managed outside a specific MCP tool replica.
For a production deployment, state placement is also a reliability decision. If retry behavior, approval status, or user-visible workflow progress matters, store and manage it in the component designated to own it. A stateless tool instance can then be replaced or scaled without becoming the sole holder of critical run context.
Design a routing path for horizontally scaled tool calls
A practical rollout starts with the request path, not with autoscaling settings. Map how an agent or application reaches the MCP endpoint, how the gateway reads protocol metadata, how requests are balanced across replicas, and where the tool obtains any durable context it needs.
With the old session-dependent approach removed, the MCP protocol blog says any MCP request can land on any instance. This is the core scaling property: the routing layer no longer needs to preserve affinity simply because of the protocol. As an inference from the protocol and SDK documentation, deployments can treat suitable tool servers much like ordinary stateless web services behind load balancers, simplifying both scaling and failure recovery.
A disciplined implementation sequence
Inventory tool interaction requirements.
Identify which tools need only client-triggered requests and which depend on subscriptions, unsolicited notifications, or per-client isolation.
Move durable state to its owner.
Ensure workflow records, authorization data, approval decisions, and other long-lived information are not held only in a tool replica’s protocol session.
Enable stateless transport where supported.
For example, the Ruby SDK documents
stateless: true
for simple tool servers; the C# SDK documents stateless server support and its feature boundaries.
Put replicas behind a standard load balancer.
Use a routing approach compatible with any request reaching any healthy instance, rather than retaining protocol-driven stickiness.
Route using
Mcp-Method.Configure the gateway to use the protocol er as request metadata instead of relying on deep packet inspection.
Apply discovery caching carefully.
Cache
tools/list
only as long as the server’s
ttlMs
permits, and ensure tool metadata changes are reflected according to that contract.
Validate failure behavior.
Confirm that an in-flight or subsequent request can be handled by another instance without depending on private transport state from the prior replica.
This sequence avoids a common mistake: treating a load balancer as the entire solution. Load balancing distributes requests, but stateless behavior comes from server design and from moving essential continuity into the right durable systems.
Keep gateway rules narrow and understandable
It is tempting to place more and more business logic at the edge once a routing er is available. That often recreates coupling in another form. A gateway should route, apply appropriate edge controls, and expose useful observability; the tool service should remain responsible for tool-specific interpretation and execution.
This division is particularly useful for teams operating hosted MCP endpoints or converting websites, documentation, and files into MCP-accessible tools. The serving layer can scale independently, while the content ingestion, indexing, retrieval, and product-specific access controls remain governed by the systems responsible for them.
Use caching and observability without reintroducing hidden state
Stateless does not mean blind. Production teams still need to understand which tools are being invoked, whether requests succeed, where failures occur, and how tool discovery changes over time. The key is to make observability a shared operational capability rather than a reason to keep opaque, replica-local session data.
The MCP release candidate provides one concrete caching mechanism: tools/list responses may be cached for as long as ttlMs allows. That is a protocol-defined boundary, not a license to cache all tool results indefinitely. Tool invocations may depend on changing data, authorization, or side effects, so caching decisions beyond discovery must follow the semantics of the individual tool.
Operational questions worth answering before launch
Can an operator identify the requested MCP method from request metadata?
Can the team distinguish discovery traffic from tool invocation traffic?
Does a tool execution record enough information in its owned systems to investigate an outcome without relying on one replica’s memory?
Are authorization and approval decisions evaluated in the components that own those policies?
Does tool discovery respect the advertised
ttlMs
period when caching is enabled?
Can a replacement instance serve a subsequent request without reconstructing an old protocol session?
These are design-review questions rather than a prescribed monitoring product or metric list. The appropriate telemetry depends on the organization’s tools and compliance requirements. What matters architecturally is that observability follows requests and durable decisions across the system, instead of depending on affinity to one process.
For enterprise deployments, this separation also improves clarity around compliance boundaries. A hosted MCP endpoint can provide the scalable tool interface, but policy enforcement, source permissions, audit requirements, and retention rules must still be designed around the organization’s actual data and governance model. Stateless routing simplifies one layer; it does not eliminate responsibility in the others.
Understand the trade-offs of removing sticky sessions
Removing protocol-level sticky routing has clear benefits, but it is not a universal upgrade for every server. The benefit is strongest where horizontal elasticity, simple failover, and independent replicas matter more than server-initiated, session-specific features.
The MCP materials make the trade-off concrete. Stateless mode preserves client-triggered interactions and supports client-requested elicitation prompts, yet the C# SDK documentation says it drops session-dependent capabilities such as unsolicited notifications, resource subscriptions, and per-client isolation. A team should make that choice intentionally at the tool level.
What becomes simpler
Requests can be distributed across healthy instances without protocol-level affinity.
A plain round-robin load balancer can support the routing model described in the MCP release candidate.
Infrastructure does not need deep packet inspection solely to make MCP routing decisions.
Failure recovery is easier to reason about when no single replica owns an essential protocol session.
Capacity can align more naturally with bursty, parallel, and chained tool usage.
What still needs deliberate design
Long-lived application and conversation state needs a durable owner.
Tools with server-initiated or subscription-based behavior may require a stateful design.
Authorization, tenant boundaries, approvals, and audit obligations remain application and organizational concerns.
Tool semantics determine whether any result caching is safe; the
tools/list
allowance does not generalize automatically.
Backward compatibility should be tested with the clients your deployment supports, even though the C# SDK documents that stateless servers still respond to older clients.
An alternative is to keep a stateful MCP server for the subset of capabilities that genuinely requires it, while operating simple, request-driven tools as stateless servers. This can be more practical than making every tool conform to one interaction model. The critical point is to avoid accidental statefulness: a service should not require sticky routing merely because its deployment was never separated from its workflow state.
Build for agent concurrency, not just single-tool demos
Single-tool demos can conceal scaling constraints. A workflow may look healthy when one assistant invokes one tool, then behave very differently when agents chain operations, run independent calls in parallel, or coordinate code execution with remote services. OpenAI’s programmatic tool-calling guidance specifically positions these patterns,chaining, parallelization, and filtering or combining results in code,as supported orchestration techniques.
That is why stateless tool servers and er-based routing are a natural pairing for production agent infrastructure. They do not make every downstream dependency horizontally scalable, and they do not solve poor tool design. They do remove an avoidable protocol-level barrier between concurrent requests and a pool of service instances.
For product teams, this enables a cleaner evolution path. Start with a hosted or self-managed MCP endpoint that exposes useful, bounded tools. Keep the server simple where possible, deploy replicas behind standard infrastructure, and allow the application or agent layer to own conversation continuity and workflow decisions. As usage grows, the tool tier can be expanded without tying every client’s requests to one process.
For agencies and enterprise teams, the same pattern supports clearer delivery boundaries. Content-backed MCP servers can expose approved website, documentation, or file knowledge to assistants, while the deployment model remains suitable for operational controls, observability, and separate ownership of data policy. The scalable interface is valuable precisely because it does not require the routing layer to understand every content source or tool argument.
The practical takeaway is straightforward: use stateless MCP transport for request-driven tools, route with Mcp-Method, and place durable state in the application, storage, or agent layer that owns it. This aligns with MCP’s move away from sticky sessions and with modern agent architectures that treat tool orchestration as production infrastructure.
Before adopting the pattern, classify each tool’s interaction needs, verify the stateless SDK behavior you rely on, respect ttlMs for tools/list caching, and test requests across multiple instances. Done deliberately, stateless tool servers make scaling and recovery simpler without pretending that business state, policy, or workflow ownership have disappeared.