agentic-api, also referred to as vLLM Agentic API, is the stateful agentic
API layer for vLLM core. It is optimized
for vLLM core and implemented in Rust. Its job is to own stateful agentic APIs,
including Responses and Messages, and to orchestrate server-side tool execution
for tool calls generated by vLLM core.
This roadmap uses:
- vLLM core for
vllm-project/vllm, the inference engine and model-serving project. - vLLM Agentic API for this repository,
vllm-project/agentic-api.
vLLM Agentic API should let vLLM core focus on inference while this repository handles API state, continuation, tool-call execution, and the higher-level orchestration needed by stateful agentic applications.
Project Goals
1. Responses API Hydration
The first project goal is reliable state hydration for the Responses API.
This means supporting previous_response_id so clients can continue from a
stored response without replaying the full conversation history. The server must
persist the model-visible items needed for continuation, rehydrate them in the
right order, append new input, and send the resulting request to vLLM core.
Core work:
- Store Responses API state needed for continuation.
- Rehydrate prior input, assistant output, tool calls, and tool outputs.
- Preserve OpenAI-compatible request and response shapes.
- Support streaming and non-streaming Responses API flows.
- Keep storage semantics explicit and testable.
2. Codex Support Through Responses
The second project goal is to support Codex through the Responses API.
Codex depends on stateful continuation, tool-call preservation, and a broad set
of agentic tool and response shapes. agentic-api should accept Codex-compatible
Responses traffic, preserve the fields Codex needs, and execute or return tool
calls according to the ownership model for each tool.
Core work:
- Preserve Codex-used tool declarations and response item shapes.
- Support client-owned tools, gateway-owned tools, and provider-owned tools.
- Implement the tool calls and API behavior needed for practical Codex sessions.
3. Server-Side Tool Execution
The project should execute server-side tool calls generated by vLLM core.
When vLLM core produces a tool call that is owned by the gateway, vLLM Agentic API should dispatch it, capture the result, append the tool output, and continue the agentic loop. The tool system should make ownership explicit so each tool call has a clear execution path.
Tool ownership means:
- Gateway-owned tools are executed by vLLM Agentic API.
- Client-owned tools are preserved and returned to the client for execution.
- Provider-owned tools are passed through to vLLM core or another upstream provider surface; vLLM Agentic API does not execute them.
Ownership should be resolved from the request, the configured tool registry, and the tool type. vLLM Agentic API should only execute tools that resolve to a configured gateway-owned handler. Unknown, unsupported, or ambiguous tool shapes are preserved and returned or passed through; they are never executed by default.
Requests may opt into parallel tool calling for gateway-owned built-in tools: a single turn can invoke the same built-in tool more than once (for example, two web searches). Agentic API forwards that model-generation preference upstream, then executes emitted gateway calls through a bounded, configurable window. Calls to different tool names can overlap; calls to the same name overlap only when that handler declares it safe. Results retain model call order and are all appended before continuing the agentic loop (#181).
Initial and expected tool areas include:
- File search.
- Web search.
- Web fetch.
- Computer use.
- Shell tools.
- MCP-backed tools and resources.
- Function tools owned by the gateway.
4. Messages API
The third major API goal is Messages.
Messages should build on the same persistence, rehydration, and execution foundations as Responses, while exposing the API surface expected by agentic clients that organize state around message objects.
Messages should adapt into shared execution primitives for items, tool calls, and tool outputs instead of forking a Responses-specific execution loop.
Core work:
- Define the Messages API state model.
- Map Messages state into the shared execution and storage primitives.
- Preserve compatibility with Responses where the APIs overlap.
- Add tests that cover continuation and tool-call behavior through Messages.
5. Broader Stateful Agentic APIs
After Responses, Codex support, and Messages, the project should add the remaining stateful APIs needed by core agentic applications.
This should be driven by concrete client needs and vLLM core integration points, not by adding broad abstractions ahead of use. Each new API should reuse the common storage, continuation, and tool-execution foundations wherever possible.
Near-Term Focus
The near-term focus is to make the Rust implementation dependable and easy to extend.
- Keep orchestration logic reusable across standalone and gateway deployments.
- Keep HTTP serving, gateway integration, and tool execution thin around the shared stateful agentic API implementation.
- Keep the implementation Rust-first and optimized around vLLM core's inference surface.
- Treat model aliasing and routing as deployment concerns that may live at the standalone server or gateway edge; they are not part of tool execution semantics.
- Expand compatibility tests for Responses, Codex-shaped traffic, tool calls, streaming, and continuation.
- Document behavior as it stabilizes, especially where compatibility or tool ownership rules are subtle.
Longer-Term Direction
Longer-term work should improve production readiness and performance without moving inference responsibilities out of vLLM core.
- Production storage backends, retention policy, and compaction.
- Observability for request lifecycle, tool execution, and continuation.
- Gateway integration with Praxis and other deployment surfaces.
- Cached prefix continuation and other latency optimizations where vLLM core owns rendering, tokenization, and KV-cache execution.
- Coordination with vLLM core and llm-d where token identity, prefix routing, and renderer boundaries matter.
Non-Goals For Now
These are intentionally out of scope until the core APIs and execution model are stable.
- Reimplementing tokenization, chat templates, or model-specific rendering in this repository.
- Building a broad plugin platform before the built-in execution model is settled.
- Adding stateful agentic APIs without a concrete client or vLLM core integration need.
- Moving vLLM core inference responsibilities into vLLM Agentic API.