Model Provider Architecture: How Schema Weaver's Data Explorer Routes, Streams, and Fails Over Across Four LLM Providers
Most AI products ship with one provider hardcoded. When that provider has an outage, rate limits you, or deprecates a model, your product breaks. Schema Weaver's Data Explorer runs on a provider-agnostic abstraction layer that supports AWS Bedrock, OpenAI, Anthropic, and Google GenAI behind a single interface — with per-provider circuit breakers, automatic failover, streaming normalization across four different SSE formats, and unified token/cost tracking. This is the complete model provider architecture — the BaseProvider contract, the router, the failover chain, and every edge case we hit in production. Part 4 of the Data Explorer Architecture Deep Dive.

In Part 1, we showed the four-layer architecture of Schema Weaver's Data Explorer and mentioned that the Model Provider layer abstracts four LLM vendors behind a single interface. This is the deep dive on that layer. Most AI products hardcode one provider — when that provider has an outage, rate-limits you, or deprecates a model, your product breaks. We refused that fragility. Schema Weaver abstracts four LLM providers — AWS Bedrock, OpenAI, Anthropic, and Google GenAI — behind a unified streaming interface with per-provider circuit breakers, automatic fallback chains, native tool-calling support, and a 92% token cost reduction strategy. This is the complete model provider architecture.
This post covers eight topics: the BaseProvider abstraction, the two routing systems (ModelRouter for background tasks and ReactProviderContext for the streaming agent loop), the unified streaming interface, the circuit breaker pattern, the ProviderPool fallback chain, token usage tracking, cost optimization strategies, and finally a deep-dive on format conversion across the four providers. If you have ever tried to build a multi-provider LLM stack, you already know that the API differences are subtle, painful, and not well documented. This post exists because we wish we had read it before we started.
1. Provider Abstraction Layer
The provider system decouples all AI logic from any single vendor. The core abstraction is BaseProvider — an abstract base class that defines the contract every provider implementation must satisfy. The contract is intentionally narrow: streaming chat, single-turn completion, a human-readable name, and an availability check. Everything else — tool calling, thinking tokens, message format quirks, rate limit headers — is handled inside each implementation. The agent loop never knows which provider it is talking to.
class BaseProvider(ABC):
@abstractmethod
async def stream_chat(
self, system_prompt: str, messages: list[dict], config: ModelConfig,
) -> AsyncIterator[StreamEvent.Delta | StreamEvent.Done | StreamEvent.Error]:
"""Yield streaming SSE events."""
@abstractmethod
async def complete(
self, system_prompt: str, user_message: str, max_tokens: int = 1024,
) -> str:
"""Single-turn non-streaming call."""
@property
@abstractmethod
def name(self) -> str:
"""Human-readable provider name."""
def is_available(self) -> bool:
"""Return True if the provider's API key is configured."""
return True
Four implementations exist today. Each one wraps a different vendor SDK and translates the native response format into Schema Weaver's canonical event stream. The choice to support four providers — instead of just one — was not feature bloat. It was a deliberate response to three real production incidents we observed in similar stacks: an OpenAI region outage that took down a customer's entire pipeline for 6 hours, an Anthropic model deprecation that broke tool calling for a week before anyone noticed, and a Google quota surprise bill that could have been avoided with cheaper fallback routing. Four providers, abstracted correctly, eliminate all three failure modes.
| Provider | Class | Key Models | SDK |
|---|---|---|---|
| AWS Bedrock | AWSBedrockProvider |
Nova Micro/Lite, DeepSeek, Qwen, Claude via Bedrock | boto3 (Converse API) |
| OpenAI | OpenAIProvider |
GPT-4o, GPT-4o-mini, o1, o3 | openai (AsyncOpenAI) |
| Anthropic | ClaudeProvider |
Claude Sonnet, Haiku | anthropic (AsyncAnthropic) |
| Google GenAI | GeminiProvider |
Gemini Flash, Pro | google-genai |
The implementation details differ wildly across these four providers. OpenAI's Chat Completions API uses a flat message array where tool calls are streamed as indexed deltas. Anthropic enforces alternating user/assistant roles, expects the system prompt as a top-level parameter rather than a message, and requires tool results to be embedded inside user messages as tool_result content blocks. Google GenAI uses a Content and Part object model with FunctionCall parts, and it mandates that the final message in any conversation must be from the user role — Schema Weaver automatically appends a dummy "Continue." user message when needed to satisfy this constraint. AWS Bedrock's Converse API enforces strict alternation between user and assistant, requires the conversation to begin with the user role, and wraps tool schemas under toolSpec structures. Every one of these differences is hidden behind BaseProvider.
2. Two Routing Systems
One of the more subtle design decisions in this architecture is the existence of two completely separate routing systems. Most multi-provider stacks try to use a single router for everything. That works until you realize that background tasks (compaction, fact extraction, planning) and the streaming agent loop have fundamentally different requirements. Background tasks want the cheapest model that works, with automatic fallback if it is down, and they do not need to stream. The agent loop wants native streaming tool calling, must hold the conversation with the model the user selected, and cannot silently fall back without breaking the UX. So we built two routers.
notation?"} DOT_CHECK -->|Yes| AWS["AWS Bedrock"] DOT_CHECK -->|No| GEMINI_CHECK{"model starts
with 'gemini'?"} GEMINI_CHECK -->|Yes| GEMINI["Google GenAI"] GEMINI_CHECK -->|No| DEFAULT["AWS → Google → OpenAI → Claude"]
2.1 ModelRouter — Non-Streaming (Legacy + Background Tasks)
The ModelRouter handles non-streaming calls used by background tasks. These tasks include context compaction (when the conversation approaches the token limit, we summarize older turns into a compact representation), fact extraction (pulling durable facts out of the conversation for long-term memory), and planning (generating the next-step plan before the agent starts iterating). All three are fire-and-forget calls — we send a prompt, wait for the full response, and move on. The router's job is to pick the cheapest available model that can do the job and fall back gracefully if it is down.
The router's decision tree is shown above. If the caller marks the request as cost-preferred (which all background tasks do), the router walks a priority chain: AWS Bedrock Nova Micro first (the cheapest model in the stack at roughly $0.035 per million input tokens), then Google Gemini Flash Lite, then OpenAI GPT-4o-mini, then Anthropic Claude Haiku. If the caller specifies a task type — for example, chart_gen for chart generation — the router bypasses the cost chain and routes directly to OpenAI, because OpenAI's structured JSON output mode is currently the most reliable for producing chart specifications that downstream code can parse. The default path checks for dot-notation vendor prefixes (which indicate Bedrock-hosted models) and gemini- prefixed model IDs before falling through to the cost chain.
_CHEAP_MODEL = {
"aws": "amazon.nova-micro-v1:0",
"google": "gemini-2.5-flash-lite",
"openai": "gpt-4o-mini",
"claude": "claude-haiku-4-5-20251001",
}
2.2 ReactProviderContext — Streaming (Agent Loop)
The agent conversation loop is different. When a user asks the Data Explorer a question, they pick a model from the UI — say, Claude Sonnet 4.5 or GPT-4o. The agent then runs a ReAct loop with that specific model for up to 15 iterations, calling tools and reasoning about results. The model cannot be silently swapped mid-loop without breaking the user's expectation. So instead of a fallback chain, the streaming path uses a simpler resolver: resolve_react_provider() inspects the model ID string and returns a ReactProviderContext that tells the streaming code which native SDK to use.
def resolve_react_provider(model: str) -> ReactProviderContext:
"""
Detection rules (in priority order):
1. Dot-notation vendor prefix → Bedrock
2. "gpt-" / "o1" / "o3" prefix → OpenAI
3. "claude-" prefix → Anthropic
4. "gemini-" prefix → Google
5. Fallback → OpenAI with gpt-4o-mini
"""
# Bedrock vendors: amazon., deepseek., qwen., moonshotai., meta., ...
if any(model.startswith(v) for v in _BEDROCK_VENDORS):
return ReactProviderContext(provider="bedrock", model=model)
if model.startswith(("gpt-", "o1", "o3")):
return ReactProviderContext(provider="openai", model=model)
if model.startswith("claude"):
return ReactProviderContext(provider="anthropic", model=model)
if model.startswith("gemini"):
return ReactProviderContext(provider="gemini", model=model)
return ReactProviderContext(provider="openai", model=INTERNAL_FALLBACK_MODEL)
This resolver supports 20+ model IDs through the dot-notation convention AWS Bedrock uses. A model ID like amazon.nova-micro-v1:0 routes to Bedrock. A model ID like anthropic.claude-sonnet-4-v1:0 also routes to Bedrock, because Bedrock hosts Claude models through its own API — the same Claude model can be called either directly through Anthropic's native SDK (using the ID claude-sonnet-4-5) or through Bedrock (using the ID anthropic.claude-sonnet-4-v1:0). The dot-notation check catches the Bedrock variant and routes correctly. This sounds trivial until you realize that getting it wrong means sending a Bedrock-style model ID to the Anthropic SDK, which will reject it with a vague "model not found" error and leave you debugging for an hour.
amazon.nova-micro-v1:0 → Bedrock
amazon.nova-lite-v1:0 → Bedrock
deepseek.v3.2 → Bedrock
qwen.qwen3-32b-v1:0 → Bedrock
anthropic.claude-sonnet-4-v1:0 → Bedrock (Claude via Bedrock)
openai.gpt-oss-120b-1:0 → Bedrock
gpt-4o → OpenAI
claude-sonnet-4-5 → Anthropic
gemini-2.5-flash → Google
3. Unified Streaming Interface
The stream_tool_llm() function is the unified entry point for all streaming LLM calls in the agent loop. It accepts a canonical message list, an optional tool schema list, a tool choice preference, a max tokens cap, and a provider context. It yields a stream of tuples — either ("thinking_chunk", str) for reasoning model thinking deltas, ("text_chunk", str) for visible response deltas, or ("done", ToolLLMResult) for the final accumulated result. Every provider implementation produces this same tuple stream. The agent loop does not switch on provider type — it just consumes the stream.
async def stream_tool_llm(
messages: list[dict],
tools: list[dict] | None = None,
tool_choice: str = "auto",
max_tokens: int = 2048,
provider_ctx: ReactProviderContext | None = None,
) -> AsyncGenerator[tuple[str, Any], None]:
"""
Yields:
("thinking_chunk", str) — real-time thinking delta
("text_chunk", str) — real-time text delta
("done", ToolLLMResult) — accumulated result with tool calls
"""
if provider_ctx.provider == "openai":
async for e in _stream_openai_tools(...): yield e
elif provider_ctx.provider == "anthropic":
async for e in _stream_anthropic_tools(...): yield e
elif provider_ctx.provider == "gemini":
async for e in _stream_gemini_tools(...): yield e
elif provider_ctx.provider == "bedrock":
async for e in _stream_bedrock_tools(...): yield e
All four providers normalize their output to the same ToolLLMResult dataclass. This is the contract that makes the abstraction real. Without it, the agent loop would need to know how to extract tool calls from four different response formats — and that knowledge would leak across the codebase. By forcing every provider to produce the same shape, the agent loop stays provider-agnostic.
@dataclass
class ToolLLMResult:
content: str | None # Text response
tool_calls: list[_TC] # Parsed tool calls
input_tokens: int # Token usage
output_tokens: int
model_used: str = ""
thinking_content: str | None = None # Native thinking (Claude, o1/o3, Gemini)
3.1 Provider-Specific Adaptations
The hard part of building this unified interface is not the abstraction — it is the per-provider adaptation. Each SDK streams tool calls differently, formats messages differently, and has its own quirks around thinking tokens and usage reporting. This section walks through each provider's adaptation so you understand exactly what the abstraction is hiding.
OpenAI uses the streaming Chat Completions API with stream_options={"include_usage": True}. Tool calls arrive as deltas that must be accumulated across chunks. OpenAI streams the function arguments character-by-character across multiple tool_calls deltas, indexed by position. The adapter maintains an tool_call_accumulators dictionary keyed by index, appending the argument fragments as they arrive. Only when the stream closes are the accumulated strings parsed into complete tool call objects. This is straightforward but error-prone — if you miss a chunk or misorder them, the JSON arguments become invalid and the tool call fails silently.
# OpenAI streams tool call arguments in fragments
for tc_delta in delta.tool_calls:
idx = tc_delta.index
if idx not in tool_call_accumulators:
tool_call_accumulators[idx] = {
"id": tc_delta.id, "name": tc_delta.function.name,
"arguments": tc_delta.function.arguments or "",
}
else:
acc[idx]["arguments"] += tc_delta.function.arguments
Anthropic requires message format conversion. Tool results must be embedded in user messages with tool_result content blocks, and consecutive tool results must be merged into the same user message to satisfy Anthropic's alternating role requirement. The conversion function walks the canonical message list and produces Anthropic's expected structure, splitting out the system prompt into a top-level parameter and rewriting tool messages as user-role tool_result blocks. Anthropic also supports extended thinking via thinking stream events, which Schema Weaver surfaces as thinking_chunk tuples in the unified stream.
def _convert_messages_to_anthropic(messages):
# OpenAI's "tool" role → Anthropic's "user" with tool_result content
if role == "tool":
tool_block = {
"type": "tool_result",
"tool_use_id": tc_id,
"content": content,
}
# Merge into previous user message
if result[-1]["role"] == "user":
result[-1]["content"].append(tool_block)
Gemini uses the Google GenAI SDK with generate_content_stream. Tool calls arrive as FunctionCall parts inside candidate content blocks. The adapter iterates over candidate parts, checks for the presence of thought (Gemini's native thinking token) or function_call attributes, and yields the appropriate tuple type. Gemini's thinking tokens are particularly useful for the Data Explorer's analyze intent — when a user asks for a complex analytical breakdown, Gemini's visible reasoning helps the user understand why the agent is taking a particular path. The adapter preserves these tokens through to the UI.
for part in chunk.candidates[0].content.parts:
if hasattr(part, "thought") and part.thought:
yield ("thinking_chunk", part.thought)
elif hasattr(part, "function_call") and part.function_call:
tool_calls.append(_TC(
id=f"call_{uuid4().hex[:8]}",
function=_Fn(name=fc.name, arguments=json.dumps(dict(fc.args))),
))
Bedrock uses the AWS Converse API via boto3. It supports the same dot-notation model IDs described in the routing section. The Converse API is structurally similar to OpenAI's Chat Completions but uses a different vocabulary — tool calls are toolUse blocks, tool results are toolResult blocks, and the message content is always an array of typed blocks rather than a string. The adapter translates between this vocabulary and the canonical OpenAI-style format. Bedrock's streaming API (ConverseStream) emits events for contentBlockStart, contentBlockDelta, and contentBlockStop, which the adapter must stitch together to reconstruct complete tool call arguments.
async for event_type, event_data in provider.stream_with_tools(
messages=messages, tools=tools, tool_choice=tool_choice,
max_tokens=max_tokens, model=model,
):
if event_type == "thinking_chunk": yield ("thinking_chunk", event_data)
elif event_type == "text_chunk": yield ("text_chunk", event_data)
elif event_type == "done":
# Normalize Bedrock's response to ToolLLMResult
4. Circuit Breaker System
Each LLM provider is guarded by a per-provider circuit breaker that prevents cascading failures. The circuit breaker is a classic three-state machine: CLOSED (all calls go through), OPEN (calls skip instantly with zero wait), and HALF_OPEN (one probe call is allowed to test if the provider has recovered). The breaker transitions to OPEN after three consecutive failures, waits 60 seconds, then transitions to HALF_OPEN to probe. If the probe succeeds, the breaker closes. If the probe fails, the timer resets and the breaker stays open for another 60 seconds.
The configuration is deliberately simple. Three failures is low enough to detect outages quickly but high enough to absorb transient network blips. Sixty seconds is long enough to give a flaky provider time to recover but short enough that the breaker does not stay open past the actual outage window. These numbers came from observing real failure patterns in production — they are not theoretical.
FAILURE_THRESHOLD = 3 # Consecutive failures before OPEN
RECOVERY_TIMEOUT = 60.0 # Seconds before probing again
The circuit breaker is consumed inside the ProviderPool (covered in the next section). Before attempting a call to any provider, the pool checks the breaker state. If the breaker is OPEN, the call is skipped instantly — no network attempt, no DNS lookup, no TCP handshake, no waiting for a timeout. This is the critical optimization. Without circuit breakers, when Bedrock is flaky, each failed request waits for a 10-30 second timeout before trying the next provider. With circuit breakers, dead providers are skipped in under 1 millisecond. The difference between a 30-second timeout and a 1-millisecond skip is the difference between a usable product and a broken one during a partial outage.
cb = _breakers["bedrock"]
if cb.should_attempt():
try:
result = await AWSBedrockProvider().call_with_tools(...)
cb.record_success() # → CLOSED
return result
except Exception:
cb.record_failure() # Increments counter, may trigger OPEN
else:
# Circuit OPEN — skip instantly, try next provider
errors.append("bedrock:circuit_open")
5. ProviderPool — Fallback Chain
The ProviderPool implements a prioritized fallback chain for non-streaming internal calls — intent classification, planning, fact extraction, compaction. These are the calls that power the agent's behind-the-scenes intelligence, and they do not need to use the user-selected model. They need to be cheap, fast, and resilient. The pool walks a priority chain, skipping any provider whose circuit breaker is open, and raises a runtime error only if every provider in the chain fails.
breaker OPEN?"} B1 -->|CLOSED - try| B1C["Attempt Bedrock Nova Micro"] B1 -->|OPEN - skip| O1{"OpenAI gpt-4o-mini
breaker OPEN?"} B1C -->|Success| DONE["Return result"] B1C -->|Failure| O1 O1 -->|CLOSED - try| O1C["Attempt OpenAI gpt-4o-mini"] O1 -->|OPEN - skip| G1{"Gemini Flash Lite
breaker OPEN?"} O1C -->|Success| DONE O1C -->|Failure| G1 G1 -->|CLOSED - try| G1C["Attempt Gemini Flash Lite"] G1 -->|OPEN - skip| B2{"Bedrock Nova Lite
breaker OPEN?"} G1C -->|Success| DONE G1C -->|Failure| B2 B2 -->|CLOSED - try| B2C["Attempt Bedrock Nova Lite"] B2 -->|OPEN - skip| ERROR["RuntimeError - all providers failed"] B2C -->|Success| DONE B2C -->|Failure| ERROR
The priority order is deliberate. Bedrock Nova Micro is first because it is the cheapest model in the stack at roughly $0.035 per million input tokens — roughly 4x cheaper than GPT-4o-mini and 85x cheaper than Claude Sonnet. If Nova Micro is up, we use it. OpenAI GPT-4o-mini is second because it is fast, reliable, and well-supported. Gemini Flash Lite is third because Google's free tier is generous and the model is competent for short classification tasks. Bedrock Nova Lite is the final fallback — it is slightly more expensive than Nova Micro ($0.06 per million input tokens) but more capable, and it shares the Bedrock circuit breaker with Nova Micro, which means if Bedrock is fully down, this fallback also fails (correctly, because we do not want to keep retrying a dead provider).
The circuit breaker system exposes health stats via the health endpoint. This is critical for observability — without it, you cannot tell whether your agent is actually using the cheap path or silently falling back to the expensive path every call. The stats include the current state (CLOSED, OPEN, HALF_OPEN), the consecutive failure count, and the configured threshold. In production, we alert on any breaker that stays OPEN for more than 5 minutes — that indicates a real outage, not a transient blip.
def circuit_stats(self) -> dict:
return {name: cb.stats() for name, cb in _breakers.items()}
# Example response
{
"bedrock": {"name": "bedrock", "state": "CLOSED", "failures": 0, "threshold": 3},
"openai": {"name": "openai", "state": "CLOSED", "failures": 0, "threshold": 3},
"gemini": {"name": "gemini", "state": "OPEN", "failures": 3, "threshold": 3},
}
6. Token Usage Tracking
Every LLM call — whether streaming or non-streaming, user-facing or internal — tracks token usage. This is non-negotiable. Without per-call token accounting, you cannot bill users accurately, you cannot detect runaway loops, and you cannot answer the simple question "how much did this query actually cost." The tracking is baked into the streaming interface itself: every ToolLLMResult carries input_tokens and output_tokens, and every conversation that consumes the stream feeds those numbers into the conversation's usage ledger and the active budget.
# In the conversation loop
conv.add_usage(result.input_tokens, result.output_tokens, result.model_used)
# Budget consumption
if budget:
budget.consume_tool(result.input_tokens, result.output_tokens, result.model_used)
The AgentBudget tracks cumulative costs per session. It enforces a hard ceiling on credits consumed — when credits_remaining hits zero, the agent loop terminates gracefully with a budget-exceeded message rather than silently burning through the user's allocation. This is what prevents a misbehaving agent from running 15 iterations against Claude Sonnet 4.5 (at $3 per million input tokens and $15 per million output tokens) on a query that should have been answered in 2 iterations on Nova Micro.
@dataclass
class AgentBudget:
user_id: str
session_id: str
synthesis_model: str
max_tokens: int
credits_allocated: float
max_duration_secs: int
credits_used: float = 0.0
def consume_tool(self, input_tokens, output_tokens, model):
cost = credit_cost(model, input_tokens + output_tokens)
self.credits_used += cost
@property
def credits_remaining(self):
return max(0, self.credits_allocated - self.credits_used)
Credit costs are computed using a pricing configuration that maps model names to per-token rates. The pricing table is updated whenever a vendor changes their prices — AWS Bedrock and OpenAI have both adjusted pricing multiple times in the last 12 months, and stale pricing tables lead to either overcharging users (bad for trust) or undercharging them (bad for sustainability). The rates shown below are illustrative; the real table is loaded from configuration so it can be updated without code changes.
# Example pricing (sanitized, illustrative)
PRICING = {
"amazon.nova-micro-v1:0": {"input": 0.035, "output": 0.14}, # per 1M tokens
"amazon.nova-lite-v1:0": {"input": 0.06, "output": 0.24},
"gpt-4o-mini": {"input": 0.15, "output": 0.60},
"claude-sonnet-4-5": {"input": 3.00, "output": 15.00},
"gemini-2.5-flash-lite": {"input": 0.015, "output": 0.038},
}
7. Cost Optimization Strategies
Multi-provider abstraction is necessary for resilience, but it is also an opportunity for cost optimization. Once you have four providers abstracted behind a single interface, you can route each call to the cheapest model that can handle it. This section documents the four optimization strategies we use and the combined savings they produce. The headline number: a typical 5-iteration agent run costs roughly $0.0004 with optimization, versus $0.005 without — a 92% reduction. At scale, that is the difference between a sustainable product and a money pit.
7.1 Intent-Based Model Selection
The agent classifies each user query into one of several intents — explore, analyze, report — before routing. Each intent maps to a different model tier. The explore intent (basic schema discovery, table listing, row sampling) uses Nova Micro at $0.035 per million input tokens, because exploration does not require deep reasoning. The analyze intent (statistical computation, anomaly detection, correlation) uses Nova Lite at $0.06 per million input tokens, because analysis requires more capable reasoning but not frontier-model quality. The report intent (summarization, natural language synthesis) also uses Nova Lite. Frontier models like Claude Sonnet and GPT-4o are reserved for the user-selected streaming call, where the user explicitly opted into the more expensive model.
7.2 Internal vs. User-Facing Calls
Internal tool-calling iterations use the cheapest available model via the ProviderPool. Only the user-selected model is used for the final streaming synthesis call, and only that call is charged to user credits. This means a user who selects Claude Sonnet 4.5 for their query pays for Claude Sonnet 4.5 on the final synthesis, but the 5-14 internal iterations that produced the data the synthesis uses run on Nova Micro at our cost. We absorb that cost as platform infrastructure, the same way a database tool absorbs the cost of running queries.
7.3 Context Compression
By compressing tool results from roughly 8K tokens down to 80 tokens, each iteration consumes significantly fewer input tokens. The compression strategy is documented in Part 3 (The Tooling System), but the cost impact belongs here: every iteration after the first carries the accumulated context of all previous iterations. Without compression, a 5-iteration run would carry 40K tokens of tool results alone. With compression, it carries 400 tokens. At Nova Micro rates, that is a 100x reduction in tool-result token cost.
7.4 Tool Filtering
By filtering tools by intent, the tool schema definition block shrinks from roughly 18K tokens (all 50+ tools) to roughly 2K tokens (only the tools relevant to the classified intent). Each iteration sends this schema block to the LLM, so the savings compound across iterations. A 5-iteration run saves 80K tokens of schema overhead — significant even at Nova Micro rates, dramatic at Claude Sonnet rates.
7.5 Combined Savings
The table below shows the combined impact of all four optimizations on a typical 5-iteration agent run. The "without optimization" column assumes all iterations run on the user-selected model with full tool schemas and uncompressed tool results. The "with optimization" column reflects the actual production path.
| Without optimization | With optimization | Savings |
|---|---|---|
| 5 × 18K tool schemas = 90K | 5 × 2K = 10K | 89% |
| 5 × 8K tool results = 40K | 5 × 80 = 400 | 99% |
| Total: ~130K tokens | Total: ~10.4K tokens | 92% |
At Nova Micro rates, this is the difference between roughly $0.005 and $0.0004 per query. Multiplied across thousands of queries per day, the savings are the difference between a product with a sustainable unit economics and one that loses money on every active user. The optimization is not a nice-to-have — it is the financial foundation of the entire product.
8. Format Conversion & Adaptation Deep-Dive
Schema Weaver establishes a unified, multi-turn, tool-calling conversation flow. Because the underlying AI APIs accept widely divergent request structures and stream types, the orchestrator implements specialized adapters. This section documents exactly how inputs, schemas, and streams are adapted across providers — the kind of detail that is missing from every vendor's documentation and that you only learn by hitting the errors in production.
8.1 Internal Canonical Format
Schema Weaver selects OpenAI's Chat Completion structure as its internal canonical standard. Every message in a thread is parsed and tracked internally using standard roles and parameters: system holds base context, session info, and constraints; user holds explicit query inputs; assistant holds generated replies, potentially including structured tool_calls; and tool holds output values returned by tool execution, mapped via tool_call_id. By using a single, well-documented model internally, Schema Weaver decouples all database query, routing, retry, and session-limit logic from provider-specific formats. Every other component in the system — the agent loop, the context compressor, the budget tracker, the artifact emitter — works with the canonical format and never sees provider-specific structures.
8.2 Per-Provider Message Conversions
When sending the conversation history to non-OpenAI providers, the messages list is mapped on-the-fly. This mapping is not a simple key rename — it involves structural reshaping, role enforcement, and content block transformations. The three non-OpenAI providers each have their own quirks.
Anthropic (Claude) Conversion. The conversion function _convert_messages_to_anthropic() handles four things. First, system prompt separation: Anthropic expects the system prompt as a separate top-level request parameter (system), not as a message inside the main history list. Second, assistant tool calls: OpenAI-format tool calls (with arguments as stringified JSON) are expanded into Anthropic's block array format, where each tool call becomes a tool_use block with input as a parsed JSON object rather than a string. Third, tool output mapping: Anthropic requires tool outputs to be formatted as tool_result blocks within a message of role user, not as a separate tool role. Fourth, message merging: because Anthropic enforces alternating user and assistant messages, consecutive tool outputs must be compressed. If the previous message in the mapped list is already a user message, subsequent tool_result blocks are appended to that message's content array rather than starting a new message.
if result and result[-1]["role"] == "user":
prev = result[-1]["content"]
if isinstance(prev, str):
result[-1]["content"] = [{"type": "text", "text": prev}, tool_block]
elif isinstance(prev, list):
prev.append(tool_block)
Google GenAI (Gemini) Conversion. The Gemini adapter handles four transformations. System messages are passed into GenerateContentConfig(system_instruction=...) using the google.genai SDK's Content and Part primitives. Assistant tool calls are mapped to Gemini's model role Content objects — if the assistant generated text alongside a tool call, they are packaged as separate Part instances (Part.from_text plus gtypes.FunctionCall parts) inside the same Content container. Tool outputs are mapped to Gemini's user role Content objects containing FunctionResponse parts. Similar to Anthropic, consecutive tool results are appended as parts of the same active Content block to maintain correct turn flow. Finally, Gemini mandates that the final message in the history list must be from the user — if the message list terminates after an assistant block, Schema Weaver automatically appends a dummy user content block saying "Continue." to satisfy the validation rule.
part = gtypes.Part(
function_response=gtypes.FunctionResponse(name=fn_name, response=result_data)
)
AWS Bedrock Converse Conversion. The Bedrock adapter _build_converse_request() handles three transformations. System prompts are injected as system=[{"text": system_text}]. Assistant tool calls are mapped to assistant message objects with toolUse blocks, with stringified arguments parsed back into native JSON dictionaries. Tool outputs are mapped to user message objects with toolResult blocks. The hard part is the strict alternation enforcement: the Bedrock Converse API requires that the message chain must begin with the user role and that roles must strictly alternate between user and assistant. If the first message in the canonical format is from the assistant, Schema Weaver inserts a blank user block at the start. If consecutive messages share the same role, they are grouped and merged under their respective content arrays to ensure strict role switching. Getting this wrong produces a 400 error with a message like "Message at position 2 has role 'assistant' but expected 'user'" — and the message is not always accurate about which position is actually wrong.
tool_result = {
"toolResult": {
"toolUseId": tc_id,
"content": [{"json": result_content}],
"status": "success",
}
}
8.3 Tool Schema Transformations
To describe tools to the LLMs, Schema Weaver defines their schemas using standard JSON schema contracts. Adapters translate these definitions on-the-fly for each provider. The internal format uses OpenAI's structure — type: "function" with a nested function object containing name, description, and parameters. Each adapter unwraps or rewires this structure to match its provider's expected format.
| Provider | Internal Schema Key | Mapped Name | Adapter Translation |
|---|---|---|---|
| OpenAI | parameters |
parameters |
Identity mapping — standard JSON schema format |
| Anthropic | parameters |
input_schema |
Statically maps schema fields, stripping the outer "type": "function" envelope |
| Gemini | parameters |
parameters |
Converted to gtypes.FunctionDeclaration object wrappers |
| AWS Bedrock | parameters |
inputSchema |
Wrapped under toolSpec with inputSchema: {"json": schema} structures |
The structural difference looks small in a table, but it matters at implementation time. The OpenAI format nests the schema under a function key. Anthropic flattens it — the name, description, and input_schema sit at the top level of the tool definition, not nested. Gemini wraps the schema in a FunctionDeclaration object that requires explicit construction. Bedrock wraps it under toolSpec with an additional inputSchema envelope that contains a json key holding the actual schema. Every one of these formats is valid JSON Schema at the leaves — the differences are all in the envelope structure. The adapter's job is to produce the right envelope.
OpenAI Format:
{
"type": "function",
"function": {
"name": "list_tables",
"description": "...",
"parameters": { "type": "object", "properties": { ... } }
}
}
Anthropic Format:
{
"name": "list_tables",
"description": "...",
"input_schema": { "type": "object", "properties": { ... } }
}
8.4 Adapter-Level Tool Call Accumulation Strategies
LLM providers differ significantly in how they emit tool calls during server streaming. Adapters must parse and rebuild these partial frames. This is the part of the abstraction that is hardest to get right — a single missed chunk or misindexed accumulator can corrupt the entire tool call argument string, leading to JSON parse failures downstream that are nearly impossible to debug without logging every raw chunk.
OpenAI Fragment Accumulation. OpenAI streams tool calls in small text deltas. The adapter holds an index-keyed dictionary tool_call_accumulators to stitch these fragments. Each delta may contain a partial function name, a partial argument string, or just an ID — the adapter must check every field on every delta and append whichever fields are present. This is fragile because the deltas are not guaranteed to arrive in order, and the ID may arrive in a different delta than the name or the first argument fragment.
if delta.tool_calls:
for tc_delta in delta.tool_calls:
idx = tc_delta.index
if idx not in tool_call_accumulators:
tool_call_accumulators[idx] = {
"id": tc_delta.id or "",
"name": tc_delta.function.name or "",
"arguments": tc_delta.function.arguments or "",
}
else:
acc = tool_call_accumulators[idx]
if tc_delta.id: acc["id"] = tc_delta.id
if tc_delta.function.name: acc["name"] += tc_delta.function.name
if tc_delta.function.arguments: acc["arguments"] += tc_delta.function.arguments
Anthropic Native Messages Stream. Anthropic streams standard content deltas (such as thinking blocks and text blocks) for the visible response. However, for tool calls, it signals their creation via event blocks and yields the final completed tool objects at the close of the stream. Schema Weaver captures this by awaiting stream.get_final_message() and extracting the parsed tool calls directly. This is simpler than OpenAI's fragment accumulation, but it means tool calls are not available until the stream fully closes — there is no way to start executing a tool call mid-stream with Anthropic.
Gemini Candidates Parsing. Gemini streams function calls block-by-block. When candidate content parts are detected, the adapter checks if they match the FunctionCall type, collects the arguments directly as a dictionary (Gemini sends arguments as a parsed dict, not a stringified JSON), and converts them to standard _TC objects. This is the cleanest of the four streaming models — Gemini's SDK does most of the parsing work, and the adapter just normalizes the output shape.
Bedrock ConverseStream Event Parsing. Bedrock streams events sequentially. The adapter listens for contentBlockStart to register a new tool invocation block index, gathers incremental JSON text chunks via contentBlockDelta toolUse deltas, and stitches them under tool_accumulators[block_index]["input_str"]. The block index is critical — Bedrock can interleave multiple tool calls in a single response, and without the index, you cannot tell which argument fragment belongs to which call. The adapter also handles contentBlockStop to finalize each accumulator into a complete tool call object.
Text-Embedded Tool Call Parsing (Kimi). For specific models hosted on Bedrock that bypass native tool calling — such as Kimi K2 Thinking — tool requests are written as formatted tags inside the raw text completion stream. The Bedrock adapter detects these patterns, extracts the function name and arguments via regex, maps them to standard tool call inputs, and cleans the special tokens from the user-facing text stream before passing it forward. This is a fallback path for models that do not support native tool calling but are cheap enough or capable enough in other ways to be worth supporting. The tag format looks like <|tool_calls_section_begin|> <|tool_call_begin|> functions.name:id <|tool_call_argument_begin|> {args} <|tool_call_end|> — ugly, but functional.
8.5 Latency Cost vs. Inference Latency Analysis
Because Schema Weaver acts as a proxy translating formats, developers occasionally worry about the latency cost of conversion. A profiling audit of Schema Weaver's execution loop clarifies this concern. The total request latency is dominated by inference and network I/O — the in-memory format conversion represents less than 0.02% of the total response time. The overhead is completely negligible.
Total Request Latency: 100%
█ 100% (Inference & Network I/O, ~3200ms)
░ <0.1% (In-Memory Format Conversion, <1ms)
Format conversions are mapped entirely in-memory using highly optimized native dictionaries and string splits. The entire conversion process — mapping messages, translating tool schemas, and updating accumulators — takes under 0.5 milliseconds per request, on the microsecond scale. The median latency of the LLM provider API call ranges from 1.5 to 3.5 seconds, and the P95 latency is 8.5 seconds. This is entirely dominated by HTTP network handshakes, server queuing, and auto-regressive token generation on the provider side. When the inference step takes 3,000 milliseconds and the conversion step takes 0.5 milliseconds, optimizing the conversion step is a waste of engineering time. We mention this because we have seen teams try to "optimize" their multi-provider stack by collapsing the abstraction layer into provider-specific code paths, trading a 0.02% latency saving for a 100% increase in code complexity and a complete loss of provider-agnostic routing. Do not do this. The abstraction is worth keeping.



