LLM Cost Tracking For AI Agents
How to implement LLM cost tracking for AI agents by customer, workflow, step, model, provider, retry, and request before the provider invoice arrives.
LLM cost tracking for AI agents means recording every supported model call with provider, model, token counts, latency, status, customer_id, workflow, step, retry context, and request scope. The useful output is not a token spreadsheet. It is a customer-level cost record that shows which workflow, loop, model, or account created spend before the provider invoice arrives.
- - How do I track LLM cost for each customer?
- - What should an AI agent cost event include?
- - How do I attribute OpenAI and Anthropic spend to agent steps?
- - How do I track AI agent cost without sending prompts?
- - How do I control AI spend from agent loops and retries?
Why LLM Cost Tracking Cannot Wait For The Bill
Agentic systems can trigger dozens of model calls from one user action. A single run may include planning, retrieval, generation, critique, formatting, retries, and reflection loops. AI cost becomes an output of system behavior, not a number you can estimate once from a provider pricing page.
The failure mode is familiar: usage grows, a teammate routes one step to a more capable model, retry logic fires more often, and the next provider invoice jumps. Without cost attribution, no one can explain which customers, workflows, steps, or model choices created the spike.
Menlo Ventures estimated that model API spending more than doubled from late 2024 to mid-2025 as production inference grew. Treat that as market context, not a Pylva performance claim. The operational takeaway is narrower: production LLM usage moves fast enough that teams need customer-level cost records before invoices arrive.
In one widely reported OpenClaw example, autonomous coding agents reportedly generated a seven-figure OpenAI API bill in 30 days. Most teams will not see that scale, but the shape of the problem is the same: loops, retries, model mix, and background agents can move cost faster than manual review.
Pylva is SDK-first cost infrastructure for AI agent companies. It focuses on cost-shaped telemetry such as provider, model, token counts, latency, status, customer ID, and step name. It is not a prompt logging product, a full observability replacement, or an infrastructure monitor for self-hosted model clusters.
Step-By-Step Setup
Effective LLM cost tracking starts with request-level detail across three layers: the customer, the workflow, and the individual model call. Start with one production workflow, then expand once the event shape is stable.
- Choose a stable customer identifier such as
workspace_id,org_id, orstripe_customer_id. Avoid raw email addresses and other direct personal identifiers. - Define a clear naming scheme for agent workflows and steps. Examples:
onboarding_agent.plan,onboarding_agent.fetch_docs, andonboarding_agent.summarize. - Instrument supported LLM calls where they happen, inside the service or agent runtime. Include background agents, scheduled tasks, batch jobs, retries, and fallback paths.
- Compute cost server-side by combining usage with provider pricing tables. Client-side cost scripts and spreadsheets drift when model prices, retry logic, and workflow coverage change.
- Review cost metrics in order: cost per request, cost per agent workflow, cost per customer, cost per model, and cost per provider. Then connect the data to budget rules, alerts, and billing workflows.
| Setup Decision | Why It Matters | Example |
|---|---|---|
| Customer identifier | Groups spend by the account that caused it. | acme-corp |
| Workflow name | Keeps cost aligned to product flows. | support_triage |
| Step name | Shows which decision or action created cost. | compose_reply |
| Provider + model | Separates vendor and model mix from workflow volume. | openai / gpt-4o-mini |
| Tokens in + out | Explains the main model-usage driver. | 1200 / 320 |
| Status + latency | Connects cost to failures, retries, and slow paths. | success / 842ms |
Why Provider Dashboards Are Not Enough
OpenAI, Anthropic, and general cloud cost tools can show aggregate LLM cost at the project or account level. They answer "how much did we spend?" They usually do not answer "which customer, workflow, retry loop, or product step created the spend?"
That gap matters for AI agent businesses because cost is not only a vendor bill. It becomes customer margin, plan design, throttling logic, usage pricing, and engineering prioritization.
- Provider dashboards usually cannot show cost per customer, workspace, account, subscription tier, or plan margin.
- Retries and tool failures often look like normal usage on an invoice, even when they came from one broken agent loop.
- Workflow-step attribution is required to see whether planning, retrieval, summarization, critique, or final generation is driving AI spend.
- Engineering teams need request-level traces for expensive paths, while founders and finance teams need customer-level profitability.
Key Cost Dimensions To Track
Break AI spend down into the units you use to operate the product: customers, agents, workflows, steps, models, and providers. Track both the individual call and the aggregate business entity.
Cost Per Request
This is the total cost for a single inbound user request, including downstream model calls, agent loops, and tool-triggered calls. Use it to detect expensive outliers and compare prompt or model changes.
Averages are not enough. Agent costs often have long tails, so inspect distribution, p95, and repeated outliers by workflow.
Cost Per Agent Workflow And Run
Capture the full cost of an autonomous run: planning, retrieval, tool use, critique, and final response. One support agent where triage costs $0.02, summarization costs $0.15, and answer drafting costs $0.40 has a different optimization path from one where every step uses the same frontier model.
Cost Per Step And Tool
Attribute cost to named steps such as retrieve_documents, expand_query, generate_answer, and critique_answer.
Step-level tracking lets engineering leaders ask whether spend belongs to reasoning, retrieval, final generation, or repeated tool handling.
Cost Per Model And Provider
Track spend by provider and model so routing decisions are visible. Agent loops often mix expensive frontier models with cheaper models. Without attribution, a silent routing change can move the cost base without showing up as a clear product decision.
Cost Per Customer And Plan
Attach every event to customer_id and subscription context. That makes gross margin per customer, feature adoption by cost, and enterprise-plan profitability visible.
Internal showback or chargeback becomes possible when teams can slice spend by customer, workspace, team, or cost center.
Design The Attribution Model
Cost attribution assigns AI cost to meaningful business entities: customers, teams, use cases, and agent workflows, not just API keys. Without attribution, a total spend number cannot tell you what to optimize or which customers to reprice.
Keep the schema small enough that dashboards remain readable. It should support engineering diagnosis and finance review without turning every event into an unbounded metadata dump.
- Customer ID: use a stable, low-cardinality identifier such as
org_id,account_id, orworkspace_id. - Workflow name: define a controlled set such as
support_triage,inbox_copilot,research_assistant, oretl_enrichment_agent. - Step name: use consistent names such as
support_triage.classify_intent,support_triage.retrieve_context,support_triage.compose_reply, andsupport_triage.critique_reply. - For chat-style agents, decide whether your financial view is per message, per session, or per completed task. Conversation history can make per-session cost very different from per-request cost.
- Engineering should own the tags emitted by each LLM call. Finance or operations should map those tags to cost centers, customer plans, and margin views.
Implement Tracking In Code
Instrument where LLM calls are made, inside the TypeScript or Python services that orchestrate agents. Capture provider, model, tokens in, tokens out, status, latency, customer ID, and step name for every supported call.
Pylva takes an SDK-first approach. The TypeScript SDK instruments OpenAI, Anthropic, and Vercel AI calls. The Python SDK instruments OpenAI and Anthropic clients. Pylva computes cost server-side and does not transmit prompts, completions, raw user messages, tool inputs, or tool arguments.
For TypeScript services, identify supported provider calls inside planning, tool selection, reflection, and final generation. Wrap the unit of work so each call inherits customer and step context.
import { init, track } from "@pylva/sdk";
init({ apiKey: process.env.PYLVA_API_KEY! });
await track("acme-corp", { step: "evaluate" }, async () => {
return openai.chat.completions.create({
model: "gpt-4o-mini",
messages,
});
});Use Python Tracking Contexts
For Python batch jobs or backend agent runners, instrument all supported OpenAI and Anthropic client calls used by background agents. Make sure retries and fallbacks are tracked so cost is visible rather than hidden as latency.
Use the same step vocabulary across TypeScript and Python services. A support agent that calls classify in one service and classify_ticket in another is harder to compare than one shared label.
import os
import pylva
from openai import OpenAI
pylva.init(api_key=os.environ["PYLVA_API_KEY"])
client = OpenAI()
with pylva.track_context(customer_id="acme-corp", step="evaluate"):
client.chat.completions.create(model="gpt-4o-mini", messages=messages)Handle Agent Loops And Retries
Agent loops and retries are where AI spend often spirals. Planning steps call models repeatedly. Tools fail and trigger fallbacks. Guardrails re-run generations. Without explicit tracking, those costs stay invisible until the invoice arrives.
A study on agentic coding tasks found that agent runs can consume far more tokens than simple chat, with same-task runs varying substantially in token consumption. Use findings like this as a warning about variance, not as a prediction for every product.
- Record every retry as a separate model call with status, step name, and retry context.
- Add a loop marker such as
loop_iterationor a stable suffix such asplan_retry_1when repeated work matters for diagnosis. - Look for steps where one user-visible answer triggers multiple hidden model calls.
- Use cost data to cap loop iterations, downgrade intermediate steps, improve stopping conditions, or route high-volume tasks to cheaper models where quality allows.
Connect Metrics To Budgets And Billing
Tracking LLM cost is useful only when it changes operating decisions. Aggregate cost by customer_id, workflow, step, and model so engineering can diagnose spikes and leadership can review margin.
If Anthropic prompt caching is already changing Claude input economics, use the Anthropic prompt caching cost tracking page to connect cache reads, cache writes, customer context, and billing-period review.
Budget alerts can help manage thresholds. Hard enforcement depends on architecture and available state, so use careful rollout: advisory alerts first, then routing or hard stops where the product can degrade safely.
For usage-based or hybrid SaaS pricing, connect trusted usage records to billing workflows. The goal is not to pass provider invoices directly to customers. The goal is to create a business-owned usage record that can support pricing, invoicing, and customer-facing usage views.
Where Pylva Fits
Many teams already use traces, APM, logs, and metrics tools for application performance and agent observability. Pylva focuses specifically on LLM cost tracking and cost attribution for AI agents.
Pylva instruments supported TypeScript and Python LLM clients directly where agents run, sends cost-shaped telemetry, computes cost server-side, and connects usage records to budget and billing workflows.
Pylva does not replace full-stack observability or prompt evaluation. It does not stream raw user text, prompts, completions, tool inputs, or tool arguments. It is not an infrastructure monitoring tool for self-hosted model clusters.
For the broader buyer workflow around margin views, runtime controls, and billing-ready usage records, read the AI agent cost management software page.
A 30-Day Plan To Control AI Spend
A practical rollout does not require instrumenting every agent path on day one. Start with the workflows that have real customer traffic, visible owner accountability, and enough volume to expose repeated calls or expensive model choices.
- Week 1: define customer_id, workflow names, and step names. Inventory all agent workflows, providers, retries, background jobs, and non-LLM cost sources.
- Week 2: instrument supported TypeScript and Python LLM calls with customer and step context. Confirm retries and fallback paths emit separate telemetry.
- Week 3: build first reports for cost per customer, workflow, model, and step. Review expensive sessions with engineering and product.
- Week 4: add advisory budget rules, anomaly alerts, and billing-review workflows. Test model downgrades, prompt changes, cache behavior, and loop caps where quality can be measured.
- After launch: run a weekly 30 to 45 minute review with engineering, product, and finance. Treat LLM cost like latency and error rate: an operating metric reviewed alongside product KPIs.
Keep Privacy Boundaries Clear
LLM cost tracking should not require prompt capture. Pylva telemetry is designed around cost-shaped metadata such as provider, model, token counts, latency, status, customer_id, and step.
Pylva does not redact customer_id, step, or metadata values. Use stable account or workspace identifiers, and do not place emails, phone numbers, raw user messages, prompts, completions, or tool inputs in those fields.
Turn Tracking Into Cost Management
LLM cost tracking is the first layer. Once model usage is attributed by customer, workflow, and step, teams can compare model mix, inspect high-cost accounts, account for retry loops, decide which non-LLM tools need tracking, and set budget rules where the product can degrade gracefully.
Use this guide for the LLM telemetry layer. Use the AI agent cost management page when the question expands to runtime controls, customer-level margin, usage-based billing, and plan design.
Frequently Asked Questions
Is token counting enough for AI agent cost tracking?
No. Token counts explain part of the bill, but a useful cost record also needs customer_id, workflow, step, provider, model, latency, status, retry context, and enough request scope to explain what created the spend.
Can I track cost without sending prompts?
Yes. Pylva expects cost-shaped telemetry, not prompt or completion text. Keep prompts, completions, tool inputs, and raw user messages out of telemetry fields.
Does cost_usd come from the SDK?
No. The SDK sends usage signals and the backend calculates cost from pricing tables.
Which provider SDKs are covered first?
The current TypeScript SDK covers OpenAI, Anthropic, and Vercel AI. The Python SDK covers OpenAI and Anthropic.
Can I track LangGraph or LangChain runs?
Yes. Pylva has LangGraph and LangChain callback handlers for graph and node attribution. Use either callback attribution or provider auto-instrumentation for the same runtime so you do not double-count model calls.
What should I use as customer_id?
Use an account, workspace, organization, or tenant identifier. Avoid personal contact information because Pylva does not redact customer_id, step, or metadata values.
How should retries be counted?
Count every retry as its own model call with status, step, and retry context. Otherwise failed work can look like normal successful usage on the invoice.
Does LLM cost tracking replace observability?
No. Observability is still useful for traces, latency, errors, evals, and debugging. LLM cost tracking adds the customer, workflow, pricing, budget, and billing context those tools may not own.
Related reading
AI Agent Cost Management Software
The buyer page for customer-level AI cost management, budget rules, and billing-ready usage records.
Anthropic Prompt Caching Cost Tracking
The buyer page for Claude teams that need prompt caching economics tied to customers, workflows, budgets, and billing records.
Per-Customer AI Cost Attribution: See AI Cost By Customer, Plan, And Workflow
How to attribute AI agent cost by customer, plan, workflow, model, retry, tool call, and non-LLM usage before margins drift.
Non-LLM Cost Tracking For AI Agents
How to track search, speech, vector database, workflow, and other non-LLM API costs next to model spend.
Report Usage, Not Cost
Why AI agent instrumentation should emit raw usage metrics while the backend calculates dollars.