LLM cost control
LLM cost control means bounding and predicting LLM inference spend before it happens: pre-call estimates, integer-credit reservations, spend caps, a circuit breaker, and near-duplicate-prompt detection, all enforced before a provider call runs. Caisson's ai-meter package wires these guards together, then trues each reservation to the provider's actual reported usage afterward.
In code
const signature = computeMinHashSignature(
shingle(normalizePrompt(core.messages)),
numHashes,
);
const bucketKeys = lshBands(signature, bands, rows);
const candidates = await config.store.candidates(core.accountId, core.scope, bucketKeys);
let best: { callId: string; similarity: number; at: Date } | null = null;
for (const candidate of candidates) {
const similarity = jaccardEstimate(signature, candidate.signature);
if (similarity >= threshold && (best === null || similarity > best.similarity)) {
best = { callId: candidate.callId, similarity, at: candidate.at };
}
}
// Insert AFTER matching: a call must never match its own just-inserted signature.
await config.store.insert(core.accountId, core.scope, bucketKeys, {
callId: core.callId,
signature,
at: new Date(),
});How it holds
Every call clears the breaker before it spends anything
reserve() checks the circuit breaker first, then estimates the cost and debits the wallet before the provider is ever invoked; an open breaker or a short wallet throws a 402 with nothing written, inside the same withTenant transaction.
Dedup detects a repeat before estimate or debit ever run
checkDedupGate hashes the prompt into a MinHash/LSH signature and matches it against recent calls, called before reserve(), so a caller can skip the whole reservation on a near-duplicate; the gate only detects, it never auto-skips or moves a wallet credit itself.
A crossed hard cap trips the next call, not this one
evaluateCaps runs after the reservation that lands spend at or past the hard limit, so that call is already billed; tripBreaker fires from inside the same transaction and the very next reserve() 402s before a provider is touched.
Settled to actual usage, not the estimate
reconcile() trues the reservation against the provider's reported tokens: a feature_grant refunds an over-reservation, a feature_debit charges a shortfall, and the usage_event (account, call_id) unique constraint makes a retried settlement land exactly once.