A concurrency ladder against PayPilot's webhook ingest path. Every number below carries the command that produced it.
I had never load tested PayPilot. That meant that if anyone asked me where it breaks, the honest answer was that I would be guessing, and a guessed bottleneck sounds exactly like an expert one right up until someone measures it.
So this is two experiments, not one. The first is the ordinary question: how far does it go and where does it fall over. The second is the one I actually built the harness for.
Does the audit trail survive the load. PayPilot's whole design claim is that deterministic rules decide anything touching money, the model only drafts language, and every run records which rule fired and on what input. That claim is easy to hold at one request a second. A system that stays fast while quietly losing its record has failed in the way that costs you an argument with an auditor rather than an argument with a user, and nothing in a normal load test would notice.
Single uvicorn worker, SQLite in WAL mode on disk, 200 requests per step.
| concurrency | throughput rps | p50 ms | p95 ms | p99 ms | errors |
|---|---|---|---|---|---|
| 1 | 328.1 | 3.0 | 3.6 | 4.0 | 0 |
| 2 | 382.4 | 4.9 | 6.3 | 7.4 | 0 |
| 4 | 413.8 | 9.1 | 10.4 | 18.3 | 0 |
| 8 | 409.1 | 18.5 | 22.6 | 29.6 | 0 |
| 16 | 395.6 | 38.5 | 49.2 | 57.6 | 0 |
| 32 | 255.1 | 98.0 | 294.9 | 358.5 | 0 |
| 64 | 140.2 | 246.4 | 1030.1 | 1146.6 | 0 |
Throughput peaks at 4 and then goes backwards. 414, then 409, 396, 255, 140. That distinction matters more than the peak does. A saturated system plateaus: you stop gaining, but you do not lose. A system that gets slower in absolute terms as you add concurrency is spending the extra capacity on waiting, which is contention.
Latency scales linearly with concurrency. 3.0ms at 1, 4.9 at 2, 9.1 at 4, 18.5 at 8, 38.5 at 16. Each doubling of concurrency doubles the wait almost exactly. That is the signature of near-perfect serialisation: every request queueing behind every other one, rather than any of them running together.
Zero errors at every step. It degrades, it does not fail.
The temptation is to name the cause now. The store opens one SQLite connection and guards it with a lock, and FastAPI runs sync endpoints in a threadpool, so the shape fits. Fitting the shape is how you end up confidently fixing the wrong thing. So: remove one clause at a time and watch which removal moves the number.
WAL was already on, so the default rollback journal was never in play. No run spent.
Same ladder with the database entirely in memory, filesystem out of the path.
| at 64 concurrent | throughput rps | p99 ms |
|---|---|---|
| disk, WAL | 136.6 | 1177.4 |
| in memory | 133.7 | 1253.6 |
Identical inside noise. Disk is not the bottleneck. Removing storage from the system changed nothing at the point where it collapses.
Disk is not free, it is just not the ceiling. Below the knee, in-memory is genuinely quicker: 350.4 against 297.1 rps at 1 concurrent, and 461.8 against 406.7 at 16. It buys headroom below the knee and nothing at all above it.
The single-worker baseline also reproduced across separate runs, 140.2 then 136.6 rps at 64 concurrent, inside 3 percent. The collapse is a property of the system, not of one run.
If the serialisation is inside the process, more processes should help. Four uvicorn workers, same ladder, same database file.
| at 64 concurrent | throughput rps | p99 ms |
|---|---|---|
| 1 worker | 136.6 | 1177.4 |
| 4 workers | 190.3 | 791.3 |
39 percent more throughput and a third off p99. At 16 concurrent it is 539.5 against 406.7 rps.
And it is not four times. That is the finding, not the 39 percent. If the only serialisation point were inside one process, four processes should have approached four times. They bought 1.4. Something serialises across processes too.
What I have not proved. The obvious candidate is SQLite itself: WAL permits concurrent readers and exactly one writer per database file, so four processes writing still queue at the file. I have not established that, so I am not claiming it. What the experiment does support is stronger than a guess and weaker than a diagnosis: the ceiling is not disk, it is only partly per-process, and the write path is serialised somewhere that survives process separation.
The honest conclusion is architectural rather than a tuning knob. If this needed to go past roughly 500 requests a second on the write path, the answer is a store that supports concurrent writers, not more workers. For what PayPilot is, a dunning agent reacting to payment failure webhooks, 400 to 500 per second is several orders of magnitude more than the problem requires, which is why nobody had hit it.
At every concurrency step, on one worker and on four, with p99 over a second:
failures with no state: 0
failures with no reason code: 0
messages with no failure row: 0
rows written == 200s returned: every stepThe audit trail did not degrade under load. Every decision the agent made still carried what state it reached and why it reached it. Each model call additionally emitted its own record with a prompt template id, a SHA-256 of the prompt, which guards passed, and whether injection was suspected.
That is the claim I actually wanted to be able to make, and now it has a measurement under it rather than an assertion.
Two of my own instruments were wrong before the system was, and both were caught by measuring something I already knew the answer to.
My first version computed the nearest rank as round(pct / 100 * n + 0.5). round in Python rounds halves to even, so the median of 1 to 10 came back 6. A test over 1 to 10, where a human can count the answer, failed immediately. The same function had a second defect underneath: pct / 100 * n turns the 90th of 10 into 9.000000000000002, which ceilings to 10 and silently returns the wrong percentile. Percentiles have no obvious sanity check, which is exactly why they need a test whose expected values you can verify by counting on your fingers.
My first version treated "a failure row with no transition row" as a lost audit trail. At concurrency 1 it reported five orphans out of five requests, and I nearly wrote that up as a finding. Reading the store settled it: transitions are written on state change only, and an opening state is not a change, so a freshly ingested failure legitimately has none. Every one of the five was correct behaviour.
That check is now reported rather than asserted, with the mechanism written next to it. A number nobody can explain is a coin flip with a green tick on it: when it passes you learn nothing, and when it fails you cannot tell whether the system is wrong or the rule is.
The third thing that went wrong was smaller and worth saying anyway. The first five requests of the ladder wrote 195 rows for 200 requests, and my probe flagged it. It was the smoke test: five invoice ids had already been used, and the ledger deduplicated exactly those five. The mismatch detector found real idempotency working correctly. I have left the detail in because a probe that only ever returns zero has not been shown to work.
Target is POST /webhooks/stripe, signed. Not the demo route: that one runs the graph and returns a draft without touching the ledger, so it cannot answer the second question at all. The Stripe route is where the writes happen, which makes it both the money path and the only path where the audit question exists. A verified Stripe webhook also skips the shared rate limiter, so the curve measures the system rather than the limiter.
Every request carries a unique event id and a unique invoice id. The route replays a cached result for a repeated event id and the ledger keys state on the invoice id, so reusing either would have measured the idempotency cache and reported it as throughput.
OPENAI_API_KEY="" LANGFUSE_PUBLIC_KEY="" LANGFUSE_SECRET_KEY="" \
RESEND_API_KEY="" STRIPE_API_KEY="" PAYPILOT_SEND_EMAIL=0 ADMIN_TOKEN="" \
STRIPE_WEBHOOK_SECRET="loadtest-secret" \
RATE_LIMIT_MAX=100000 RATE_LIMIT_GLOBAL_MAX=100000 \
PAYPILOT_DB_PATH=data/loadtest.db \
.venv/bin/python -m uvicorn app.api:app --host 127.0.0.1 --port 8077
STRIPE_WEBHOOK_SECRET="loadtest-secret" PAYPILOT_DB_PATH=data/loadtest.db \
.venv/bin/python scripts/loadtest.py \
--url http://127.0.0.1:8077/webhooks/stripe \
--ladder 1,2,4,8,16,32,64 --requests 200 --settle 1Every outbound credential is blanked rather than reasoned about. The app loads a dotenv file on import and that file holds a real model key, live tracing keys, and mail and payment credentials. Starting it naively would have put paid model calls in a load loop and shipped fourteen hundred traces to a real project. Setting each to an empty string works because the loader does not override a variable already present in the environment, which I verified by running it rather than by trusting the documentation. The app then reports {"mock": true, "model": "mock"} on its config endpoint, which is the system saying the model is off rather than me saying it.
What this deliberately does not measure. With no API key the drafting node takes the template path, so these numbers are PayPilot's own machinery and say nothing about a model provider's latency. That is the point. A real key puts a network call in the hot loop that dominates everything and hides what I am looking for.