Running 500 prompts without burning money or getting rate-limited

Dry-run first, concurrency that does not trigger 429s, retries that cost less than they save, and how to know what a batch actually cost.

Batch prompt runs fail for two boring reasons: the bill is bigger than expected, and the relay starts returning 429 halfway through. Both are avoidable with five habits.

1 · Dry-run before you spend anything

Render the prompt for the first row and read it. Almost every wasted batch is a prompt with a {{variable}} that does not exist in the data, so it was sent empty 500 times.

python prompt_runner.py --template inbox_triage --data emails.csv --dry-run

A good runner also tells you which referenced columns are missing before the first call.

2 · Concurrency 4, not 40

Relays rate-limit per key. Twenty parallel calls get you a queue of 429s, three retries each, and a bill for the retries. Four workers finishes a 500-row batch in a reasonable time and finishes it once.

ThreadPoolExecutor(max_workers=4)

3 · Retries must be cheap and bounded

Exponential backoff with a small cap, and no retry on the errors that will never succeed:

time.sleep(min(2 ** attempt, 8))   # 2s, 4s, 8s
retries = 3                        # then record the failure, do not loop

Retrying a 401 three times costs three times as much and fixes nothing. Retry 429 and 5xx; fail fast on 400, 401, 403, 404.

4 · Measure cost per row, not per batch

Log usage.total_tokens per row and store it. Then you can answer the only useful questions: which input type is expensive, and does a shorter prompt change quality?

{"row":7,"ms":1840,"tokens":412,"output":"..."}

A 500-row batch at 400 tokens in and 120 out is roughly 260k tokens. If that number surprises you, the prompt is carrying context it does not need — most batch prompts are 80% boilerplate, and trimming the instructions that do not change behaviour is usually a 3 to 5 times saving.

5 · Test on five, read all five

Run --limit 5, then actually read the outputs. Schema-shaped output can be perfectly well-formed and completely useless: every field populated, every field wrong. Shape is a necessary condition, not a sufficient one.

The runner that does all of this

prompt_runner.py in Prompt Pack: dry-run, workers (default 4), bounded backoff, per-row latency and token counts, and failed rows kept in the workbook with their error text. 19 USDT, and the 12 prompts it ships with all have fixed schemas and explicit uncertainty fields.