A job gets a few retries, then a dead letter row
A worker that retries forever keeps a job that can never succeed at the head of the queue. Count the attempts on the row and park the job when it runs out.
UPDATE jobs SET
attempts = attempts + 1,
last_error = $2,
status = CASE WHEN attempts + 1 >= 5 THEN 'dead' ELSE 'pending' END,
run_at = now() + interval '1 second' * power(2, attempts)
WHERE id = $1;A permanent failure, a 422 or a constraint violation, goes straight to dead without burning the five. Alert on how many dead rows there are, because a table nobody reads is a slower way of dropping the job.
queuesreliability