Queues in production — idempotency, retries, and failed_jobs
The previous chapter moved mail sending outside the request. It works, but it is not something you can leave unattended.
Nobody notices when the worker dies. Failed jobs are merely recorded, and nobody looks at them. And the same job can run twice. This chapter handles all three.
Jobs run twice
There is a premise you have to accept when using a queue.
A job is not necessarily executed "exactly once." It runs at least once, but it can run two or more times.
Here is one example of why.
- The worker takes a job off the queue
- It sends the mail (success)
- Just before marking the job complete, the worker process dies
- After a while, that job is taken off the queue again as unprocessed
- Another copy of the mail is sent
The processing itself succeeded, but it died before recording completion. There is always a gap between "sent it" and "recorded that it was sent." A server restart, a deployment, running out of memory — any of them can land in that gap.
It is the same structure as "another request slips in between the check and the update" from chapter 14. There we closed it with a database constraint. For jobs, you write them so the result is the same when they run twice.
Writing idempotently
Chapter 14 defined idempotency as "however many times you send the same request, there is one result." The same applies to jobs.
Take the payment webhook from chapter 13. Run "record the payment" twice and it gets recorded twice.
Create the job with php artisan make:job RecordPayment. It is dispatched from the controller built in chapter 13. We wire it up once the job is finished, so for now just settle its arguments. $eventId is the identifier the payment provider assigns to each notification; a resend carries the same value.
class RecordPayment implements ShouldQueue
{
use Queueable;
public function __construct(
public readonly int $orderId,
public readonly string $eventId,
public readonly int $amount,
) {}
public function handle(): void
{
// ↓ this is what we design
}
}
Written straightforwardly, it looks like this.
public function handle(): void
{
Payment::create([
'order_id' => $this->orderId,
'provider_event_id' => $this->eventId,
'amount' => $this->amount,
]);
}
We close it with the same tool as chapter 14: a unique constraint on the event ID the payment provider assigns.
Let us add the columns this chapter needs, all together.
public function up(): void
{
Schema::create('payments', function (Blueprint $table) {
$table->id();
$table->foreignId('order_id')->constrained();
$table->string('provider_event_id')->unique();
$table->integer('amount');
$table->timestamps();
});
}
public function up(): void
{
Schema::table('orders', function (Blueprint $table) {
$table->timestamp('confirmation_sent_at')->nullable();
$table->string('shipment_status')->nullable();
});
}
The unique constraint on provider_event_id is the star of this section.
Do not forget to fix #[Fillable] on the models too. Chapter 14 said "creating the migration and forgetting the attribute is an easy shape to fall into." This is exactly that moment.
#[Fillable(['order_id', 'provider_event_id', 'amount'])]
class Payment extends Model
{
public function order(): BelongsTo
{
return $this->belongsTo(Order::class);
}
}
#[Fillable([
'user_id', 'customer_email', 'status', 'total_amount',
'idempotency_key', 'confirmation_sent_at', 'shipment_status',
])]
Miss the two columns on Order and this chapter's code breaks silently. As chapter 3 covered, an attribute not in #[Fillable] is silently discarded by update(). Nothing errors. The "record that it was sent" written next stays null every time, and the confirmation email is resent on every retry.
public function handle(): void
{
$payment = Payment::firstOrCreate(
['provider_event_id' => $this->eventId],
['order_id' => $this->orderId, 'amount' => $this->amount],
);
if (! $payment->wasRecentlyCreated) {
return;
}
NotifyWarehouse::dispatch($payment->order);
}
Put the first-time-only processing below the early return. Here it queues the shipping instruction job built in the previous chapter.
That line is the reason for checking wasRecentlyCreated. Keeping the record to one row is no use if the shipping instruction gets queued twice.
RequestShipment from the previous chapterIn the previous chapter, the RequestShipment listener queued NotifyWarehouse as soon as an order was confirmed. That means the shipping instruction went out before payment was confirmed. It was a simplification to demonstrate queue syntax; as a business flow it is too early.
Move the shipping instruction dispatch into this job and delete app/Listeners/RequestShipment.php. Leave it in place and the warehouse gets two instructions: one at order time and one at payment time.
Fix tests/Feature/WarehouseNotificationTest.php from the previous chapter as well. That test pinned down "placing an order queues the warehouse notification job," and that behavior no longer exists. As when authentication arrived in chapter 11, move it to the new expectation rather than deleting it. Its new home is this chapter's test, which runs RecordPayment twice and checks Queue::assertPushed(NotifyWarehouse::class, 1).
If having "record the payment" and "instruct shipping" in one job bothers you, you can dispatch a PaymentRecorded event and receive it in a listener. It is the same judgment as the split covered in chapter 15.
firstOrCreate returns the matching row if there is one and creates it otherwise. wasRecentlyCreated tells you whether you just created it or it already existed. It is the same attribute used in chapter 14's controller to split the status code.
Always add the unique constraint. firstOrCreate is two steps — look, then create if absent — so with two workers running at once both decide "absent." It is the same gap as chapter 14. The database constraint is the last line of defense.
Wiring up the receiver
With the job in place, dispatch it from chapter 13's controller. The notification the payment service sends looks like this.
{
"id": "evt_1a2b3c",
"event": "payment.succeeded",
"order_id": 12,
"amount": 5400
}
id is the per-notification identifier and does not change on a resend. That becomes $eventId.
use App\Jobs\RecordPayment;
public function handle(Request $request)
{
if ($request->input('event') !== 'payment.succeeded') {
return response()->json(['status' => 'ignored']);
}
RecordPayment::dispatch(
$request->integer('order_id'),
(string) $request->input('id'),
$request->integer('amount'),
);
return response()->json(['status' => 'accepted']);
}
The 200 is not because the recording finished. It goes as far as receiving it and queueing it. There is no reason to keep the payment service waiting, so this is the correct response. The mechanism for noticing a processing failure is this chapter's failed_jobs.
Replace the happy path of tests/Feature/PaymentWebhookTest.php from chapter 13 with this form.
use App\Jobs\RecordPayment;
use Illuminate\Support\Facades\Queue;
test('a webhook with a valid signature queues the job', function () {
Queue::fake();
config(['services.payment.webhook_secret' => 'test-secret']);
$payload = [
'id' => 'evt_1a2b3c',
'event' => 'payment.succeeded',
'order_id' => 12,
'amount' => 5400,
];
$body = json_encode($payload);
$signature = hash_hmac('sha256', $body, 'test-secret');
$this->call(
'POST',
'/api/v1/webhooks/payment',
content: $body,
server: [
'CONTENT_TYPE' => 'application/json',
'HTTP_X_PAYMENT_SIGNATURE' => $signature,
],
)->assertStatus(200);
Queue::assertPushed(RecordPayment::class);
});
Queue::fake() is required. The test environment's QUEUE_CONNECTION is sync, so without faking, the job runs on the spot the moment it is dispatched. No order row exists for order_id, so it fails on the payments foreign key constraint. What this test wants to confirm is only that the signature got through and the job was queued. The job's body is confirmed separately in the test section below.
Not everything can be made idempotent
Some processing, like sending mail, cannot be undone once it leaves. Recording on your side that you sent it lets you prevent the second one.
Add that record to the listener written in chapter 15. The previous chapter added ShouldQueue, so this listener now runs through a queue.
public function handle(OrderPlaced $event): void
{
$order = $event->order;
if ($order->confirmation_sent_at !== null) {
return;
}
$order->user->notify(new OrderConfirmed($order));
$order->update(['confirmation_sent_at' => now()]);
}
This is still not complete. Die right after the notification and just before the record, and a second copy goes out. It only narrows the gap.
How far you go depends on the damage from a duplicate. Two confirmation emails is a different weight from two payment runs. For irreversible things like transfers or charges, use the external service's idempotency key (the same idea as the Idempotency-Key from chapter 14).
Designing retries
By default a failed job is tried a few times before giving up. The right count and interval depend on why it failed.
In Laravel 13 you write them as attributes1.
use Illuminate\Queue\Attributes\Backoff;
use Illuminate\Queue\Attributes\Tries;
#[Tries(5)]
#[Backoff([10, 30, 120])]
class NotifyWarehouse implements ShouldQueue
{
// ...
}
Tries is the total number of attempts and Backoff is how many seconds to wait after a failure. Pass an array and the first wait is 10 seconds, the second 30, and the third onward 120.
The same settings can be written as properties such as public $tries = 5;. Define a tries() method and it takes precedence. This series uses attributes, matching the form the Laravel 13 documentation shows, just like #[Fillable] in chapter 3.
The --tries=3 at worker startup is the default when the job specifies nothing. If the job has an attribute or property, that wins. When it looks like "I set the worker to 3 but it tried 5 times," suspect the job's own setting.
Why put a gap between attempts
When the warehouse system is down, trying 5 times with no gap fails 5 times. You gave it no time to recover.
And if you have a lot of these jobs, you flatten a partly recovered system with requests again. Spacing attempts out is also for their sake.
Decide the count and interval by why it failed.
| Reason for failure | The right shape |
|---|---|
| A transient outage or a network blip on their side | Try a few times, with gaps |
| The request content is invalid (4xx) | Give up after one. It will never go through |
| They are under load (429) | Wait exactly as long as they asked |
Trying every failure the same number of times is sloppy. Trying an invalid job 5 times just fails 5 times while clogging the queue.
Bounding by deadline
You can bound by time rather than count1.
public function retryUntil(): DateTime
{
return now()->plus(minutes: 30);
}
It tries any number of times for 30 minutes, then gives up. It suits "keep trying until it recovers, but not forever."
Bounding by exception count
With a large Tries, you can also give up after a certain number of exceptions1.
#[Tries(25)]
#[MaxExceptions(3)]
For processing that hits an external service's rate limit and waits again, you need many attempts, but you want to stop after 3 real errors. Attempts and failures can be counted separately.
Handling failed jobs
As the previous chapter mentioned, a job that exhausts its attempts is recorded in the failed_jobs table.
php artisan queue:failed # list them
php artisan queue:retry <uuid> # requeue a specific one
php artisan queue:retry all # requeue everything
php artisan queue:flush # delete the records
Being able to re-run is the payoff for using a queue. Back when it was synchronous, a failed mail send was simply the end of it.
Build something that notices failures
Being recorded is the same as not existing if nobody looks. As the end of the previous chapter said, this is the dividing line for production operation.
You can register something to be called on failure1.
use Illuminate\Queue\Events\JobFailed;
use Illuminate\Support\Facades\Queue;
public function boot(): void
{
Queue::failing(function (JobFailed $event) {
Log::error('A job failed', [
'connection' => $event->connectionName,
'job' => $event->job->resolveName(),
'exception' => $event->exception,
]);
});
}
Write it in boot, not register. The reason covered in chapter 2 applies here too.
Logging alone still requires someone to go and look. Push it to chat, or send it to a monitoring service — make it come to you. Chapter 21 covers logging and monitoring together.
Clean up on the job side
Define a failed() method on the job class and it is called when giving up is decided.
public function failed(?Throwable $exception): void
{
$this->order->update(['shipment_status' => 'failed']);
}
Record the state as "failed" and you can pick it up later. Record nothing and the order sits at "processing." To the user it looks like an order that never ships.
Not letting the same job overlap
Separate from running twice, there is the problem of the same job running concurrently.
Say a stock aggregation runs every 5 minutes. If it takes 7 minutes, the next run starts before the previous finishes. Two of them rewrite the same data at once.
There are two tools.
ShouldBeUnique — do not queue it
Implement ShouldBeUnique and the same job is not queued again while one is already on the queue1.
use Illuminate\Contracts\Queue\ShouldBeUnique;
class AggregateStock implements ShouldQueue, ShouldBeUnique
{
// ...
}
It rejects on the queueing side, so the queue does not fill up with the same job.
WithoutOverlapping — do not run them at once
WithoutOverlapping is middleware that allows queueing but prevents concurrent execution1.
use Illuminate\Queue\Middleware\WithoutOverlapping;
public function middleware(): array
{
return [new WithoutOverlapping($this->order->id)];
}
Jobs sharing the argument key wait until the previous one finishes. Use it when you want serialization per order or per customer. Orders A and B are processed in parallel, while two jobs for order A queue up behind each other.
These two require a cache driver that supports atomic locks. The supported ones are memcached, redis, dynamodb, database, file, and array1.
The case needing care is running the application on several servers. The file driver looks at a different file on each server, so server A and server B each queue one copy of the same job. The lock means nothing.
With several servers you need a driver every server looks at in common, such as Redis. Define a uniqueVia() method and you can point just the lock at a different driver1.
Keeping workers alive
queue:work is just a process. Close the terminal and it stops. It stops on a server restart and on an abnormal exit from an exception.
Left stopped, jobs pile up. You need something to watch the process and restart it when it dies.
On Linux, Supervisor is the common choice. Its configuration is a declaration of "keep this command running."
[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/app/artisan queue:work --sleep=3 --tries=3 --max-time=3600
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
user=www-data
numprocs=2
redirect_stderr=true
stdout_logfile=/var/www/app/storage/logs/worker.log
stopwaitsecs=3600
process_name is mandatory if numprocs is 2 or more. Each process needs a distinct name, and without %(process_num) Supervisor cannot load the configuration.
numprocs sets how many run at once. More is faster, but it also increases the number of database connections.
Specify user. Without it the worker runs as root, and the log and cache files it creates end up owned by a different user than the web server's.
stopwaitsecs is generous. It is how long Supervisor waits for an in-flight job to finish when stopping the worker. Too short and processing is killed partway.
--max-time bounds the worker's own lifetime. Running for a long time creeps memory upward, so it exits periodically on its own and Supervisor starts it again.
When running in containers, the container orchestration takes this role rather than Supervisor. Define the worker as a separate container and configure it to restart on exit.
What matters is not Supervisor itself but that something is responsible for watching and restarting.
Restart on every deployment
Here we collect on the problem mentioned in the previous chapter. A worker loads the code at startup and keeps running.
Deploy and place new code and the running worker keeps processing with the old code. You fixed a bug, and only the queue-driven processing stays broken.
php artisan queue:restart
Run this at the end of a deployment. It does not interrupt jobs in flight. It signals "finish what you are processing, then exit." Supervisor then starts a new process, and that one loads the new code.
Forget this one line and the deployment changes nothing. Build it into your deployment procedure. Chapter 21 covers it.
Tests
The previous chapter used Queue::fake() to confirm dispatch. This chapter's concern is the execution side, so we call jobs directly.
Running twice still leaves one row
Idempotency is confirmed by executing the same job twice.
use App\Jobs\NotifyWarehouse;
use App\Jobs\RecordPayment;
use Illuminate\Foundation\Testing\RefreshDatabase;
use Illuminate\Support\Facades\Queue;
pest()->use(RefreshDatabase::class);
test('running the job twice with the same event ID records one payment', function () {
Queue::fake();
$order = Order::factory()->create();
$job = new RecordPayment($order->id, 'evt_12345', 3600);
$job->handle();
$job->handle();
$this->assertDatabaseCount('payments', 1);
Queue::assertPushed(NotifyWarehouse::class, 1);
});
It just calls handle() twice. The same situation as a real re-run, reproduced in a single process. Unlike chapter 14's concurrency, this one tests straightforwardly.
Two things are checked: that there is one payment, and that the shipping instruction was queued once.
There is a reason the second one is needed. With only the first, removing the wasRecentlyCreated early return entirely leaves the test green. firstOrCreate returns the existing row on the second call, so payments stays at one. "Not recorded twice" and "first-time-only processing does not run twice" are different things. The real harm is in the latter, and pinning it down is what matters.
Revert firstOrCreate to a plain create and the first goes red; remove the early return and the second goes red.
State remains after a failure
test('a failed job records the failure on the order', function () {
$order = Order::factory()->create();
(new NotifyWarehouse($order))->failed(new RuntimeException('The warehouse API is not responding'));
expect($order->fresh()->shipment_status)->toBe('failed');
});
failed() is an ordinary method too, so you can call it directly. What remains after a failure is as worth confirming as what happens on success.
Confirming that #[Tries(5)] takes effect is not worth the trouble in a test. You would have to run a real worker and cause five failures.
An attribute is configuration, not logic. If you want to confirm it, run a worker once and watch it land in failed_jobs. Reserve tests for things like idempotency, which are hard to notice when broken.
What matters in production
Horizon is Redis-only
Laravel has Horizon, a tool that shows queue state on a screen. It only works when your queue runs on Redis2. It does not work with the database driver, and it does not support Redis Cluster.
It is worth considering alongside a move to Redis once you are at scale. Queue depth, processing times, and failed jobs become visible on a screen.
Keep the timeout shorter than retry_after
A worker has an upper bound on the time one job may take, and the queue connection setting has a value (retry_after) meaning "a job past this many seconds is presumed dead and requeued."
When the latter is shorter than the former, a job still running gets requeued. Two copies of the same job run at once. connections.database.retry_after in config/queue.php defaults to 90, so if you have jobs exceeding 90 seconds, revisit that value or the job's timeout. It is the most avoidable cause of the "runs twice" from the top of this chapter.
Do not make one job too big
The previous chapter mentioned this too, but it bites harder in operation. A big job loses more when it fails.
Process 10,000 records in one job and die at record 9,000, and the re-run starts from record 1. Written idempotently the result is still correct, but it takes twice as long. Split into 100-record chunks and you redo only what you lost.
Separate the queues
Send every job to one queue and heavy jobs make light ones wait. While 10 PDF generations are backed up, no confirmation emails go out.
Split destinations with the previous chapter's Queue::route() and split workers per queue as well. Do not mix what a user is waiting on with what they are not.
Check the production configuration
Repeating the previous chapter, but this is the most common accident. With QUEUE_CONNECTION still sync, none of the asynchronous machinery is running. Nothing errors. It is just slow.
Summary
- A job can run twice. There is always a gap between "processed it" and "recorded that it was processed"
- Write idempotently. Unique constraint +
firstOrCreate+wasRecentlyCreatedis the basic form - Irreversible processing cannot be fully protected. Narrow the gap, then judge by the damage from a duplicate
- Retries go in
#[Tries]and#[Backoff]. Spacing attempts out is also for the other side's sake - Do not retry a failure caused by invalid content. Decide the count by why it failed
- Failed jobs stay in
failed_jobsand can be requeued withqueue:retry - Do not just record — build something that notices. Push it to a notification with
Queue::failing - Record state with the
failed()method. Do nothing and it sits at "processing" ShouldBeUniqueprevents queueing;WithoutOverlappingprevents concurrent execution- Both need a cache with atomic locks. Several servers require a shared driver
- Keep workers alive with Supervisor or the equivalent. Run
queue:restartat the end of a deployment
What to read next
Queues are now operable. So far it has all been about deferring slow processing; the next chapter, Caching — how much correctness you trade for speed, covers not doing the processing at all. If you return the same answer to the same question every time, remembering it is enough. We design what may be cached, when to throw it away, and what you give up in exchange for speed.
Exercises
The following job causes a problem when it runs twice. How do you fix it?
public function handle(): void
{
$this->order->update(['status' => OrderStatus::Confirmed]);
$this->order->user->increment('loyalty_points', 100);
}
Sample answer
200 points get added.
The status update is fine. Writing Confirmed twice leaves it Confirmed. An operation that writes the same value gives the same result however many times you run it.
The problem is increment. It adds to the current value, so it grows by however many times it ran. Two runs make it 200.
There are two ways to fix it.
One is to make the state change a condition — process only when it is not yet Confirmed.
if ($this->order->status === OrderStatus::Confirmed) {
return;
}
That alone still lets both through when two workers run at once. It is the same gap as chapter 14. Lock the row, or serialize per order with WithoutOverlapping.
The other is to record the addition separately. Record "points awarded for this order" in a table with a unique constraint and do nothing if it already exists. The addition itself cannot be made idempotent, so make the decision of whether to add idempotent instead.
Which you choose depends on whether you need a history of point awards. If you do, the latter is natural.
How do you respond to "failed jobs can be restored with queue:retry, so one attempt is enough"?
Partly right, but it shifts the operational burden onto people.
It is true that nothing is lost, because queue:retry exists. The problem is that nothing progresses until someone notices and types a command.
Push failures that a retry a few seconds later would clear — a network blip — onto people, and things stall overnight and at weekends. Confirmation emails do not go out until the next morning. Retry automatically and it resolves before anyone notices.
On the other hand, raising the count is not the answer either. When the request content is invalid, no number of attempts gets it through. Attempts occupy the queue and make other jobs wait.
The response is to propose separating the kinds of failure. Transient outages: automatically, a few times, with gaps. Permanent failures: give up after one and land in failed_jobs. And on top of that, build something that notifies you when a job lands in failed_jobs.
"Can be restored" and "someone will restore it" are different things. Keep queue:retry as the last resort while reducing the number of jobs that reach it — that is what retry design is.