Skip to content
Tyrone Software
PHP Coding

Laravel jobs that run twice: retry_after versus timeout

Abstract illustration for the article Laravel jobs that run twice: retry_after versus timeout

A payment confirmation email goes out twice, and only on the days when the mail provider is slow. The job is not buggy. The queue configuration hands it to a second worker before the first has finished.

Two clocks

Laravel's queue has two related settings, and the relationship between them is the whole story:

  • retry_after (in config/queue.php, per connection) is how long a reserved job may stay reserved before the queue assumes the worker died and makes the job available again.
  • The job timeout (the --timeout option of queue:work, or a $timeout property on the job class) is how long a worker lets one job run before it kills it.

If a job can run longer than retry_after, the queue releases it while it is still running, and another worker picks it up. The Laravel documentation states that the timeout should always be at least a few seconds shorter than retry_after so that a job is killed before it would be retried.

Timeline of a queue job that runs longer than retry_after and is started again by a second worker
Setting the job timeout below retry_after kills the first run before the second can start.

How to set them

// config/queue.php
'redis' => [
    'driver' => 'redis',
    'retry_after' => 120,
],
php artisan queue:work --timeout=90 --tries=3

Here the worker stops a job after 90 seconds and the queue would only release it after 120. Choose retry_after from the slowest legitimate job, and put long jobs on their own queue with their own worker, timeout and matching retry_after. The value belongs to the connection, so jobs with very different lengths on one connection force the largest number on all of them.

Timeouts need the pcntl PHP extension, which the worker uses to interrupt a job. Without it, --timeout cannot be enforced, and a stuck job blocks a worker for good.

Design for repetition anyway

Queues give at-least-once delivery. Even with perfect settings, a worker that crashes after doing the work and before acknowledging it produces a duplicate. Make jobs idempotent:

  • Record a completed marker (an ID or a unique key) in the same database transaction as the effect, and skip if it is already there.
  • Pass the operation ID to external providers that support idempotency keys.
  • Use ShouldBeUnique for jobs where only one instance should exist at a time, remembering that the lock has its own expiry.
  • Dispatch inside the transaction only with afterCommit, so a worker never runs a job for data that is not committed yet.

Operational details

  • Workers are long-running processes that load your code once. After a deploy, run php artisan queue:restart, or workers keep running the old code.
  • Supervisor's stopwaitsecs should be greater than your longest job, otherwise a restart kills jobs mid-way.
  • Configure tries, backoff and a failed() method or the failed jobs table, and alert on it. A silent failure table is a data loss mechanism.
  • Watch queue length and the age of the oldest job. A queue that is short but stuck looks fine in the length graph.

SQS has its own version of the same setting

If the queue driver is Amazon SQS rather than Redis or a database, the retry_after value in config/queue.php is not the number that controls this behaviour at all: SQS manages redelivery itself through a per-queue setting called the visibility timeout, configured on the queue in AWS, not in Laravel's own configuration file. A message becomes visible to other consumers again once the visibility timeout expires, regardless of what retry_after says, so the two numbers have to be kept in sync by hand: set the SQS queue's visibility timeout to comfortably exceed your job's actual timeout, the same relationship described above between retry_after and --timeout, just configured in a different place and easy to miss because Laravel's own config file looks like the whole story.

Keep reading

Technologies

  • Python
  • PHP
  • Laravel
  • Django
  • PostgreSQL
  • MySQL
  • HTML5
  • CSS3
  • Bootstrap