A Series of Unfortunate Jobs

A developer documented two common Laravel queue pitfalls that can silently break production jobs. The first involves retryUntil(), whose expiration timestamp is evaluated when a job is dispatched rather than when it begins processing, causing delayed jobs to fail immediately with MaxAttemptsExceededException. The second is that Laravel Horizon's actual default for tries is zero, meaning unlimited retries, contrary to the documented single-attempt behavior unless tries is explicitly set to 1.

You can't tell me Laravel queues haven't bitten you at least once. It's great that we get them out of the box, but man, they can be confusing as hell sometimes. Over the years, I've screwed up, more than once, and collected quite a few lessons along the way. Claude is currently Pondering.. , I'm waiting, and bored, so shall we? I want you to look at the code below and tell me if you can spot anything wrong: php <?php namespace App\Jobs; use DateTime; use Illuminate\Foundation\Queue\Queueable; use Illuminate\Contracts\Queue\ShouldQueue; class SendAbandonedCartReminder implements ShouldQueue { use Queueable; public function retryUntil : DateTime { return now - addMinutes 10 ; } public function handle : void { // business logic } } SendAbandonedCartReminder::dispatch $cart - delay now - addDay ; Nothing sus, right? The customer left their cart, so we nudge them a day later. And if the mail provider is having a bad day, we keep retrying for 10 minutes, then give up. Except that reminder will never be sent. Not once. If you spotted it, you probably learned this one during a fun debugging session 😅. If not, the misconception here is assuming that retryUntil starts counting from the moment the job starts processing. Well, kind sir, that's where you're wrong. retryUntil is evaluated when the job is pushed onto the queue, and the expiration timestamp is baked into the job payload. By the time the job becomes available to the workers, a day later, that timestamp is long gone. The worker fails it with the one and only MaxAttemptsExceededException , and handle never runs. Same drill, take a look at the code below: php <?php namespace App\Jobs; use Illuminate\Foundation\Queue\Queueable; use Illuminate\Contracts\Queue\ShouldQueue; class SyncOrderToCrm implements ShouldQueue { use Queueable; public function handle : void { // business logic } } There is no $tries in sight, sooo, what happens if the job fails? Good question, that is one good question. For as long as I can remember, I've used the database driver locally. It's just convenient right? Production, however, is a different beast. That's Horizon territory, and it only makes sense. Now, Horizon ships with a supervisor called supervisor-1 , and for the love of God, it's ugly. I like giving supervisors proper names, like what do you mean supervisor-1 ? So, naturally, I defined a data-import-supervisor , and the config looked like this: js 'defaults' = , // the ugly supervisor-1 gone 'environments' = 'production' = 'data-import-supervisor' = // no 'tries' = 1 'connection' = 'redis', 'queue' = 'data-import' , 'balance' = 'auto', 'maxProcesses' = 10, 'timeout' = 1800, , , , Now, I knew for a fact that when $tries isn't specified, the job is attempted once. I'd lived by that for years, never passing a single argument to queue:work locally with the DB driver, and every failed job being attempted exactly once. So, in my head, Horizon would behave the same. I mean, it wraps queue:work , so it only made sense to omit what was obvious. I hate bloat. And to be fair, it wasn't just me. The docs had my back on this one: If you don't set the tries option, Horizon defaults to a single attempt , unless the job class defines $tries , which takes precedence over the Horizon configuration. What I didn't freaking know is that this only holds if tries is explicitly set to 1 somewhere. Horizon's actual default is zero https://github.com/laravel/horizon/blob/5.x/src/Console/SupervisorCommand.php L36 , and zero means unlimited. So, back to the question. What happens when the job fails? queue:work , the job is attempted once and marked as failed. Pretty much what you'd expect. tries in it, the job is retried indefinitely. Becuase why would you? I know, uncalled for. But hey, now you know: always pin your $tries . Unique jobs. We all have at least one in our app, right? php <?php namespace App\Jobs; use Illuminate\Foundation\Queue\Queueable; use Illuminate\Contracts\Queue\ShouldQueue; use Illuminate\Contracts\Queue\ShouldBeUnique; class SyncOrderToCrm implements ShouldQueue, ShouldBeUnique { use Queueable; public function handle : void { // business logic } } The way it works is simple: the job gets pushed, it acquires a lock, a worker processes it, the lock gets released, and la vie en rose. My question is: what happens to the lock if the job never gets to release it? Well, first, how do we even end up there? There are multiple ways, one of which you've already met. Remember retryUntil from Gotcha 1? It's back. Take the same job, make it ShouldBeUniqueUntilProcessing , and give it an hour to live: php <?php namespace App\Jobs; use DateTime; use Illuminate\Foundation\Queue\Queueable; use Illuminate\Contracts\Queue\ShouldQueue; use Illuminate\Contracts\Queue\ShouldBeUniqueUntilProcessing; class SyncOrderToCrm implements ShouldQueue, ShouldBeUniqueUntilProcessing { use Queueable; public function retryUntil : DateTime { return now - addHour ; } public function handle : void { // business logic } } Now dispatch it on a bad day. The queue is drowning, and the job sits there for two hours. A worker finally picks it up, sees that retryUntil is in the past, and fails it without ever running it. Fair enough, that's what we asked for. Except, with ShouldBeUniqueUntilProcessing , the lock is released the moment processing starts, so the failure path assumes it's long gone and doesn't touch it. But processing never started. The lock is still there, and from now on, every dispatch of that job is silently skipped. So, how long till it expires? Well yeah, technically we'll all be dead by 2286, so call it never. Still, 3 drivers, 3 different answers. Database https://github.com/laravel/framework/blob/v13.30.1/src/Illuminate/Cache/DatabaseLock.php L122-L129 , Redis https://github.com/laravel/framework/blob/v13.30.1/src/Illuminate/Cache/RedisLock.php L34-L41 , File https://github.com/laravel/framework/blob/v13.30.1/src/Illuminate/Cache/FileStore.php L480-L485 . But hey, now you know: always set an explicit uniqueFor . No code this time, more of a situation. Paraphrased from a Laracasts thread https://laracasts.com/discuss/channels/laravel/appjobstestjob-has-been-attempted-too-many-times : I'm setting up a job on Laravel 11 with Horizon, default configuration, and $timeout = 900 on the job. I keep getting App\Jobs\TestJob has been attempted too many times . The thing is, the job doesn't fail and it isn't retried. I added some logging, and after the error is raised I still see TestJob completed successfully . The job takes 2 or 3 minutes at most. I've run out of ideas. Any idea what might be causing this? Yes, it's retry after . It's the number of seconds a job can stay reserved before the queue assumes it died. Once that time passes, the job goes back onto the queue for another worker to pick up. The default is 90 seconds. So while the first worker was still busy processing the job, a second worker picked it up, saw that it had already used its single attempt, and threw the exception without ever running it. Meanwhile, the first worker finished and logged a success. Pretty fun to debug right? What's worse is that, had $tries been higher, the second worker would have actually run it. You'd then have your job processed twice, so if your job isn't idempotent, you're in for one hell of a debugging session. So, kids, what did we learn today? That's right, your retry after should always be higher than your job's timeout . Technically, this one isn't a queue issue, but it does bite you when working with queues. So yea might as well include it. Imagine you're building an API integration where you're allowed to make 10 requests per minute. The API requests are queued, obviously, and the first thought is to use Laravel's RateLimited job middleware to enforce that rule. So you do exactly that. js RateLimiter::for 'crm', fn = Limit::perMinute 10 ; php <?php namespace App\Jobs; use Illuminate\Foundation\Queue\Queueable; use Illuminate\Contracts\Queue\ShouldQueue; use Illuminate\Queue\Middleware\RateLimited; class SyncOrderToCrm implements ShouldQueue { use Queueable; public function middleware : array { return new RateLimited 'crm' ; } public function handle : void { // business logic } } Somehow, God knows why, you still get spammed with 429 errors, what?? After a while of caveman debugging, you realize that Laravel's rate limiter is a fixed-window counter: it counts hits for 60 seconds, then resets to zero and forgets the previous minute ever happened. Now say one request lands at 12:00:00 and starts the window. Nothing else happens until 12:00:59, when 9 requests land at once. All allowed, the counter is at 10. At 12:01:00 the counter resets, and the next 10 sail through. The provider, however, doesn't care about your windows. It sees 19 requests in a minute and hands you a sweet 429 . But hey, next time you're building an API integration, you'll know. Hopefully not the hard way. This one is less painful than the previous ones, but depending on your codebase, it could be something you have right now and have never noticed. Every job in this article used one trait, Queueable . Inspect it, and that one trait is really four: trait Queueable { use Dispatchable, InteractsWithQueue, QueueableByBus, SerializesModels; } Of these, SerializesModels is probably the coolest. Its sole job is to "represent" models in a way that's efficient and can be reconstructed later. Otherwise, we could end up with a huge payload that some drivers, like SQS, would flat out reject. Anyway, somewhere in your code, you have a model with a couple of relationships loaded: php $order = Order::query - with 'items' = fn Builder $query = $query- latest - limit 3 , 'payments' = fn Builder $query = $query- where 'status', 'captured' , - find $id ; Then another piece of code dispatches a job with that model: SyncOrderToCrm::dispatch $order ; If you inspect the payload sitting in your queue, you'll find something like: O:23:"App\Jobs\SyncOrderToCrm":1:{s:5:"order";O:45:"Illuminate\Contracts\Database\ModelIdentifier":5:{s:5:"class";s:16:"App\Models\Order";s:2:"id";i:5;s:9:"relations";a:2:{i:0;s:5:"items";i:1;s:8:"payments";}s:10:"connection";s:5:"mysql";s:15:"collectionClass";N;}} Now, here's the thing. Do you see any information beyond the class, the ID, and the names of the loaded relationships? No. Not the limit 3 , not the where 'status', 'captured' , nothing. See where I'm going with this? So when the worker unserializes the payload to reconstruct the model, all it can do is fetch the order by ID and load every relationship on that list. These are the exact queries it runs: select from orders where orders . id = ? limit 1 select from items where items . order id in 5 select from payments where payments . order id in 5 Now, if your job depends on those loaded relationships please don't , you could end up with some terrible edge cases. Take this handle : php public function handle Crm $crm : void { $crm- updateOrder $this- order- id, 'paid total' = $this- order- payments- sum 'amount' , 'recent items' = $this- order- items- pluck 'sku' , ; } You filtered payments down to captured ones before dispatching, so paid total looks right locally. On the worker, payments is every payment on that order, including failed attempts and refunds, and the CRM now says the customer paid more than they actually did. Same story for items . You loaded the latest three, but the CRM gets all of them. And even if you don't depend on them, if one of those relationships happens to have eighty thousand rows, the worker will hydrate eighty thousand models, hit its memory limit, and die. So yes, keep in mind how SerializesModels works. In most cases, if not all, if for some reason you have to pass the model to a job instead of its identifier, strip its relationships with the WithoutRelations attribute and load exactly what the job needs inside handle . But hey, now you know. Looks like my Claude instance finished cogitating, so that'll do. If all of this taught me one thing, it's to be explicit with my jobs. The defaults are well thought out, sure, but I'd rather read a $timeout than assume one. Explicit is predictable, and predictable means less guesswork about what's going on when things break.