Running Two Instances Means Every 'Once' Becomes 'Twice'

I maintain a Laravel app that used to run everything, nginx, PHP-FPM, the queue workers, the scheduler, in a single container on DigitalOcean’s App Platform. After running for a while, that container would quietly stop processing new jobs while the process itself kept running. Chasing that turned into most of a year, and along the way it turned out to be at least three different problems wearing the same symptom.

First guess: something in PHP-FPM is leaking

The first theory was that long-lived worker processes were accumulating state they shouldn’t, so we recycled them: kill a PHP-FPM worker after 200 requests, kill idle workers outright so they’d stop holding database connections open. That helped, somewhat.

It also introduced a new failure that hadn’t existed before: a worker due for recycling doesn’t know or care whether it’s in the middle of a long job when its number comes up. Short jobs never noticed. A job that ran long enough to still be going when its worker hit the recycle threshold got cut off mid-execution, sometimes with no error at all, just a job that silently never finished.

We swapped one problem for another: now individual long jobs sometimes just vanished mid-run instead. We increased how long supervisor waits before force-killing a process, more than once, to give long jobs time to finish before a recycle or a deploy cut them off.

A detour that went nowhere

Around the same period we tried more PHP-FPM pool tuning, more worker processes, fewer child processes per worker, and separately tried running the PHP-FPM master under a different user. That last one got reverted almost immediately. Some of what we tried here led nowhere, and was still worth ruling out.

First attempt at isolating the worker

Later on, we gave the queue workers and the scheduler their own container, separate from the API, so a slow worker restart couldn’t affect anything serving requests. We reverted it not long after. The scheduler had been running through the OS’s own cron, firing artisan schedule:run every minute in that separate container, and something about that setup wasn’t holding up, so we merged everything back into one container and switched the scheduler from cron to Laravel’s own long-running schedule:work loop instead. First split, undone.

The bug that had nothing to do with any of this

Deep into the staleness hunt, while doing another pass on worker reliability, we found this sitting in the queue config, untouched since before any of the above started:

// before
'retry_after' => 90,

// after
'retry_after' => 900,

The workers ran with --timeout=300, five minutes before Laravel itself would kill a job. retry_after is a different number: how long the queue driver waits before deciding a job must have died and releasing it back onto the queue for someone else to pick up. It was set to 90 seconds. Any job that legitimately ran past a minute and a half, still well inside its real five-minute budget, looked abandoned, got requeued, and got picked up by another worker while the first one was still quietly finishing it. Two workers, same job, same data, at the same time.

Splitting the container again, properly this time

Around the same point, we split the worker and the API into their own containers again, this time for good: an API-only config, and a worker config running three worker processes side by side plus one dedicated scheduler process. Three worker processes sharing a queue is fine, Laravel reserves a job atomically, so two of them can never grab the same row. The scheduler doesn’t get that guarantee, and the split made it possible for the worker config to run as more than one instance with nothing stopping it.

The duplicate that followed

Later, a weekly reminder job started sending the same reminder twice. The worker container had ended up running as more than one instance, and two copies of schedule:work, one per instance, don’t know about each other. The fix was:

$schedule->job(new ReminderJob)
    ->weekly()
    ->onOneServer()
    ->withoutOverlapping();

onOneServer() is a lock in whatever cache the containers already share, so only one instance actually runs the job even though every instance still evaluates the schedule. It held.

The instance nobody was counting

Jobs weren’t just running twice around this time, some ran three or four times, and the logs made it look like one instance misbehaving, because a staging environment had been set up with the same environment variables as production, including the live credentials, and pointed at the same log channel. What gave it away was the instance ID on each log line: two different IDs kept showing up where I expected one, pointing at a second instance with a real copy of the customer database and a real way to reach it.

That second instance stacked on top of production’s own problem: the same worker container running as more than one instance that caused the reminder job to duplicate was still happening, just not yet caught with a lock everywhere it needed one. Production duplicating a job on its own, plus an uncounted second instance running that same job against the same real data, is how something ends up happening four times instead of two.

If there’s a single lesson across all of this, it’s to architect the deployment before you build it, decide up front how many instances of each process can run and what’s shared between environments that shouldn’t be. But where do you actually draw the line between architecting ahead and just fixing things as they come up, so you learn from them?