launchd Job Failed Silently: Exit 28, Zero-Byte Log

My daily metrics report stopped running on 2026-07-30 at 21:30. I found out on 2026-07-31 at 18:05 — 20.6 hours later, and only because I was digging through launchctl output looking for something else. Nothing alerted me. The job that tells me how the business is doing died, and the way I learned about it was luck.

The error log was configured correctly and contained nothing. That combination — a real failure, a correctly wired StandardErrorPath, and zero bytes in it — is what this post is about, because the standard advice for debugging launchd jobs is "read the error log," and here that advice returns an empty file.

What launchd actually knew

launchd had the answer the whole time. It just never volunteered it:

$ launchctl print gui/501/com.mmm.daily-report
        state = not running
        runs = 10
        last exit code = 28

Exit 28. Meanwhile ops/schedule/report.log, the file the job appends to when it succeeds, ends at [2026-07-29 21:30] daily-report sent. And daily-report.err.log — the path named in the plist — was 0 bytes, last modified 2026-07-21 21:30. Ten days without a single character written to it, including the night the job failed.

Per the launchd tutorial that most search results point at, launchctl list gives you three columns: PID, last exit code, and label. A - in the PID column is normal for a calendar-interval job that runs and exits. The exit code is the part carrying the signal, and it is the part I had never looked at. I had been checking that my jobs were loaded, which is a different question from whether they work. A 78 in that column is the spawn family, and a refused load never gets a column at all; the launchctl error 5 probe map covers both.

Why the error log was empty

The whole failure lives in one line, ops/schedule/daily-report.sh:7:

set -euo pipefail
STATS=$(curl -s --max-time 30 "https://go.picklog.cc/stats?key=${STATS_ADMIN_KEY}")

Three things compose here, and each one is individually reasonable.

--max-time 30 gets exceeded, so curl exits 28. The curl documentation defines 28 as "the specified time-out period was reached," and on the same page: "When curl exits with a non-zero code, it also outputs an error message (unless --silent is used)." I used --silent. The manual is explicit that -s means "don't show progress meter or error messages," and that -S exists specifically to put the error messages back.

Then set -e aborts the script at line 7. The Telegram send is on line 60 and the log append is on line 64. Neither runs.

I reproduced the flag half of this in isolation, against a deliberately slow endpoint:

$ cat repro.sh
set -euo pipefail
OUT=$(curl -s --max-time 2 "https://httpbin.org/delay/10")

$ ./repro.sh 2>err.txt; echo "exit=$?"; wc -c <err.txt
exit=28
       0

# same script, -s changed to -sS:
exit=28
curl: (28) Connection timed out after 2075 milliseconds

One character. -s versus -sS is the entire difference between a silent death and a logged one, and it is invisible at the plist layer where I was doing my checking. The shell itself keeps a similar family: zsh's no matches found and its sibling defaults kill an unattended one-liner with error strings that never name the mechanism.

Why 30 seconds wasn't enough

I called the same endpoint three times in a row and timed it:

Call	HTTP	Total time	Body
1 (cold)	200	54.096s	17,914 B
2 (warm)	200	4.714s	17,914 B
3 (warm)	200	4.688s	17,914 B

Cold is 1.8× the timeout; warm clears it with room to spare. So this job doesn't fail every night, it fails on the nights the worker happens to be cold — which is worse than failing consistently, because an intermittent fault buys itself more time before anyone notices. Why /stats is slow at all is something I already documented when I measured what actually exhausts the Workers KV free tier: that endpoint does a full scan, and the scan is the expensive part.

Four states, not two

Once I started reading exit codes I audited all of it. The repo holds 8 plists; 7 are installed in ~/Library/LaunchAgents; and the health of those 7 splits further:

Label	runs	last exit code	Reading
daily-content	19	0	working
social-dispatch	885	0	working
social-planner	3	0	working
daily-revenue	2	0	working
daily-report	10	28	failing
threads-token	0	(never exited)	not due yet
weekly-review	0	(never exited)	not due yet
karma	not loaded at all		never installed

com.mmm.karma.plist has been sitting in ops/schedule/ since 2026-07-22. Its target script ops/growth/karma-run.sh exists and is executable. The job was simply never bootstrapped, and the giveaway is that karma-run.err.log doesn't exist — launchd creates that file when it first spawns the job, so a missing error log means zero runs ever. Nine days of a scheduled task that was only ever scheduled in my head.

Update, 11 August 2026: reading the exit code turned out to be necessary but not sufficient. An audit of 138 scheduled runs found that Claude Code exit code 1 covers five unrelated failures, and that two runs reporting it had already published a post before dying — so a non-zero status on its own does not mean the job did nothing. A later bench test found the counter has a second blind spot: there is no next-fire-date anywhere in launchctl print and no spawn record in the unified log, so a missed StartCalendarInterval firing leaves no artefact at all.

The two zero-run jobs are the trap in the other direction. threads-token fires on Weekday 1 at 09:20 and was installed on a Tuesday; weekly-review fires on Weekday 0 at 07:00 and was installed on a Wednesday. Neither has met its day yet, so runs = 0 is correct behaviour. runs = 0 and (never exited) cannot distinguish "healthy but not due" from "silently broken." You have to read it against the schedule, which means a health check that only greps launchctl list is guessing on exactly the jobs that run least often.

Update, August 6: weekly-review met its day on August 2 and died on a usage limit before writing anything. Its artifact check passed anyway, against last week’s file, thanks to a one-line date bug — the post-mortem became the case for a cron dead man’s switch.

Update, August 8: the one-character fix below stayed in the repair queue while the job died nightly through August 8 — ten missed reports between diagnosis and action. The curl side of this failure — what exit code 28 actually means, what --retry does with a timeout, and why launchctl reports it as 7168 — now has its own reference post.

The finding that reversed my conclusion

I grepped every curl call in the repo expecting daily-report.sh to be the careless one. Across 12 shell scripts under ops/ there are 12 curl invocations:

Flag	Count
-s / --silent	12 / 12
-S / --show-error	0 / 12
-f / --fail	0 / 12
--max-time or --connect-timeout	1 / 12

The single call with a timeout is the one that broke. Every other script in this repo would have sat through that 54-second cold response without complaint, and if the endpoint had hung instead of merely being slow, they would have waited indefinitely, because curl imposes no overall time limit unless you ask for one. daily-report.sh failed because it was the only script doing the responsible thing. The other eleven don't fail; they hang, which on this rig is the more expensive outcome — daily-content.sh holds a mkdir lock while it runs and only reclaims a stale one after -mmin +180, so one hang costs up to three publishing slots.

Neither daily-report.sh nor daily-revenue.sh has a trap or any failure notification. I built a Telegram ops channel precisely so this rig could tell me when something breaks, and then wrote the reporting jobs so the Telegram call sits below the line most likely to fail. The alarm was wired downstream of the fault.

Update (2026-08-08): the same absence-shaped failure hit again from a different direction: the macOS /tmp cleanup daemon deleted a virtualenv this fleet imports daily, with no error in any log — the deletion is scheduled system behavior, not a fault.

What I'm changing

The mechanical fix is the pattern healthchecks.io recommends for exactly this: curl -fsS --max-time 10 --retry 5. -f turns HTTP 4xx/5xx into a non-zero exit instead of a success carrying an error page, -S restores the message under -s, and the timeout is explicit. That is a one-line change in 12 places.

The structural fix matters more, and it's the one this rig didn't have: a job needs to prove it ran, rather than staying quiet when it doesn't. Absence has to be the alarm. The failure mode is well known outside my repo — the author of CronSafe built it after a client's backup script stopped running for 11 days and nobody noticed until a restore was needed. My 20.6 hours is the same shape, caught earlier by accident rather than by design.

Worth being clear that StartCalendarInterval doesn't help here. When I compared launchd and cron on this machine, the advantage I found was that launchd catches up missed runs after sleep. It catches up runs that never started. A run that started and exited 28 is, as far as launchd is concerned, complete.

I haven't applied either fix yet. This slot went to the measurement, and the same guardrail that produced this post — one unit of work per scheduled run — says the change is its own run. The next morning the same missing alert cost more than a status report: the Claude Code weekly limit stopped 23 publishing slots and nothing announced it, because the publishing script has no notification path either. The counterexample on the same machine is the Threads token refresher, which alerts on both failure and staleness because its 60-day credential dies for good if it stops running. The prompts and scheduler scripts behind this setup are in the Playbook.

The part I still don't know: whether the 2026-07-30 failure was a genuinely cold worker or something slower upstream that night. I have exit 28 and today's 54-second cold call, which makes timeout the obvious reading, but I did not capture the response time at 21:30 and I can't reconstruct it. That gap exists for the same reason the rest of this post does — the script wasn't recording anything.

Update, 2026-08-04: the same shape of silence reappeared two layers up. A claude -p publish run exited 0 with its final steps unrun, because it waited for a background deploy that print mode kills about five seconds after the final result. Different tool, same lesson: the success signal sat downstream of the failure point.

Update, 2026-08-07: Anthropic now ships the mechanism this failure was missing. The self-hosted runner in Claude Code 2.1.224 makes every poll double as a heartbeat and a lease — if a runner goes quiet for about 60 seconds, the server requeues its session. My 20.6 hours of undetected silence is what the absence of that loop costs.

Update, August 12: still 28, and a blind spot I missed

Thirteen days on, this job is still broken. launchctl print gui/501/com.mmm.daily-report reports last exit code = 28 tonight, and report.log still ends at its 2026-07-29 21:30 line. I also found a hole in the health check I proposed above: a job killed by a signal prints no last exit code line at all, because launchd renames the field to last terminating signal. Grepping for the exit code goes blind exactly when something kills the job. I measured that and nine other outcomes in launchctl print last exit status.

Update, August 14: how to read that zero-byte file

One correction to how I read the empty err.log above. I had treated its stale mtime as a clue about when the job last ran; it is not one. Spawning a probe agent twice with nothing written to stderr leaves the file at 0 bytes with the mtime byte-identical across both spawns, because opening for append never touches it. So a zero-byte launchd log is dated the first time the job ever ran. The same round of experiments turned up a related trap in launchd log rotation: an unwritable StandardOutPath makes launchd refuse to run the job at all, reporting 78: EX_CONFIG while still incrementing runs.

Update, 2026-08-15: the same silence, inverted

The lesson here was that a failing job can produce no signal, so absence of output is not evidence of health. Auditing my Claude Code hooks turned that around. A hook that exits 0 without printing anything is recorded nowhere, so in that surface silence is what success looks like, and the instinct this post trains would send you off to rewrite a hook that was working. Across 2,867 sessions of transcripts my shell hook had zero records while running the entire time; the measurements are in a post on Claude Code hooks that look like they are not working.

Silent death costs more than the missing signal. A run killed without its EXIT trap firing also leaves its lock directory on disk, and that is how a reboot stranded my publisher's lock across six scheduled slots before anything reclaimed it — the macOS flock alternative in /usr/bin/lockf holds its lock in the kernel, where a dead process cannot keep it.

Update (2026-08-28): this failure family has a new record holder on this machine: a neighboring LaunchAgent whose KeepAlive loop crashed every 31 seconds for 91 days — 192,257 identical log lines — with nothing wired to tell a human. Same lesson, bigger number: launchd keeps its contract indefinitely; alerting is your job.

Update, 5 September 2026: One more silent failure: when a run hangs, launchd drops the calendar fires that land on it, so a stuck 15:00 job erases 16:30 and 18:00 with no log line. Is Claude Code stuck? 255 unattended runs, two real hangs measures that from 44 days of this log and adds the transcript-silence watchdog that would have caught it.

Everything here was measured on this rig on 2026-07-31 (Darwin 25.4.0, KST): launchctl print gui/501/<label> for all 8 job labels, three timed calls to our own /stats endpoint with curl -w, a two-script reproduction of the -s versus -sS stderr difference against httpbin.org, and a grep of all 12 shell scripts under ops/ for curl flags. The 20.6-hour figure is the gap between the missed 21:30 run on 2026-07-30 and my reading the exit code at 18:05 today. curl behaviour is quoted from the official manual and everything.curl.dev rather than inferred; the launchctl list column semantics come from launchd.info. No fix has been deployed as of publication, and the karma job is still not installed.

Update, 2026-08-16: the same script produced a second lesson in exit codes months later. A later publishing run misread a failure as exit 0 when the zero was really coming from a pipe, and chasing it turned up that curl --fail returns 22 over HTTP/1.1 but 56 over HTTP/2 — measured on six hosts. The pipefail sweep across ops/ is still not done.

Update, 2026-08-25: a zero-byte log does not always mean the job never wrote to it. Python block-buffers sys.stdout at 131,072 bytes whenever the destination is not a terminal, and a launchd job’s StandardOutPath never is, so a script that prints a few hundred bytes of progress leaves the log file empty until the process exits. I measured that redirecting to a file behaves exactly like piping: zero bytes for five seconds, then 131,072 at once. Check isatty() before concluding the job produced nothing.