Scheduled AI agents with a human in the loop

There's a version of "AI agents" that gets demoed a lot: you give a model a goal, it goes off and does things, and it comes back with results. That works on stage. On a real server, running every day, with nobody watching in real time, it's a different problem.

I run scheduled AI agents on infrastructure I control. They do research, prepare drafts, run checks and summarize what happened overnight. What makes that workable isn't the model. It's everything around it: what the agent is allowed to touch, how much it can spend, how long it can run, how you find out when it fails, and who decides whether its output goes anywhere.

This is the practical setup, stripped of anything specific to my own stack. It builds on the idea in AI as an operating layer, not a chatbot: AI prepares, humans approve.

Treat each agent as a job, not a personality

The first design decision is to think of an agent as a scheduled job with a narrow purpose, not as a general assistant. Each one should have:

One clear task. "Draft summaries of yesterday's new support messages," not "help with support."
Defined inputs. Which files, folders, database tables or feeds it reads.
A defined output location. Where its work lands, in a predictable format.
A schedule. When it runs, in a timezone you've written down. I keep servers on UTC and convert in my head, because mixed timezones in schedules cause more confusion than they're worth.

If you can't describe the job in two sentences, it's probably two jobs.

Scope permissions tightly

A scheduled agent runs without you watching, so assume it will eventually do something you didn't expect. The question is how much damage that can cause.

Run it as a dedicated user, not as root or your own account.
Give it read access only to what it needs and write access only to its output directory.
Don't let it deploy, send, or publish. Those actions belong to a separate, human-triggered step.
Keep credentials out of its reach unless the task truly requires them, and then scope those credentials to the minimum.
Restrict which tools or commands it may call. Most agent runners let you set an allowlist. Use it.

The goal is that the worst outcome of a confused agent is a bad file in a review folder, not a changed production system.

Budgets, timeouts and health checks

Models can loop. Tasks can hang. An API can slow down. Without limits, a scheduled job that normally takes minutes can run for hours and cost real money doing nothing useful.

Every agent run should have:

A hard timeout. If it isn't done in a reasonable window, kill it and record that it timed out.
A spending or usage cap per run, set with some headroom above what a normal run uses. Heavier tasks get a bigger cap on purpose, not by accident.
An exit status that means something. Success, failure, timeout, or "ran but produced nothing" should be distinguishable in the logs.
A lock, so a slow run doesn't overlap with the next scheduled one.

Then add health checks that look at the outcome, not just the process. A job that "ran successfully" but wrote an empty file is not healthy. Check that the output exists, is recent, and looks like what it should look like.

A daily green/red report

The single most useful piece of this setup is boring: one status report, every day, that says whether each job is green or red.

Mine is an email. It lists each scheduled job, whether its last run succeeded, when it last produced output, and anything that needs attention, including how many items are waiting for review. If everything is fine, it's a short, green email I can read in ten seconds. If something is wrong, it's red at the top, and I know where to look.

A few rules make this report trustworthy:

It must arrive even when everything is fine. Silence should never mean "all good," because silence is also what a dead reporting job looks like.
It must be generated independently of the jobs it reports on. If the report depends on the same thing that broke, it will fail quietly with it.
It should be terse. Red items first, then counts, then details.

On top of the daily report, add immediate failure alerts for anything that shouldn't wait until tomorrow: a job that failed several times in a row, a disk filling up, a credential rejected.

Outputs land in a review queue

Nothing a scheduled agent produces goes straight to a customer, a website or a production system. It goes into a review queue.

The queue can be simple. A set of folders works: pending, approved, rejected. Each item includes what the agent produced, what inputs it used, when it ran and which version of the instructions it followed. The person reviewing sees everything needed to make a quick decision.

From there:

Approve moves it forward to whatever actually ships it, which is a separate, deliberate step.
Reject records it, ideally with a short reason.
Edit lets the human fix it, and the difference between the draft and the edited version is saved.

That last part matters. The edits and rejections are the feedback that tells you whether the agent is getting better or worse, and which instructions need work. Without it, you're running agents on faith.

One warning: review queues can quietly pile up. If approving is tedious, people stop doing it, and the queue becomes a graveyard. Keep the review surface fast, show the pending count in the daily report, and if a type of item is almost always approved unchanged, that's a signal worth examining, not an excuse to stop looking.

Log everything you'd want during a post-mortem

When something goes wrong, you'll want to know what the agent saw, what it did and what it produced. Keep:

Start and end time, exit status and duration for every run
The instruction version used
Inputs referenced (paths or IDs, not necessarily full contents)
Outputs written and where
Usage and cost per run
Any errors, verbatim

Rotate logs so they don't fill the disk, but keep enough history to compare this week to last month. Patterns show up over time that no single run reveals.

Boring on purpose

None of this is exotic. It's the same discipline you'd apply to any scheduled job on a server: least privilege, limits, monitoring, logs. The only new part is that the job is producing judgment-shaped output, which is why a human stays in the loop before anything ships.

That combination is what makes scheduled agents useful rather than stressful. The agents do the tedious preparation overnight. The report tells you in the morning whether everything ran. The queue shows you what's waiting. You decide what goes out. It's a small amount of structure, and it's the difference between trusting a system and hoping it behaved.