← All field notes

Automation

Automation that survives real operations

A useful automation is not merely a successful demo. It needs visible state, controlled failure, human ownership, and a path through exceptions.

Automation is often evaluated at the moment it works: a request arrives, data moves, a record is created, and a notification appears. The demonstration is satisfying because a visible task disappears.

Production operations are less forgiving. Inputs are incomplete. APIs time out. policies change. People correct records manually. The same request arrives twice. A case needs judgment that the automation does not have.

The difference between a demo and an operational system is how it behaves when the expected path breaks.

Make state visible

An automation that runs invisibly is difficult to trust.

The people responsible for the process should be able to answer:

  • Did the workflow start?
  • What step is it on?
  • What information did it use?
  • Did it complete?
  • If it stopped, why?
  • Who owns the next action?

This does not always require a dashboard. A clear status in the system where people already work may be enough. The important part is that state is durable and understandable without reading logs or asking the developer who built the workflow.

Design failure as a normal path

Retries are useful for temporary problems such as rate limits and network failures. They are not a complete failure strategy.

If a required identifier is missing, repeating the same request will not create it. If a policy exception needs approval, more retries only delay the moment a person becomes involved.

Failures should be classified:

  • temporary failures can be retried safely
  • data failures need correction or enrichment
  • business exceptions need a decision
  • system failures need technical attention

Each class needs a destination. A useful failure state includes the relevant context, the action required, and a named owner or queue.

Preserve idempotency

Operational automations often receive the same event more than once. A person resubmits a form. A webhook retries delivery. A queue redelivers a message after a timeout.

If processing the same event twice creates two invoices, two accounts, or two customer notifications, the automation is not safe.

Idempotency means the workflow can recognize work it has already handled and avoid producing a duplicate result. This usually requires a stable identifier, a record of completed actions, and careful handling of partially completed runs.

It is a technical property with a direct business outcome: people can recover from uncertainty without making the problem worse.

Keep human judgment explicit

Not every decision should be automated.

Rules work well when the required information is available, the outcome is predictable, and mistakes are easy to detect or reverse. Human review remains valuable when context is incomplete, consequences are significant, or policy intentionally allows discretion.

A strong workflow does not hide this boundary. It routes standard cases automatically and presents exceptions to a person with enough context to decide quickly.

The decision can then be recorded as structured information. Over time, repeated decisions may reveal a rule worth automating. Until then, the system supports judgment instead of pretending to replace it.

Give every automation an owner

Automations often outlive the person who created them. Credentials expire, schemas change, vendors update APIs, and business rules evolve.

Every production workflow needs an owner responsible for:

  • approving changes to its behavior
  • responding when it fails
  • reviewing whether it still serves the process
  • maintaining credentials and dependencies
  • documenting the important decisions built into it

Ownership does not have to sit with engineering alone. The business owner defines correct behavior; technical ownership keeps the implementation reliable.

Automate a controlled slice

The safest first automation is bounded. It has a clear trigger, explicit inputs, a small number of actions, and a visible completion state.

Once that slice is reliable, extend it. Add another integration, cover another exception, or reduce another manual handoff.

This incremental approach may feel less dramatic than automating an entire department in one project. It produces systems people can understand, operate, and improve.

The standard for automation should not be “it ran successfully once.” The standard should be “the team knows what it did, can recover when it fails, and remains in control as the operation changes.”