Heartbeat Monitoring for Jobs That Fail Silently

Why silent failure needs its own kind of check

Ordinary monitoring answers a question about something that exists: is this site reachable, is this certificate still valid, is this DNS record still correct. It works because there is something to go and look at. A scheduled job has no such surface. A nightly backup that silently stopped running three weeks ago looks exactly like a nightly backup that ran perfectly, right up until the moment you need the backup.

A heartbeat check inverts the question. Instead of Nivaronix reaching out to your job, your job reports in to Nivaronix. Silence is the signal. If the report does not arrive on time, that absence is what opens the incident.

What people watch with them

Anything scheduled, unattended, and quiet when it breaks:

• Nightly backups — The canonical case. A backup that stops running produces no error anyone reads, and the gap is usually discovered by the restore that needed it.
• Cron jobs and scheduled scripts — A cron entry that was removed, a host that was rebuilt, a crontab that never survived the migration. Cron reports failure to a mailbox nobody watches, and reports a job that never fired at all to nobody.
• ETL and data pipeline runs — A pipeline that skips a night leaves yesterday's data sitting there looking plausible. Nothing downstream errors; the numbers are just quietly stale.
• Queue workers and consumers — A worker that dies stops draining its queue rather than announcing anything. Have the worker ping at the end of each processing cycle and its silence becomes visible.
• Certificate renewal jobs — The renewal job itself, as distinct from the certificate. Nivaronix can already watch a certificate's expiry; a heartbeat on the renewal job tells you the automation stopped weeks before the expiry date makes it obvious.

Interval and grace period

A check is configured with exactly two numbers, and both mean precisely what they say:

Expected interval

How often the job is expected to ping. A whole number of seconds, and at least 60 — the API rejects anything shorter or non-integer rather than quietly rounding it.

Grace period

Slack added on top of the interval before silence counts as missed. Zero or more seconds; it defaults to zero. This is what absorbs a job that runs a little late without paging anyone — a nightly backup whose runtime drifts by twenty minutes should not be an incident.

The deadline is the last ping plus the interval plus the grace period. Nothing else feeds into it: no averaging, no learned baseline, no adjustment based on past behaviour.

The four states a check can be in

Status is calculated at read time from the last ping and the check's own configuration, not stored on the row and refreshed on a timer. A status you are looking at is never older than the request that fetched it.

What happens when a heartbeat is missed

1. The job pings — Each successful run sends one request to the check's ping URL. The timestamp of the most recent ping is recorded, and that timestamp is what the next deadline is measured from.
2. A sweep finds the silence — A sweep runs on a schedule as part of the same monitoring cycle that drives Nivaronix's other checks, and evaluates every heartbeat check against its deadline. A check past interval plus grace is missed.
3. One incident opens — A missed check opens a real Incident against the asset the check belongs to — the same incident and notification pipeline a Monitor uses, not a separate side-channel alert. The incident is recorded at critical severity, titled with the check's name, and describes when the last ping arrived (or states that none ever has) alongside the configured interval and grace period.
4. Exactly one, per missed window — A missed window is claimed under a unique key before the incident is created, so two sweeps overlapping cannot both open one for the same silence. A check that misses, recovers, and misses again gets a distinct incident for each outage, because a recovery moves the window forward.
5. A ping resolves it — When a ping finally arrives while an incident from that miss is still open, the incident is resolved automatically and stamped with a root cause recording that the job reported in and recovered on its own. Nobody has to close it by hand.

The ping URL is a credential

Each check has one ping URL whose only identifying component is a long, randomly generated, URL-safe token. It is never sequential and never derived from your account, your asset, or the check's name.

The endpoint takes no authentication headers and no request body, and accepts either GET or POST, so a plain curl line at the end of a shell script, a cron entry, or a CI step is enough to wire it up. A successful ping returns a small JSON acknowledgement.

• The token is returned once, when the check is created, and again when it is rotated — the same posture as an API key. Copy it into your job at that moment.
• If a ping URL is exposed, rotate the token. Rotation issues a new one immediately and permanently invalidates the old — which also means every job still calling the old URL stops being heard from, so update them in the same change.
• A request bearing a token that matches no check gets a flat not-found response, identical whatever the reason. The endpoint is public, so it is deliberately not usable to probe whether a given token exists.
• Each token has a generous per-token request ceiling — well above what a job pinging once per interval needs, and there to keep a misconfigured or retrying script from hammering the endpoint.

Setting one up

1. Create the check against one of the assets in your account, give it a name you will recognise in an alert — "Nightly backup job" rather than "hb-1" — and set its interval and grace period.
2. Copy the ping URL returned at creation into your job, as the last thing the job does on a successful run. Putting the ping at the end, after the work, is the point: a ping fired at the start reports that the job started, not that it finished.
3. Leave it. A heartbeat check is never polled and costs nothing to hold until it misses.

How many checks each plan includes

Heartbeat checks are counted separately from monitors and do not consume your monitor allowance, because a heartbeat check is never polled — it costs nothing to hold until it misses. The limit is enforced when a check is created.

Common questions

What is a heartbeat check?

A dead-man's-switch for a scheduled job. You give Nivaronix an expected interval and a grace period, and a ping URL to call. Your job calls that URL every time it completes. If a ping does not arrive within the interval plus the grace period, the check is missed and an incident opens.

Does a heartbeat check know whether my job worked correctly?

No. It only knows whether a ping arrived in time. It never sees the job, its output, its logs, or its exit code. A job that runs, does the wrong thing, and pings anyway is a healthy heartbeat check. Heartbeats catch the run that did not happen — not the run that happened badly.

What happens when a heartbeat is missed?

A real incident opens against the asset the check belongs to, at critical severity, through the same incident and notification pipeline that a monitor uses. It is titled with the check's name and describes when the last ping arrived — or that none ever has — along with the configured interval and grace period. Exactly one incident opens per missed window, even if sweeps overlap.

Do I have to close the incident myself when the job recovers?

No. When a ping arrives while the incident from that miss is still open, it is resolved automatically and recorded as having recovered because the job reported in.

Is the ping URL a secret?

Yes. The random token in the URL is the only credential the endpoint checks, so anyone holding the URL can report a successful run on your behalf — which would keep the check quiet while your job is actually dead. Store it like an API key, and rotate it if it is exposed. Rotation invalidates the old URL immediately, so update every job using it at the same time.

What is the shortest interval I can set?

Sixty seconds. The interval must be a whole number of seconds and at least 60; anything shorter is rejected rather than silently rounded. The grace period can be zero or any whole number of seconds above it.

Do heartbeat checks use up my monitor allowance?

No. They are counted under their own separate plan limit, because a heartbeat check is never polled and costs nothing to hold until it misses.