Build Better Stack

The build, in order

1. 1

   The check

   hideshow

   One function from a monitor to a result, with a timeout that actually fires and distinct error kinds.

   1. 1.1Create the project and the three tables

      monitors (id, name, url, interval_seconds, timeout_ms, expected_status, keyword, enabled, status, consecutive_failures, last_checked_at, last_change_at), checks (id, monitor_id, checked_at, ok, status_code, latency_ms, error), incidents (id, monitor_id, started_at, ended_at, cause). Index checks(monitor_id, checked_at).

      Files server.mjsdb.mjscheck.mjs

      terminalcopy

      mkdir uptime && cd uptime && git init && npm init -y && npm pkg set type=module
      mkdir data && cp .env.example .env

   2. 1.2Write check(monitor)

      fetch with an AbortController timeout, follow redirects, read at most 64 kB of body for the keyword, record latency. Return ok plus one of: dns, tls, timeout, status, keyword, as the error.

   3. 1.3Add a CLI to run one check by hand
      terminalcopy

      node scripts/check.mjs https://example.com 200 "Example Domain"

   done when · tick each as it passes

   • A good URL records ok with a sane latency
   • A URL that never responds fails at the timeout, not later
   • A 404 against expected 200 fails with the code recorded
   • A keyword monitor fails when the word is absent from a 200 body

   next phase
2. 2

   Scheduler

   hideshow

   Each monitor on its own interval, concurrently with a cap, and one bad monitor never stops the loop.

   1. 2.1Write the loop

      Every 5 seconds find monitors due (last_checked_at plus interval before now), run up to 10 concurrently, catch every exception per monitor. On boot treat every monitor as due.

   2. 2.2Store a checks row per run and update the monitor's last_checked_at

   done when · tick each as it passes

   • Monitors at 30, 60 and 300 seconds keep their cadence over ten minutes
   • A monitor pointed at a hanging host does not delay the others
   • A thrown exception inside one check leaves the loop running

   next phase
3. 3

   Transitions and incidents

   hideshow

   Down only after N consecutive failures, up on the first success, one incident per outage.

   1. 3.1Implement the transition rule

      Increment consecutive_failures on a failed check; at FAILURES_BEFORE_DOWN flip to down. Any success resets the counter and flips to up.

   2. 3.2Open and close incidents on the transitions

      Open a row on the flip to down with the first failure's error as cause; set ended_at on recovery. Incidents, not checks, are what people read later.

   done when · tick each as it passes

   • A single blip opens no incident
   • Two consecutive failures open one
   • Recovery closes it with a correct duration
   • Flapping up/down/up/down produces one incident, not four

   next phase
4. 4

   Alerting

   hideshow

   One message down, one up, retries that never stall the loop.

   1. 4.1Write the sender for ALERT_FORMAT

      Down: monitor name and the error. Three retries with backoff, then record the failure and move on.

   2. 4.2Send the recovery message with the outage duration

      Duration comes from the incident row's started_at and ended_at.

   done when · tick each as it passes

   • Taking a site down produces exactly one message
   • Keeping it down produces none
   • Recovery produces exactly one with the duration
   • A webhook URL that 500s does not stall checking

   next phase
5. 5

   Status page

   hideshow

   A public page whose numbers can be argued with, and a feed people can subscribe to.

   1. 5.1Build /status

      One row per monitor: dot, state, uptime over 24 h, 7 d and 30 d with the denominator printed (over 2,880 checks), a latency sparkline as inline SVG, recent incidents with durations. Meta refresh every 30 seconds.

   2. 5.2Add /status.rss with one item per incident

      Subscription without accounts.

   3. 5.3Seed 90 days of checks and time the page
      terminalcopy

      node scripts/seed.mjs 90

   done when · tick each as it passes

   • Percentages match a hand-written query
   • The page renders with zero history on a fresh install
   • It loads in under 200 ms with 90 days of checks
   • The RSS feed validates

   next phase
6. 6

   Retention and deploy

   hideshow

   Old checks pruned, incidents kept, live behind HTTPS on the other provider.

   1. 6.1Nightly prune of checks older than RETENTION_DAYS, never incidents
   2. 6.2Add /healthz, systemd with restart-on-failure, Caddy, and the README

      README: the single-region caveat stated plainly, the instruction to host on a different provider than the things watched, and the RSS URL.

      Files deploy/uptime.serviceCaddyfileREADME.md

   done when · tick each as it passes

   • Pruning does not distort the 30-day figure
   • The service survives a reboot with state intact
   • The status page states the single-region caveat

   next phase