Deploying a web application, and knowing when it breaks

Most deployment guides end at the moment the site loads. That is roughly halfway. The interesting failures start afterwards, and they are unusually boring ones: a certificate expires, a DNS record is edited in a hurry, a config reload drops a security header, a process dies at 04:00 and nothing restarts it. None of these announce themselves. You find out because a customer tells you, or because you happened to look.

This walks the whole path once — a small application on a plain Linux server, with TLS, behind a reverse proxy, supervised — and then spends the second half on the part that decides whether the first half stays true.

Pick a host you can reason about

The first decision is the one most likely to be made on vibes, so it is worth being explicit about what you are actually choosing between.

A platform-as-a-service — Vercel, Render, Fly, Railway — takes the server away. You push, it builds, it runs. That is genuinely the right answer for a lot of applications, and if your app is a Next.js frontend with no long-running background work, stop reading and use one. The trade is that when something behaves strangely you are debugging someone else's abstraction, and your costs are a function of traffic in a way that is hard to predict before you have traffic.

A plain virtual machine gives you the opposite trade. You get a Linux box, a fixed monthly price, and full visibility: it is your nginx, your systemd, your logs. You also get all the work. You are now the person who applies kernel updates.

For a small product that has background jobs, a database it wants next to the app, and an operator who needs the bill to be a number rather than a forecast, the VM is usually the better trade — and the host I run on is DigitalOcean. Three specific reasons, all of which are checkable:

• The price is a number, not a formula. A droplet costs the same on the day you get written about as on the day you do not. For an operator paying out of pocket, a bill that cannot surprise you is worth more than a slightly lower average bill that can.
• It is an unremarkable Linux box. Nothing you learn setting it up is vendor knowledge. The nginx config below works identically on a VM from any provider, which also means leaving is a weekend rather than a migration project.
• The documentation is the actual reason. Their community tutorials are, for this class of task, the best-maintained writing on the open internet, and they are free to read whether or not you are a customer.

What it is bad at, in the same breath: nothing here is managed. Nobody patches the box, nobody fails it over, and a single droplet is a single point of failure — which stays true no matter whose logo is on it. If that is unacceptable for what you are building, a PaaS or a managed container service is the honest answer and you should take it.

The deploy, once

Assume an app that listens on 127.0.0.1:8000. Bound to localhost, deliberately: the only thing that should be reachable from the internet is the proxy.

1. Point DNS at the box, and wait

An Arecord for your domain pointing at the server's IPv4 address, and an AAAA record if you have IPv6. Set the TTL low — 300 seconds — before you need to change anything. TTL is a promise you made in the past about how long resolvers may cache the answer, so lowering it during an incident does nothing for the resolvers already holding the old value.

Check what the world can see, not what your control panel says it saved:

dig +short A app.example.com
dig +short AAAA app.example.com

2. Terminate TLS at a reverse proxy

nginx in front, application behind. This is worth doing even for a single app, because it puts certificates, headers and timeouts in one file that is not your application code.

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    http2 on;
    server_name app.example.com;

    ssl_certificate     /etc/letsencrypt/live/app.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/app.example.com/privkey.pem;

    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header Referrer-Policy "strict-origin-when-cross-origin" always;

    location / {
        proxy_pass http://127.0.0.1:8000;
        proxy_set_header Host              $host;
        proxy_set_header X-Real-IP         $remote_addr;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

server {
    listen 80;
    listen [::]:80;
    server_name app.example.com;
    return 301 https://$host$request_uri;
}

Two details in there are the ones people lose. The always on each add_header is what makes the header appear on error responses too; without it your 500 page is served without HSTS, which is exactly the response an attacker would like to provoke. And add_header in a location block silently discards every add_header inherited from the server block — so if you later add one header inside a location, you have just deleted the other three from that path. This is nginx behaving as documented, and it is the single most common way a header that was verified once stops being sent.

Certificates from Let's Encrypt, via certbot:

sudo certbot --nginx -d app.example.com
systemctl list-timers | grep certbot   # confirm renewal is actually scheduled
sudo certbot renew --dry-run

Let's Encrypt certificates are valid for 90 days and the renewal timer is meant to handle it silently. Both of those are true and neither is the point: the renewal runs unattended, which means when it starts failing it also fails unattended. More on that below, because it is the single most common way a working site becomes an unreachable one.

3. Supervise the process

A systemd unit, so the app restarts on crash and comes back after a reboot. Running it in tmux counts as deployed for about a week.

[Unit]
Description=Example app
After=network.target

[Service]
User=app
WorkingDirectory=/srv/app
ExecStart=/srv/app/.venv/bin/uvicorn main:app --host 127.0.0.1 --port 8000
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

sudo systemctl enable --now example-app
systemctl status example-app
journalctl -u example-app -f

Then close the rest of the box: a firewall allowing 22, 80 and 443 and nothing else, SSH keys only, unattended security upgrades on. If the database runs on the same droplet, it listens on localhost.

What actually breaks after this

Everything above is now correct, and it will stay correct until it does not. In roughly descending order of how often it bites:

• The certificate. Renewal is automatic until the plugin can no longer bind port 80, or the DNS challenge stops resolving, or a config edit broke the reload hook. The timer keeps running and keeps failing, and the failure is a line in a log nobody reads. Then one morning every visitor gets a full-page browser interstitial — not a degraded site, an unusable one.
• DNS. A record edited during an unrelated change, a registrar transfer that drops a record, a nameserver change that propagates unevenly. The site works from your machine, which is holding a cached answer, and does not work from anywhere else.
• Headers. Someone adds an add_header inside a location block, reloads nginx, and three security headers stop being sent on that path. Nothing errors. Nothing looks different.
• The process. Restart=always handles a crash. It does not handle the app coming back up and returning 500 to every request because a migration is half-applied, and systemd will report that service as perfectly healthy.
• Redirects. The apex-to-www rule, or the HTTP-to-HTTPS rule, quietly becoming a loop after an edit.

What these share is that none of them are visible from inside the server. systemctl status is green for four of the five. You have to look from outside, on a schedule, forever — which is precisely the kind of task a person is bad at and a machine is good at.

Monitoring it

This is the product I build, so read the rest with that in mind. Here is what it does and what it does not.

Nivaronix checks a domain from outside it, on a schedule: TLS certificate validity and expiry, DNS records, whether the site responds at all, the security headers it returns, its cookies, and its redirect chain. When something changes it opens an incident and sends an alert. The free scan runs the same checks once, in the browser, with no account — which is the fastest way to find out whether the nginx config above is doing what you think it is.

The free tier is a free scan and one monitored domain, with no card. To put a domain under continuous monitoring you verify you control it first, which is deliberate: a monitoring tool that will watch any domain you type is a reconnaissance tool.

What it is not: it is not an SLA, and there is not one. Nivaronix runs in a single region on a single host, with no automatic failover and nightly backups. If that region has an outage, checks pause until it returns — your data is not lost, but nothing is measured during the gap. That is written up in full on how we're built, and it is the same argument as the DigitalOcean section above: a monitoring vendor that is candid about its own limits is making the strongest available case that its measurements are honest.

You do not need this product specifically. You do need something external and scheduled. An hourly cron job on a different machine that runs curl -sS -o /dev/null -w '%{http_code}' against your health endpoint and emails you when it is not 200 is a worse tool than a real one and an enormously better tool than checking manually. What is not acceptable is the default: no external check at all, and a customer as the alerting mechanism.

The checklist

Before you call it deployed:

• dig resolves the domain from a network that is not yours.
• certbot renew --dry-run succeeds, and the renewal timer appears in systemctl list-timers.
• curl -I https://app.example.com shows the security headers — and shows them on a 404 too, not only on the homepage.
• HTTP redirects to HTTPS in exactly one hop.
• The service survives sudo reboot with no intervention.
• Firewall allows 22, 80, 443 and nothing else.
• Something outside the server checks all of the above on a schedule, and can reach you when it fails.

The last line is the one that is usually missing, and it is the one that determines whether any of the others are still true next month.