Nightfix

Quieter nights
are coming.

Opensource incident response for your AWS account. It tries the obvious fix first and only pages you when that fails.

View on GitHub See it handle an incident

3:07 a.m., and nobody wakes up

Here's what an alert looks like with Nightfix, taken from the incident's timeline.

  1. The alarm fires

    checkout-api goes unhealthy and an incident opens.

  2. The fix starts

    Nightfix finds the restart runbook and holds the page while it runs.

  3. The agent checks first

    The agent in your account confirms the runbook is signed, then restarts the service.

  4. The service recovers

    It settles back to healthy.

  5. The alarm clears

    The incident closes itself, and there's a summary waiting in Slack in the morning.

  6. Nobody was paged.

If the fix doesn't work, your pager goes off right away, with everything Nightfix already tried.

How it works

  1. Alerts come in

    From CloudWatch, or anything that can send a webhook. Repeats become one incident.

  2. Runbooks run the fix

    Runbooks are files you review like code. Risky steps wait for a teammate's OK in Slack.

  3. You're paged when it matters

    If the fix fails, Nightfix pages you through PagerDuty and tells you what it tried.

Everything stays in your account

It runs code in production, so it's careful about it.

  • Nothing calls home. It all runs in your AWS account.

  • Outbound only. No inbound ports to open.

  • No tokens to leak. Agents sign in with their IAM role.

  • Signed runbooks only. You can limit each agent to the ones you've approved.

  • Secrets stay put. They're read when needed and scrubbed from output.

  • Everything is on record in an audit log that can't be quietly edited.

Fits what you already run

It can use the network, DNS, database and keys you already have.

Alerts
CloudWatch, signed webhooks
Paging and chat
PagerDuty, Slack
Sign-in
Okta, Google Workspace, Entra ID, Keycloak, Auth0, or API keys
Runs on
ECS Fargate or EKS, from Terraform modules and a Helm chart

Where it is today

An early project, built in the open.

  1. Now

    Works end to end on your machine, against stand-ins for AWS, Slack and PagerDuty.

  2. Next

    First run on real AWS, then v0.1.

  3. Later

    Built-in paging and a web dashboard.

Try it in a minute

The demo runs the whole thing on your machine. You'll need Go and PostgreSQL.

Ready for AWS? Start with the deployment guide.

git clone https://github.com/peacedouts/nightfix
cd nightfix
make demo        # fixed, nobody paged
make demo-fail   # fix fails, on-call paged