feature: Let the agent install signed releases on its own #39

Blocker: no. Ship this in the same release as the monitoring agent (v0.2.0).

Part of the monitoring agent. It comes after #10 in the same stack.

Defect

• The agent changes its binary only when the control plane sends agent.update, or when a person runs fly update.
• Nothing examines the releases on a schedule. A server stays on its version until an admin sends an update batch.
• The checksum file comes from the same release as the binary. It shows that a download is complete. It does not show who made the release.

Cost

• A fix reaches a server only when a person does an action.
• A server that the control plane cannot update (for example, an agent that is too old for a new contract) stays old.
• If the agent installs each GitHub release without a second check, each person who can push a tag on main can put code on all servers.

Benefit of the fix

• Each server installs a new release by itself, one time each day, at its own time.
• The agent installs only a release that has a signature from a key outside GitHub. The signature is the approval step.
• The command path stays. It is the path for a fast update, and the recovery path if the key is lost or stolen.

Fix

1. Add a signature file checksums.txt.sig to each release. It has the key id, the tag, the time of the signature and an ed25519 signature. The signature covers these lines and the bytes of checksums.txt.
2. Add a tool that makes a key pair and signs a release. Add make sign-release. The maintainer signs on their own computer. Do not put the private key in GitHub.
3. Put the public key in the binary.
4. Each day, at the server slot (server_id % 1440 minutes after 00:00 UTC), get the latest release.
5. If the release is not newer, stop. The agent never goes back to an older version.
6. Examine the signature. If there is no signature, or the signature is not correct, stop and write a log line.
7. If the signature is less than 24 hours old, stop. In this time, the maintainer can mark a bad release as a pre-release.
8. Install the release with the update code of enhancement: Run agent.update and agent.restart one time only #29, then exit. systemd starts the new binary.
9. Add the optional key FLY_AGENT_AUTO_UPDATE in /etc/fly/agent.env. The value off stops the check on that server.

Caution

• Keep the private key in a password manager, with a copy offline. If you lose it, send a binary with a new key through agent.update.
• The control plane must update only the agents that are older than its target. Make this change before you sign the first release after v0.2.0.

Completion checks

• A release without a signature does not install by itself.
• A signature for a different tag, or with a different checksums.txt, is refused.
• A signed release installs only after 24 hours, at the server slot.
• The agent examines the releases one time each day, also when the report interval is long.
• FLY_AGENT_AUTO_UPDATE=off stops the check.
• agent.update operates as before.