enhancement: Run agent.update and agent.restart one time only #29

Blocker: no. #30 and #9 use the same update code, so do them after this.

Part of the monitoring agent (contract v0.2.1). Order: #26 → #27 → #28 → #29 → #30, with #9.

Defect

• The contract requires that the agent polls for commands, and runs agent.update and agent.restart (sections 5 to 7).
• The poll sends each open command again until an event closes it. An agent that does not remember the commands it ran does the same command again.

Cost

• Without this, FlyWP can update or restart the agent only through SSH.
• Without a list of the commands that ran, one agent.restart restarts the agent each 10 seconds for 24 hours.

Benefit of the fix

• FlyWP updates the fleet without SSH.
• Each command runs one time, also after a crash or a resend.

Fix

1. Poll GET /agent/v1/commands in each cycle, after the sends of enhancement: Keep the agent's samples and events on disk until the control plane accepts them #27.
2. Keep a list of the command ids that the agent ran, in the state folder, for 48 hours. Skip a command whose id is in the list.
3. Run the commands one at a time, oldest first. A command that stops the process ends the cycle. The new process does the other commands.
4. For a verb that the agent does not know: do not run it. Put command.unknown in the event queue, and add the id to the list.
5. agent.restart:
   1. Add the id to the list, and save the list to disk.
   2. Exit. systemd starts the agent again (Restart=always).
   3. The new process puts command.completed in the event queue.
6. agent.update:
   1. If the agent already runs args.version or a newer version, put command.completed with its version in the event queue. Do not download.
   2. Accept only an https:// URL.
   3. Add the id and the target version to the list, and save the list to disk.
   4. Download the tarball to a temporary file next to the binary. Limit the size.
   5. Calculate the sha256 of the tarball. If it is not args.sha256, delete the file, put command.failed in the event queue, and continue with the old binary.
   6. Extract fly-linux-<arch>, make it executable, and rename it over the binary.
   7. Exit. systemd starts the new binary.
   8. The new process reads the list. If it runs the target version or a newer version, it puts command.completed with its version in the event queue. If not, it puts command.failed.
7. Put the download, check and replace code in one package. fly update (enhancement: Keep the agent correct after fly update #30) and enhancement: Examine release downloads with checksums #9 use it. Move it out of internal/utils (refer to enhancement: Refactor notes for the capabilities work #14).
8. In the same PR, add this line to the README: "Conforms to the FlyWP monitoring agent contract v0.2.1."

Caution

• The contract has no rollback. If a new binary cannot start, systemd starts it again each 10 seconds. The command stays open until it expires after 24 hours.
• The "stopped reporting" alert of FlyWP shows this problem. A rollback can be a later contract change.
• For the end-to-end test, the test server must get to the control plane through HTTPS. A local .test address does not work. Use a tunnel or a staging server.

Completion checks

• A resent agent.restart does not restart the agent again.
• A sha256 that does not agree gives command.failed, and the old binary continues.
• An update to the current version gives command.completed without a download.
• A verb that the agent does not know gives command.unknown, and the agent does not run it.
• On a test server, an update from the FlyWP admin panel shows the new version and a completed command after less than N + 1 minutes.
• The README has the pin line.