enhancement: Keep the agent's samples and events on disk until the control plane accepts them #27

Blocker: yes. #28 and #29 send their data through this.

Part of the monitoring agent (contract v0.2.1). Order: #26 → #27 → #28 → #29 → #30, with #9.

Defect

• The contract requires that the agent keeps unsent samples and events on disk (sections 4 and 6).
• The CLI has no store for this.

Cost

• Without this store, a network failure or a restart loses samples. The charts then have gaps.
• The contract requires the same event id for each resend. Without a store, a resend gets a new id.

Benefit of the fix

• An outage of up to 24 hours loses no data.
• A resend is safe. The control plane keeps one sample for each minute, and applies each event id one time.

Fix

1. Keep two queues in the state folder: samples and events. Write each change to a temporary file, sync it, then rename it.

2. Samples: keep at most 1440 (24 hours). When the queue is full, drop the oldest sample first.

3. Set recorded_at when the agent measures. Do not change it on a resend.

4. Events: make the ULID when the event goes into the queue. Keep the id with the event. Use the same id for each resend.

5. Before a value goes into a queue, compare it with the ranges of the contract. Cut data.error to 2000 characters. Reason: one bad value makes a 400, and a 400 drops the full request.

6. Send at most 240 samples and at most 100 events in one request. Send the oldest first. Send one request at a time (the limit is 30 requests each minute for each token).

7. In each cycle, do these steps in this order: send the events, send the samples with the status, then poll for commands (enhancement: Run agent.update and agent.restart one time only #29). Events first closes finished commands before the poll.

8. Apply the rules of the contract for each reply:

   Reply	Samples	Events
   200	drop the samples that were sent, also the rejected ones	drop
   400	drop, and write a log line	drop, and write a log line
   401	keep, and try again each 5 minutes	keep, and try again each 5 minutes
   422	drop	—
   429	keep, and wait for Retry-After	keep, and wait for Retry-After
   5xx, or no reply	keep, and wait longer after each failure	keep, and wait longer after each failure

9. Continue to measure while the sender waits.

10. At each start, put agent.started with the version in the event queue.

Decision necessary

1. How does the agent keep its queues on disk?
   • Recommendation: plain JSON files in the state folder, written as step 1 tells. The largest queue is 1440 samples, about 300 KB. No new dependency.
   • Other option: go.etcd.io/bbolt, a small embedded database. It adds a dependency and a file format.
2. Where do the event ids (ULIDs) come from?
   • Recommendation: github.com/oklog/ulid/v2. It is small and well known.
   • Other option: about 30 lines of our own code.

Completion checks

• The queues are the same after an agent restart and after a server reboot.
• After a 503, the samples stay in the queue. After a 400, they go, and a log line shows why.
• Sample 1441 drops the oldest sample.
• 1000 samples go in 5 requests of at most 240.
• A resent event has the same id.
• After a 429, the agent waits for the time in Retry-After.
• A kill -9 during a write leaves a queue file that the agent can read.