enhancement: Measure the server each minute for the monitoring agent #28

Blocker: no. No other issue waits for this.

Part of the monitoring agent (contract v0.2.1). Order: #26 → #27 → #28 → #29 → #30, with #9.

Defect

• The contract requires one sample each minute and the status of the server (section 4, and "What 'used' means").
• The CLI measures nothing.

Cost

• FlyWP cannot show the charts, the update status or the alerts of the server page.

Benefit of the fix

• The server page shows CPU, memory, disk and bandwidth, and the restart and update status.
• FlyWP can send an alert before a server runs out of a resource.

Fix

1. Take one sample each minute, at the cycle second of enhancement: Add fly agent run, the base of the monitoring agent #26.

2. Measure these values. All of them need no root.

   Field	Source
   cpu_percent	/proc/stat, the change since the last sample, 0 to 100
   load_1	/proc/loadavg
   memory_used_bytes, memory_total_bytes	/proc/meminfo: MemTotal − MemAvailable, and MemTotal
   swap_used_bytes, swap_total_bytes	/proc/meminfo: SwapTotal − SwapFree, and SwapTotal
   disk_used_bytes, disk_total_bytes	statfs("/"): (blocks − free) × block size, and blocks × block size
   net_in_bytes, net_out_bytes	the change since the last sample, for the interfaces of decision 1
   net_counters_reset	true when a counter goes back. Then send 0 for that minute.

3. Keep the last CPU and network counters in the state folder, with the boot id (/proc/sys/kernel/random/boot_id). After an agent restart, the next sample continues from them.

4. If the boot id changed, or the saved counters are older than 2 minutes, send 0 and net_counters_reset: true. Reason: the change then covers more than one minute.

5. Send the status in each report:

   Field	Source
   reboot_required	/var/run/reboot-required exists
   updates_total, updates_security	/usr/lib/update-notifier/apt-check, at start and then each hour (decision 2)
   os	PRETTY_NAME from /etc/os-release
   kernel	the kernel release (uname -r)
   uptime_seconds	/proc/uptime
   arch	the arch of the build (amd64 or arm64)

6. Always send the full status object. Reason: the control plane replaces the status. A field that is not sent becomes 0 or empty.

Decision necessary

Both decisions are gaps in the contract. After the decision, propose a contract change.

1. Which network interfaces does the agent count?
   • The contract does not say.
   • A test server has 27 interfaces: eth0 (public), eth1 (private network), lo, docker0, 5 Docker bridges and 17 veth interfaces.
   • Container traffic goes through a veth, a bridge and eth0. If the agent adds all interfaces, it counts the same traffic 2 or 3 times.
   • Recommendation: count the interfaces that have a hardware device (/sys/class/net/<name>/device exists). On the test server, these are eth0 and eth1. If no interface has a device, count the interface of the default route.
   • Other option: count only the interface of the default route. This ignores the private network.
2. What does the agent send when it cannot count the updates?
   • On the test server, apt-check shows 33;0 (33 updates, 0 security updates) in about 3 seconds. Thus, the agent must not run it each minute.
   • The control plane stores 0 when the agent does not send a count. Thus "not known" and "no updates" look the same.
   • Recommendation: run apt-check at start and then each hour, and always send the last counts. If apt-check is not on the server, propose a contract change that permits null ("not known").

Completion checks

• On a test server, memory and disk agree with free -b and df -B1 /.
• cpu_percent stays between 0 and 100.
• Container traffic is not counted two times. Compare with the counters of eth0.
• An agent restart makes no bandwidth spike.
• After a reboot, the first sample has net_counters_reset: true.
• The status has all fields, arch included.
• The agent measures when Docker is stopped.