Repository navigation
Agent skills for running the end-to-end paper experiments #917
Description
Goal
Any agent (Claude Code, Codex, ...) can rerun the paper's end-to-end experiments from the repo alone. Today the procedure lives in a handoff document and in scripts under the gitignored experiment_outputs/.
Layout
Follow the existing pattern: the canonical text in .agents/skills/<name>.md (linked from AGENTS.md), plus a thin .claude/skills/<name>/SKILL.md wrapper whose description lets Claude Code load it automatically.
Skills
| Skill | Covers |
|---|---|
e2e-cluster-setup |
CloudLab provisioning: containerd on /scratch, Prometheus 2.53.2 binary, slot = 2 nodes, SSH / host-suffix conventions |
random-workload-e2e |
random_workloads.py --dashboard → Hydra configs and variants → random_workload_batch.py (batch, slots, arms, resume, reruns) → copy feeder stats → validity check → collate_batch.py and plots |
trace-replay |
Alibaba / Google replay tables (make_replay.py, make_google_replay.py), value scaling (#916), --config-variant overlays |
baseline-promsketch |
Installing the PromSketch fork and its rules on a node, restoring Prometheus afterwards |
baseline-pilotdb |
The one-process PilotDB harness, exact recomputation, CPU monitor |
downstream-analysis |
Alerting F1 and Prophet agreement: per-system reference, bounded agreement metric |
cost-model |
Measured resources → instance counts and dollars at a target scale |
Each skill gives:
- exact commands and their parameters;
- validity checks the agent must run (ingest lag ≤ 10 s; stopped runs have empty latency files, so parse
prometheus_client.log; missing feeder stats read as lag 0); - known failure symptoms (
remote_writeurlhttp://None:hangs the client; ClickHouse+MV lag at K ≥ 4; Hydra# @packageheader lost when YAML is rewritten); - where outputs go and how runs are named (
s862dashboard_wNN_<arm>); - only repo-relative paths.
Prerequisites
- Move the experiment scripts into git (e.g.
asap-tools/experiments/paper/): the PromSketch / PilotDB launch harnesses (psk-pilot-goog-all/,pilotdb-goog/,promsketch-prom/node/) and the analysis scripts (paper-analysis/:downstream-apps,cost-model,motivation-regen,cpu-memory). Replace hardcoded/home/...paths with arguments. - Experiment configs are gitignored (
asap-tools/experiments/config/experiment_type/*). Either each skill regenerates them with a command, or the random-workload config sets are tracked. - The batch driver copies feeder stats and checks validity itself, so the skill doesn't have to script it.
Done when
An agent with only the repo and cluster access reruns one workload (e.g. Google w01, all arms) and its downstream analysis by following the skills, with no other instructions.
Activity