Activity

1. JacobStephens2

   added

   enhancementNew feature or requestNew feature or request

   on Nov 10, 2026
2. JacobStephens2

   marked this as blocked on Nov 10, 2026

   • Scheduled Passes start Runs during an Experiment on the same host, so Trials time out under load #1494
3. JacobStephens2

   added

   ready-for-humanRequires human implementationRequires human implementation

   on Nov 10, 2026
4. JacobStephens2

   marked this as blocking on Nov 10, 2026

   • Release #1484 and build the first 20 discriminating process Eval tasks #1887
5. JacobStephens2 commented on Nov 12, 2026

   

   JacobStephens2

   on Nov 12, 2026

   OwnerAuthor

   More actions

   Provisioned and verified factory.etadventures.com (jstephens@factory / conductor@factory over Tailscale):

   • Droplet & backups: DigitalOcean g-8vcpu-32gb (8 dedicated vCPU / 32 GB, nyc1, droplet 618505797), Ubuntu 24.04.5 LTS upgraded and booted on 6.8.0-149-generic, weekly backups enabled (Sundays 04:00 UTC), adopted in Educational-Travel-Adventures/orchestration tofu/hosts/factory.tf (Run the Security review on security.harness when it is set #932), SSH port 22 restricted to Tailscale + orchestrate, and conductor account configured (Report a failure with the Session log of the session its cause came from #933, Release 0.13.2 #934).
   • thirdshift + Scheduled Passes start Runs during an Experiment on the same host, so Trials time out under load #1494 Pass skip: thirdshift 0.30.0 installed on both factory and the Lima VM. Staggered crontab installed for thirdshift, manager-tourbots, and tourbot. Verified live during a test Experiment that scheduled cron Passes skip with Activity log entries (2026-11-12 09:13:01 Pickup run skipped: Experiment 0511b98aa10f8d31cea5f9c2b32473828bb6fb66cc4c48fb3eba1ea72060c95a is live on this machine) and that a starting Experiment waits on live Runs unless --no-wait is given.
   • Console & Lima Fleet: thirdshift-console.service running behind Caddy HTTPS (https://factory.etadventures.com/) with sign-in configured; paired as factory in the Lima VM's [[console.fleet]] (read,security,act,run,admin, reachability: ok).
   • bwrap + systemd user-scope Trial & Grader: Verified AppArmor /etc/apparmor.d/bwrap, socat, [claude] setup_token, standalone codex binary (codex-code-mode-host), grok, muse, agy, kimi (kimi-k3.1-pro Fireworks config), and opencode. Ran a two-Arm smoke Experiment end-to-end inside bwrap and thirdshift-trial-*.scope user units (MemoryHigh=1G, MemoryMax=2G, MemorySwapMax=1G) through model Grading, then cleaned up the temporary smoke suite.
   • Mac evals backup procedure: Recorded on Release #1484 and build the first 20 discriminating process Eval tasks #1887 and here — after each approved batch of Eval tasks:

     mkdir -p ~/thirdshift-evals-backup
     rsync -av --exclude 'builds/' jstephens@factory:.thirdshift/evals/ ~/thirdshift-evals-backup/
6. JacobStephens2

   closed this as completedon Nov 12, 2026
7. JacobStephens2 commented on Nov 12, 2026

   

   JacobStephens2

   on Nov 12, 2026

   OwnerAuthor

   More actions

   Handoff: finish setting up the factory host

   The issue body holds the requirements and acceptance criteria, and docs/grill-with-docs/run-capacity.md (Q1.1-Q3.2) explains why. This comment records only what the setup session did and what is left.

   Already done (2026-11-12)

   Item	State
   Droplet	DigitalOcean (ETA account, by decision), g-8vcpu-32gb = 8 dedicated vCPU / 32 GB, $252/month from the sizes API, Ubuntu 24.04.5, region nyc1 (same as thirdshift.app, judged by latency)
   Backups	Weekly, kept 28 days
   Infrastructure as code	Managed in ETA's infrastructure repo, with the DNS record applied
   OS	Timezone America/New_York, firewall configured, lingering on for the working user so systemd user scopes survive logout
   bwrap	Ubuntu blocked unprivileged user namespaces. Fixed with the per-binary AppArmor grant that thirdshift's own Setup prints (/etc/apparmor.d/bwrap, from src/setup/sandbox.rs). Tested: bwrap --unshare-all works, and so does systemd-run --user --scope -p MemoryMax=1G
   Installed	thirdshift 0.29.2 in ~/.local/bin (shell installer, so thirdshift update works). It contains #1494's fix, PR #1982, merge bb84c3a; 0.29.1 does not. Also Claude Code (native installer), Codex (npm, global), rustup stable with clippy and rustfmt, gh, Caddy (apt, not configured yet), Node 22, git, jq, build-essential, tmux
   git identity	Set to match the thirdshift repo's recent commits
   ~/.thirdshift/	Only logs/ exists. There is no config.toml yet. ~/repos/ is empty

   Still to do, in order

   1. Security updates: sudo apt update && sudo apt upgrade. Do not run do-release-upgrade: this issue specifies 24.04, and that is where Trials are proven.

   2. Logins (these need the human): claude, codex login, gh auth login as the same GitHub account the other factory hosts use, and the Resend key (thirdshift setup covers the email settings).

   3. User config: run thirdshift setup, or start from an existing host's ~/.thirdshift/config.toml and change:

      • [console] address to the factory host's name
      • other_accounts = "none"
      • fleet = []
      • [email] from to a factory-specific sender

      An existing config may also set [harness] default = "codex", a [security] harness, [trial] memory caps of 1G/2G/1G, [resources] thresholds, and a raised [pickup] failure_limit = 6. That last one is a temporary measure for A Triage of an existing Security finding fails: security-finding is rejected as its category #2018, so decide whether it belongs here.

   4. Console behind sign-in (the guide's "Console on a server"):

      thirdshift console password --email <sign-in email>
      thirdshift console install --output install-console.sh   # read it first
      sudo sh install-console.sh

      The script waits until the host name resolves. One of DigitalOcean's resolvers briefly cached NXDOMAIN, so if the DNS check fails, run it again.

   5. Fleet: on factory, thirdshift console token create lima --scopes read,act (or whatever scopes the Lima Console needs). Then on the Lima VM, thirdshift console fleet add factory <url> and paste the token.

   6. Cron (the human decides first): which repos does factory pick up? Grill Q1.1(c) says it takes Pickup work. But personal repos (thirdshift itself) are currently kept off ETA hosts because of a fleet GitHub rate limit, and factory is on ETA's account. Settle that, clone the chosen repos into ~/repos, then write the crontab, copying the stagger pattern from the other hosts. Every Pass will skip while an Experiment is live, which is Scheduled Passes start Runs during an Experiment on the same host, so Trials time out under load #1494's behaviour in 0.29.2.

   7. Acceptance checks (from this issue):

      • Start a test Experiment and confirm a scheduled Pass skips with an Activity log line.
      • Run one Eval task's single Trial in bwrap with a systemd scope.
      • Confirm the Console is reachable with sign-in and appears in the Lima Console's Fleet overview.
      • Write down the ~/.thirdshift/evals/ to Mac copy (for example rsync -a factory:.thirdshift/evals/ ~/thirdshift-evals-backup/ after each approved batch), in Release #1484 and build the first 20 discriminating process Eval tasks #1887 step 4 or in this issue's closing comment.

   Things to know

   • Resizing: CPU and RAM resize in place, but disk only grows. Resize without growing the disk so the droplet can come back down after a large Experiment (grill Q1.4).
   • The droplet's size is set in code, so change it there rather than in the DigitalOcean UI.