Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -334,7 +334,7 @@ why it is in that order.
The current top items: rehearse the firewall restore on the ProDesk bought on
2026-09-08, and then build the sensitive tier on that same box ([#404](https://github.com/Gerrrt/HomeLab/issues/404), [ADR-0034](adr/0034-run-the-sensitive-tier-on-the-prodesk-and-make-it-the-spare-hardware.md)). **Every purchase still outstanding is in one place**, the
roadmap's [*Everything still to buy*](docs/roadmap.md#everything-still-to-buy):
three items now, one later, and a rule that nothing joins them without a
six items now, one later, and a rule that nothing joins them without a
decision. This sentence used to carry the list itself, name three purchases
coupled to the UPS work and omit the tier's host entirely, which is how one
ProDesk came to be bought for two jobs.
Expand Down
2 changes: 1 addition & 1 deletion docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -189,7 +189,7 @@ hole from the monitoring VLAN into the monitored one.
| `alexander` (10.0.30.40) | 🟢 30 | [`stacks/lab`](../stacks/lab) | Prometheus, Loki, Grafana, Alloy — the lab's own observability, which never remote-writes to VLAN 99 ([ADR-0007](adr/0007-defensive-estate-and-offensive-range.md), [ADR-0020](adr/0020-run-the-lab-stack-in-a-guest-with-its-own-prometheus.md)). A guest on `Saruman`, not the hypervisor; Alloy agent (Docker) |
| `odin` (10.0.30.60) | 🟢 30 | [`stacks/soc`](../stacks/soc) | **Not built yet** — Wazuh (indexer, manager, dashboard), Velociraptor and Alloy: the security half of ADR-0007, placed by [ADR-0030](adr/0030-give-the-security-tooling-its-own-guest-and-its-own-stack.md) on a second guest on `Saruman` because `alexander`'s 8 GiB cannot hold both. The stack is authored and CI-validated ahead of the guest, the way `stacks/lab` was ahead of `alexander`; the build is [`build-the-soc-guest.md`](runbooks/build-the-soc-guest.md), behind [#414](https://github.com/Gerrrt/HomeLab/issues/414). The six domain machines report to it as agents; its Alloy pushes to `alexander`, never to VLAN 99, and the indexer's health is the one series that crosses into the lab's Prometheus. Alloy agent (Docker) |
| `oracle` (10.0.99.30) | 🔴 99 | *(none — hand-run containers)* | The Lemmiwinks wiki and its Postgres, since 2025-11-12 ([ADR-0011](adr/0011-keep-the-wiki-internal.md)); Alloy agent (Docker, `scripts/deploy-agent.sh`); the off-host copy of the firewall export (`make backup-firewall`). The estate's host for small off-host jobs — [ADR-0015](adr/0015-give-oracle-the-off-host-jobs.md) |
| `trinity` (10.0.99.40) | 🔴 99 | [`stacks/sensitive`](../stacks/sensitive) | **Not built yet** — ADR-0008's sensitive tier on the ProDesk 600 G4 of [ADR-0034](adr/0034-run-the-sensitive-tier-on-the-prodesk-and-make-it-the-spare-hardware.md), after the firewall restore is rehearsed on it ([#404](https://github.com/Gerrrt/HomeLab/issues/404)). The foundation is authored: Caddy as the published HTTPS port and step-ca as an intermediate beneath the lab CA ([#129](https://github.com/Gerrrt/HomeLab/issues/129), [#130](https://github.com/Gerrrt/HomeLab/issues/130)), with AdGuard Home behind Caddy and publishing 53 to the firewall's forwarder alone ([#135](https://github.com/Gerrrt/HomeLab/issues/135), [ADR-0010](adr/0010-keep-the-resolver-on-the-gateway.md)); Home Assistant ([#134](https://github.com/Gerrrt/HomeLab/issues/134)), Immich — four containers behind Caddy with a memory limit on each ([#132](https://github.com/Gerrrt/HomeLab/issues/132)) — Paperless-ngx with a Postgres and a Valkey of its own ([#133](https://github.com/Gerrrt/HomeLab/issues/133)) and Vaultwarden ([#131](https://github.com/Gerrrt/HomeLab/issues/131)) are authored as well; the rest follow. Alloy agent (Docker, `scripts/deploy-agent.sh`), pushing to `prometheus` like `oracle`'s |
| `trinity` (10.0.99.40) | 🔴 99 | [`stacks/sensitive`](../stacks/sensitive) | **Not built yet** — ADR-0008's sensitive tier on the ProDesk 600 G4 of [ADR-0034](adr/0034-run-the-sensitive-tier-on-the-prodesk-and-make-it-the-spare-hardware.md), after the firewall restore is rehearsed on it ([#404](https://github.com/Gerrrt/HomeLab/issues/404)). The foundation is authored: Caddy as the published HTTPS port and step-ca issuing beneath the tier's own root rather than the estate's, which is left untouched ([#129](https://github.com/Gerrrt/HomeLab/issues/129), [#130](https://github.com/Gerrrt/HomeLab/issues/130), [ADR-0037](adr/0037-give-the-sensitive-tier-its-own-root-and-issue-beneath-it-over-acme.md)), with AdGuard Home behind Caddy and publishing 53 to the firewall's forwarder alone ([#135](https://github.com/Gerrrt/HomeLab/issues/135), [ADR-0010](adr/0010-keep-the-resolver-on-the-gateway.md)); Home Assistant ([#134](https://github.com/Gerrrt/HomeLab/issues/134)), Immich — four containers behind Caddy with a memory limit on each ([#132](https://github.com/Gerrrt/HomeLab/issues/132)) — Paperless-ngx with a Postgres and a Valkey of its own ([#133](https://github.com/Gerrrt/HomeLab/issues/133)) and Vaultwarden ([#131](https://github.com/Gerrrt/HomeLab/issues/131)) are authored as well; the rest follow. Alloy agent (Docker, `scripts/deploy-agent.sh`), pushing to `prometheus` like `oracle`'s |
| `bahamut` (10.0.30.50) | 🟢 30 | *(none — Windows)* | **Not built yet** — Windows Server 2025 domain controller, PDC emulator and DNS for `ad.matrix.elysium` — Tier 0. Static, because every member finds a DC through DNS and the DCs *are* the DNS. Scraped by `alexander` on `9182`; it pushes nothing, and runs no Alloy ([ADR-0029](adr/0029-size-the-lab-domain-and-separate-its-namespace-and-clock.md)) |
| `leviathan` (10.0.30.51) | 🟢 30 | *(none — Windows)* | **Not built yet** — Windows Server 2025 second domain controller and DNS — Tier 0. Static, for the same reason. Scraped by `alexander` on `9182`; it pushes nothing, and runs no Alloy ([ADR-0029](adr/0029-size-the-lab-domain-and-separate-its-namespace-and-clock.md)) |
| `titan` (10.0.30.52) | 🟢 30 | *(none — Windows)* | **Not built yet** — Windows Server 2025 file and member server — the shares, and the NTLM relay target that only exists because 2025 requires outbound SMB signing and not inbound — Tier 1. Scraped by `alexander` on `9182`; it pushes nothing, and runs no Alloy ([ADR-0029](adr/0029-size-the-lab-domain-and-separate-its-namespace-and-clock.md)) |
Expand Down
25 changes: 24 additions & 1 deletion docs/hardware.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,30 @@ revisions of this repository treated `shiva` as the hypervisor itself.
tracked under [#413](https://github.com/Gerrrt/HomeLab/issues/413). A tower,
not a rack unit, and a 73 W part where the ADRs pictured an N100; it enters
the Compute table when it is racked — or rather placed — addressed and in
`network.md`. Two drives for the mirror are still to buy.
`network.md`. The two drives for the mirror were bought 2026-09-11 and are
the entry below, as is the boot disk ADR-0016's Ubuntu Server install
wants. Only a bracket to carry that disk in the optical bay is still to
buy, on the roadmap's [list](roadmap.md#everything-still-to-buy).
- 2× Seagate Exos X20 18 TB (`ST18000NM003D`), 3.5" SATA — bought
2026-09-11, in transit. `zion`'s ZFS mirror
([ADR-0016](adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md),
[#413](https://github.com/Gerrrt/HomeLab/issues/413)). A mirror of two is
one drive's capacity, so this is 18 TB usable, not 36. The listing's
**zero power-on hours is a claim, not a fact** — read it back with
`smartctl -a` on arrival and record what the drives actually report here,
before the mirror is built on them. Serials go here when they land. They
enter the Compute table with the NAS, which is not built.
- Intel DC S3520 240 GB, 2.5" SATA 6 Gb/s enterprise SSD with power-loss
protection — bought 2026-09-11, in transit. `zion`'s boot disk, carrying
ADR-0016's Ubuntu Server and its one compose stack, deliberately not the
mirror. A data-centre part where the roadmap asked only for "a 240–256 GB
2.5" SATA SSD": the endurance is beside the point for a boot disk, but
the power-loss protection is the same property the SM863a pair was bought
for, and a boot disk that survives a power cut is worth more here than one
that is merely fast. **It has no bracket** — the TS150's optical bay is
5.25" and this is a 2.5" drive, so an adapter is outstanding and the drive
cannot be fitted without it. Model, part number and serial go here when it
lands, with SMART read back before the install.
- 2× Samsung SM863a 960 GB (`MZ-7KM960N`), 2.5" SATA 6 Gb/s enterprise
SSDs with power-loss protection[^SM863a] — purchased 2026-09-09, in transit,
for the ProLiant's SFF bays. Bought against the number every sizing
Expand Down
16 changes: 10 additions & 6 deletions docs/roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,8 +34,7 @@ battery it was never written about.

| Item | For | Decided by | When it is needed |
| --- | --- | --- | --- |
| Two 3.5" NAS drives, capacity chosen at the till | `zion`'s ZFS mirror | [ADR-0016](adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md), [#413](https://github.com/Gerrrt/HomeLab/issues/413) | Before the NAS is built |
| A 240–256 GB 2.5" SATA SSD, and a bracket for the optical bay | `zion`'s boot disk. ADR-0016 chose Ubuntu Server and one compose stack rather than an appliance, so the OS wants a disk that is not the mirror — and that never got written down | [ADR-0016](adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md), [#413](https://github.com/Gerrrt/HomeLab/issues/413) | With the drives |
| A bracket to carry a 2.5" SSD in the optical bay | Mounting `zion`'s boot disk. ADR-0016 chose Ubuntu Server and one compose stack rather than an appliance, so the OS wants a disk that is not the mirror — and that never got written down. The SSD itself was bought 2026-09-11; this row is what is left of it | [ADR-0016](adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md), [#413](https://github.com/Gerrrt/HomeLab/issues/413) | With the drives |
| One Intel I226 2.5 GbE card on an M.2 B+M-key adapter — the part `morpheus` has, not a USB NIC | The ProDesk's second port, so a restore onto it comes up as `igc0` and asks nothing; for the rehearsal and any restore after it | [`restore-the-firewall.md`](runbooks/restore-the-firewall.md) §3, [#404](https://github.com/Gerrrt/HomeLab/issues/404) | Before the rehearsal |
| Two Windows 11 Pro keys | The lab domain's two endpoints; the four servers are free evaluations | [ADR-0029](adr/0029-size-the-lab-domain-and-separate-its-namespace-and-clock.md), [#414](https://github.com/Gerrrt/HomeLab/issues/414) | When the domain build reaches the endpoints, not before |
| A managed switch with a TLS management interface | Replacing `neo`, whose firmware will not persist a community deletion ([#84](https://github.com/Gerrrt/HomeLab/issues/84)) and which serves its admin UI over plain HTTP — through the device the password protects, and that credential is read-write. **SNMPv3 is no longer the argument**: [ADR-0036](adr/0036-poll-the-ilo-and-the-ups-card-over-snmpv3-and-keep-the-firewall-on-bsnmpd.md) found the switch's agent answers v3 on the wire and was never what blocked #85, so TLS management is what a replacement is actually bought for | [ADR-0018](adr/0018-name-the-switch-and-leave-its-ui-on-plain-http.md), [#444](https://github.com/Gerrrt/HomeLab/issues/444) | Whenever a cabling window suits — it is independent of everything else here |
Expand All @@ -59,7 +58,10 @@ capacity are still chosen at the till and recorded in
before it is bought.

**Already paid for**, in transit or on hand: the ProDesk 600 G4 (the tier's
host and the firewall's spare hardware); the TS150 NAS; two SM863a SSDs for
host and the firewall's spare hardware); the TS150 NAS and the two Exos X20
18 TB drives for its mirror, and its boot SSD
([#413](https://github.com/Gerrrt/HomeLab/issues/413));
two SM863a SSDs for
`Saruman` ([#418](https://github.com/Gerrrt/HomeLab/issues/418)); the pfSense
installer stick; the 2 TB USB drive that becomes the photo library's disk on
`trinity`; the removable medium the second age recipient lives on.
Expand Down Expand Up @@ -632,8 +634,10 @@ what left this one unfireable for months.
73 W desktop Xeon in a tower against a 6 W part in a shoebox, so it draws and
makes more, and it will not rack. What it has that an N100 box does not is
ECC memory under a ZFS mirror, four real bays, and a Kaby Lake iGPU whose
Quick Sync is what #138's transcoding needs. The drives are the outstanding
purchase — chosen at the till on cost per terabyte, per the ADR — and the
Quick Sync is what #138's transcoding needs. The drives were bought on
2026-09-11 — two Seagate Exos X20 18 TB, chosen on cost per terabyte per
the ADR — and the boot disk with them, an Intel DC S3520 240 GB,
leaving only a bracket to carry it in the optical bay. The
OS is decided: Ubuntu Server and one compose stack, not an appliance. Nothing
is configured.
Reading the enforced ruleset first changed two of the answers. **50→40 is not
Expand Down Expand Up @@ -741,7 +745,7 @@ what left this one unfireable for months.
so PBS would add a service for a capability that exists. `zion` is bought
since 2026-09-09 — the TS150 of
[#413](https://github.com/Gerrrt/HomeLab/issues/413), in transit, its two
drives still to buy — and not built, so the deferral's trigger has moved from
drives bought on 2026-09-11 — and not built, so the deferral's trigger has moved from
a purchase to a build: PBS follows the NAS answering on `10.0.40.30`, not the
box arriving. The lab has **revert and not backup** until then, and the ADR
names what gets backed up when it arrives — which is also why ADR-0029 gives PBS no disk
Expand Down
72 changes: 71 additions & 1 deletion scripts/check_docs.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
The device list must live in exactly one place. It is currently spread
across five ... and --check asserts the other copies still agree.

Nine assertions, each comparing prose against something machine-readable:
Ten assertions, each comparing prose against something machine-readable:

1. Counted claims rules, unit-test coverage, dashboards, panels,
Alloy agents, ADRs, runbooks
Expand All @@ -37,6 +37,10 @@
9. Guest claims a host that says it runs no guests, while another
row says it is a guest on it. Checkable only against
its siblings, which is why it is its own assertion.
10. Outstanding buys README.md <-> the buy table in docs/roadmap.md. The
roadmap is a source here, not a target: that table
is the one place a purchase may enter or leave, so
it is what README's count answers to.

Only present-tense documents are checked. `docs/roadmap.md` and `docs/adr/`
record what was true when the work landed — `roadmap.md` still says "(34 rules)"
Expand Down Expand Up @@ -84,6 +88,7 @@
NETWORK_MD = REPO / "docs/network.md"
ARCH_MD = REPO / "docs/architecture.md"
HARDWARE_MD = REPO / "docs/hardware.md"
ROADMAP_MD = REPO / "docs/roadmap.md"

# Present-tense documents. See the module docstring for why roadmap.md and
# adr/ are deliberately absent — they are records, not claims about now.
Expand Down Expand Up @@ -1175,6 +1180,69 @@ def named(prefix: str) -> set[str] | None:
return problems


def check_buy_list() -> list[str]:
"""README's count of outstanding purchases against the roadmap's table.

`roadmap.md` is a record and not checked as prose — see the module
docstring. This assertion reads it as a *source* instead: the buy table is
the one place a purchase is allowed to enter or leave, by that section's
own rule, so it is the only thing README's count can honestly be checked
against.

It exists because the count went stale inside a day. The reconciliation on
2026-09-10 grew the table from three rows to seven and left README saying
"three items now", in the sentence that exists *because* a purchase had
been missed once already. Every other count in this repository is asserted
here; this one was prose on both ends, so nothing caught it.

The "one later" half is checked too. `ifrit` is a paragraph rather than a
row, and a second such paragraph would make the word wrong in exactly the
same silent way.
"""
problems: list[str] = []
roadmap = ROADMAP_MD.read_text()

tables = tables_under(roadmap, re.compile(r"^## Everything still to buy\b"))
if not tables:
return [
"docs/roadmap.md has no table under 'Everything still to buy' — "
"README's count of outstanding purchases has nothing to check "
"against"
]

# The section's first table is the "Buy these" list; rows[0] is its header.
rows_now = len(tables[0]) - 1
rows_later = len(re.findall(r"\*\*One more, later", roadmap))

num = "|".join(["[0-9]+", *NUMBER_WORDS])
pattern = re.compile(
rf"({num}){WS}items?{WS}now,{WS}({num}){WS}later", re.IGNORECASE
)
text = (REPO / "README.md").read_text()
match = pattern.search(text)
if not match:
return [
"README.md no longer says 'N items now, M later' of the roadmap's "
"outstanding purchases — the claim moved or was reworded, and this "
"assertion cannot follow it. Update the pattern or drop the check "
"deliberately"
]

line = text[: match.start()].count("\n") + 1
said_now, said_later = number(match.group(1)), number(match.group(2))
if said_now != rows_now:
problems.append(
f"README.md:{line} says {match.group(1)} outstanding purchase(s), "
f"but docs/roadmap.md's buy table has {rows_now} row(s)"
)
if said_later != rows_later:
problems.append(
f"README.md:{line} says {match.group(2)} purchase(s) later, but "
f"docs/roadmap.md has {rows_later} 'One more, later' paragraph(s)"
)
return problems


def main() -> int:
f = facts()
checks = (
Expand All @@ -1188,6 +1256,8 @@ def main() -> int:
("firewall posture prose against docs/firewall-claims.yaml",
check_firewall_posture),
("guest claims against each other", check_guest_claims),
("README's outstanding-purchase count against the roadmap's table",
check_buy_list),
)

total = 0
Expand Down
2 changes: 1 addition & 1 deletion stacks/sensitive/Caddyfile
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@
auto_https disable_redirects

# Every certificate this file serves comes from step-ca over ACME, from
# the tier's own intermediate (ADR-0035, #130). `step-ca` resolves on the
# the tier's own intermediate (ADR-0037, #130). `step-ca` resolves on the
# compose network; the directory URL's middle segment is the provisioner
# name scripts/tier-ca.sh creates. trusted_roots is the TIER'S root — the
# file compose.yaml mounts from certificates/tier-ca.pem — not the estate's
Expand Down