Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 16 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,13 +68,19 @@ documents for different readers.
`promtool`, `amtool`, `alloy fmt`, a real Loki boot to parse the LogQL rules,
dashboard-JSON and datasource checks, every dashboard's PromQL parsed, plus
`gitleaks` over the full history.
- **CI that validates the documentation too.** Six assertions cross-check
this prose against the configs it describes — rule and panel counts, the
SNMP inventory against `docs/network.md`, the host/stack and ports tables
against `compose.yaml`, and a ban on image versions in prose — Dependabot
edits only `compose.yaml`, so a version written anywhere else is stale
from the next bump. A document that disagrees with the repository fails
the build.
- **CI that validates the documentation too.** Ten assertions cross-check
this prose against the configs it describes — counted claims (rules,
dashboards, panels, Alloy agents, VLANs, ADRs, runbooks), the SNMP
inventory and the compute table against `docs/network.md`, the host/stack
and ports tables against `compose.yaml`, ADR numbering, firewall posture
against `docs/firewall-claims.yaml`, guest rows against each other, this
file's outstanding-purchase count against the roadmap's buy table, and a
ban on image versions in prose — Dependabot edits only `compose.yaml`, so
a version written anywhere else is stale from the next bump. A document
that disagrees with the repository fails the build. That opening count is
now one of the claims, read from the check registry rather than kept by
hand: it said six while ten ran, and the list beside it had been overtaken
by four — drift in the sentence advertising that drift gets caught.
- **Supply chain pinned by digest.** Every image carries both a tag and a
`sha256:` digest, so a moved tag cannot change what deploys. CI enforces it;
`make pin-digests` re-resolves them from the registry. Every `docker run` in
Expand Down Expand Up @@ -332,7 +338,9 @@ Open work is tracked in
why it is in that order.

The current top items: rehearse the firewall restore on the ProDesk bought on
2026-09-08, and then build the sensitive tier on that same box ([#404](https://github.com/Gerrrt/HomeLab/issues/404), [ADR-0034](adr/0034-run-the-sensitive-tier-on-the-prodesk-and-make-it-the-spare-hardware.md)). **Every purchase still outstanding is in one place**, the
2026-09-08 — on hand since 2026-09-15, so what the rehearsal waits on is the
I226 card and the installer stick and no longer the box — and then build the
sensitive tier on that same box ([#404](https://github.com/Gerrrt/HomeLab/issues/404), [ADR-0034](adr/0034-run-the-sensitive-tier-on-the-prodesk-and-make-it-the-spare-hardware.md)). **Every purchase still outstanding is in one place**, the
roadmap's [*Everything still to buy*](docs/roadmap.md#everything-still-to-buy):
two items now, one later, and a rule that nothing joins them without a
decision. This sentence used to carry the list itself, name three purchases
Expand Down
63 changes: 43 additions & 20 deletions docs/hardware.md
Original file line number Diff line number Diff line change
Expand Up @@ -102,8 +102,9 @@ revisions of this repository treated `shiva` as the hypervisor itself.
2026-09-08, carrying the unmanaged switch that feeds `prometheus` and
`oracle` ([#110](https://github.com/Gerrrt/HomeLab/issues/110))
- HP ProDesk 600 G4 Micro — i5-8500T, 32 GB, 512 GB SSD, the same model as
`morpheus` — ordered 2026-09-08, in transit. This is `trinity`, and it is
**one box with two jobs, not two boxes**: the sensitive tier's host, and
`morpheus` — ordered 2026-09-08, **in hand since 2026-09-15** and not yet
opened. This is `trinity`, and it is **one box with two jobs, not two
boxes**: the sensitive tier's host, and
the firewall's spare hardware in a disaster
([ADR-0034](adr/0034-run-the-sensitive-tier-on-the-prodesk-and-make-it-the-spare-hardware.md)).
The documents call it "the tier's host" in one place and "the spare" in
Expand All @@ -113,7 +114,13 @@ revisions of this repository treated `shiva` as the hypervisor itself.
[#404](https://github.com/Gerrrt/HomeLab/issues/404) builds it, after the
firewall restore has been rehearsed on it. It ships with the onboard NIC
only; the I226 card the restore depends on was a separate purchase, made
2026-09-11 and the entry below.
2026-09-11 and the entry below — which has **not** landed. So what the
rehearsal waits on is no longer this box: it is that card and the installer
stick. **Checked on arrival, not assumed:** the i5-8500T, the 32 GB and the
512 GB above are the listing's claims and not yet the machine's; the serial
and the product number off the case; and the question the entry below turns
on — whether that 512 GB SSD is an M.2 stick or a 2.5" drive in a carrier.
Those go here when it is opened.
- Intel I226-V 2.5 GbE card on an M.2 B+M-key adapter — bought 2026-09-11,
in transit. The second port on the ProDesk Micro above, matching the card
in `morpheus` so a pfSense restore onto the spare brings the LAN up as
Expand All @@ -125,8 +132,11 @@ revisions of this repository treated `shiva` as the hypervisor itself.
adapter comes up as `ure0`, which is the one thing a restore must not be
asked about. It goes in the G4 Micro's second M.2 slot — **and if that
machine's 512 GB SSD turns out to be a 2.5" drive rather than M.2, the
drive carrier and this card contend for the same space.** Checked on
arrival, not assumed. Part number and the port's MAC go here when it lands.
drive carrier and this card contend for the same space.** That question
stopped waiting on this card when the G4 Micro landed on 2026-09-15:
opening the box answers it, and answering it before the card arrives is
the difference between a return and a part that fits nothing. Part number
and the port's MAC go here when it lands.
- 2 TB USB portable hard drive — on hand, previously a games console's
storage. Becomes the photo library's disk on `trinity`
([#404](https://github.com/Gerrrt/HomeLab/issues/404)): Immich's originals
Expand All @@ -137,17 +147,23 @@ revisions of this repository treated `shiva` as the hypervisor itself.
drive, which this replaces.
- Lenovo ThinkServer TS150 — Xeon E3-1225 v6 (4 cores, 3.3 GHz, Intel HD P630
with Quick Sync), 8 GB ECC, four 3.5" bays, no drives, no OS — bought
2026-09-09, in transit. The NAS `smaug` of
2026-09-09, **in hand since 2026-09-15**. The NAS `smaug` of
[ADR-0016](adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md),
tracked under [#413](https://github.com/Gerrrt/HomeLab/issues/413). A tower,
not a rack unit, and a 73 W part where the ADRs pictured an N100; it enters
the Compute table when it is racked — or rather placed — addressed and in
`network.md`. The two drives for the mirror were bought 2026-09-11 and are
the entry below, as are the boot disk ADR-0016's Ubuntu Server install
wants and the bracket that carries it in the optical bay. Nothing for this
machine is outstanding on the roadmap's
[list](roadmap.md#everything-still-to-buy) any more; what is left is a
build.
`network.md`. The boot disk ADR-0016's Ubuntu Server install wants and the
bracket that carries it in the optical bay are the entries below and landed
with it; the two drives for the mirror, bought 2026-09-11, have not. Nothing
for this machine is outstanding on the roadmap's
[list](roadmap.md#everything-still-to-buy) any more; what is left is those
two drives landing, and a build.
**Checked on arrival, not assumed:** the Xeon E3-1225 v6 and the 8 GB of ECC
above came from a listing and not from the machine, so read them off POST or
the BIOS; the machine type-model and the serial off the chassis label; how
many of the four bays came with a caddy, because a bay without one holds no
drive; and that the 5.25" optical bay is empty, which the boot disk's bracket
assumes. Those go here when it is opened.
- 2× Seagate Exos X20 18 TB (`ST18000NM003D`), 3.5" SATA — bought
2026-09-11, in transit. `smaug`'s ZFS mirror
([ADR-0016](adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md),
Expand All @@ -158,18 +174,25 @@ revisions of this repository treated `shiva` as the hypervisor itself.
before the mirror is built on them. Serials go here when they land. They
enter the Compute table with the NAS, which is not built.
- Intel DC S3520 240 GB, 2.5" SATA 6 Gb/s enterprise SSD with power-loss
protection — bought 2026-09-11, in transit. `smaug`'s boot disk, carrying
ADR-0016's Ubuntu Server and its one compose stack, deliberately not the
mirror. A data-centre part where the roadmap asked only for "a 240–256 GB
2.5" SATA SSD": the endurance is beside the point for a boot disk, but
protection — bought 2026-09-11, **in hand since 2026-09-15**. `smaug`'s boot
disk, carrying ADR-0016's Ubuntu Server and its one compose stack,
deliberately not the mirror. A data-centre part where the roadmap asked only
for "a 240–256 GB 2.5" SATA SSD": the endurance is beside the point for a
boot disk, but
the power-loss protection is the same property the SM863a pair was bought
for, and a boot disk that survives a power cut is worth more here than one
that is merely fast. The TS150's optical bay is 5.25" and this is a 2.5"
drive, so it cannot be fitted bare: **a bracket for the bay and
double-sided tape to mount it were bought 2026-09-11**, alongside the drive.
Neither is named here yet, for the same reason the drive's serial is not —
nothing is recorded before it is in hand. Model, part number and serial go
here when it lands, with SMART read back before the install.
double-sided tape to mount it were bought 2026-09-11**, alongside the drive,
and both landed with it on 2026-09-15. Neither is named here yet — that was
deferred because nothing is recorded before it is in hand, and they now are,
so it is a naming this document owes rather than one it is waiting on.
**Checked on arrival, not assumed:** the drive's model, part number and
serial, and `smartctl -a` read back **before** the install rather than after
— the same caution the Exos pair below is bought under, and for the same
reason: a listing's hours are a claim. That the bracket is a 5.25"-to-2.5"
adapter and not the 3.5" version, which would fit neither the drive nor the
bay, goes here with them.
- 2× Samsung SM863a 960 GB (`MZ-7KM960N`), 2.5" SATA 6 Gb/s enterprise
SSDs with power-loss protection[^SM863a] — purchased 2026-09-09, in transit,
for the ProLiant's SFF bays. Bought against the number every sizing
Expand Down
55 changes: 38 additions & 17 deletions docs/roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,22 +89,38 @@ capacity are still chosen at the till and recorded in
[`hardware.md`](hardware.md) afterwards, per the ADR — nothing is named here
before it is bought.

**Already paid for**, in transit or on hand: the ProDesk 600 G4 (the tier's
host and the firewall's spare hardware), and the I226 card on its M.2 adapter
that a restore onto it needs
([#404](https://github.com/Gerrrt/HomeLab/issues/404)); the CRS326 that
replaces `neo` ([#444](https://github.com/Gerrrt/HomeLab/issues/444)); the
TS150 NAS, the two
Exos X20 18 TB drives for its mirror, and its boot SSD with the bracket and
tape that mount it in the optical bay
([#413](https://github.com/Gerrrt/HomeLab/issues/413));
two SM863a SSDs for
`Saruman` ([#418](https://github.com/Gerrrt/HomeLab/issues/418)); the pfSense
installer stick; the 2 TB USB drive that becomes the photo library's disk on
`trinity`; the removable medium the second age recipient lives on; and the A1437 cell
**Already paid for and on hand**, as of 2026-09-15: the ProDesk 600 G4 —
`trinity`, the tier's host and the firewall's spare hardware
([#404](https://github.com/Gerrrt/HomeLab/issues/404)); the TS150 NAS and its
boot SSD with the bracket and tape that mount it in the optical bay
([#413](https://github.com/Gerrrt/HomeLab/issues/413)); the 2 TB USB drive
that becomes the photo library's disk on `trinity`; and the removable medium
the second age recipient lives on.

**Already paid for and still moving**: the I226 card on its M.2 adapter that a
restore onto `trinity` needs
([#404](https://github.com/Gerrrt/HomeLab/issues/404)) and the pfSense
installer stick — between them, the whole of what the rehearsal now waits on;
the two Exos X20 18 TB drives for `smaug`'s mirror
([#413](https://github.com/Gerrrt/HomeLab/issues/413)); the CRS326 that
replaces `neo` ([#444](https://github.com/Gerrrt/HomeLab/issues/444)); two
SM863a SSDs for `Saruman`
([#418](https://github.com/Gerrrt/HomeLab/issues/418)); and the A1437 cell
that replaces `prometheus`'s thirteen-year-old one
([#454](https://github.com/Gerrrt/HomeLab/issues/454)).

Four things crossed from the second paragraph to the first on **2026-09-15**:
the TS150, the ProDesk, the boot SSD, and the bracket and tape that mount it.
**No row on the table above moves** — arriving is not buying, and this section
tracks money. Nothing is racked, fitted or built by it either. What it changes
is one sentence: the two builds are no longer waiting on a van for their
machines, only for the parts named above. And what those boxes actually are is
still unread — serials, the spec off POST rather than off a listing, SMART
before an install, and whether `trinity`'s 512 GB SSD is an M.2 stick or a
2.5" drive that would contend with the I226 card for the same slot.
[`hardware.md`](hardware.md) records each of those as owed rather than
assuming the listing was right.

**Only if a decision is taken, and none is pending** — these are not on the
list, and each names what would put it there:

Expand Down Expand Up @@ -687,8 +703,12 @@ what left this one unfireable for months.
ECC memory under a ZFS mirror, four real bays, and a Kaby Lake iGPU whose
Quick Sync is what #138's transcoding needs. The drives were bought on
2026-09-11 — two Seagate Exos X20 18 TB, chosen on cost per terabyte per
the ADR — and the boot disk with them, an Intel DC S3520 240 GB,
leaving only a bracket to carry it in the optical bay. The
the ADR — and the boot disk with them, an Intel DC S3520 240 GB, with the
bracket and the tape that carry it in the optical bay bought the same day.
This paragraph said a bracket was the only thing left to buy, which had not
been true since 2026-09-11. **The box landed 2026-09-15**, and the boot disk,
the bracket and the tape with it; the two Exos drives did not, so what the
build waits on is those and a bench, not money. The
OS is decided: Ubuntu Server and one compose stack, not an appliance. Nothing
is configured.
Reading the enforced ruleset first changed two of the answers. **50→40 is not
Expand Down Expand Up @@ -795,8 +815,9 @@ what left this one unfireable for months.
did not exist, and PVE already does the snapshots the local-only answer needs
— so PBS would add a service for a capability that exists. `smaug` is bought
since 2026-09-09 — the TS150 of
[#413](https://github.com/Gerrrt/HomeLab/issues/413), in transit, its two
drives bought on 2026-09-11 — and not built, so the deferral's trigger has moved from
[#413](https://github.com/Gerrrt/HomeLab/issues/413), in hand since
2026-09-15, its two drives bought on 2026-09-11 and still moving — and not
built, so the deferral's trigger has moved from
a purchase to a build: PBS follows the NAS answering on `10.0.40.30`, not the
box arriving. The lab has **revert and not backup** until then, and the ADR
names what gets backed up when it arrives — which is also why ADR-0029 gives PBS no disk
Expand Down
29 changes: 19 additions & 10 deletions docs/runbooks/restore-the-firewall.md
Original file line number Diff line number Diff line change
Expand Up @@ -330,16 +330,25 @@ answer — which of the two ports a fresh install makes WAN and which LAN,
whether the I226 card comes back as `igc0` on the spare, how long the whole
thing takes — and to write the answers back into §3.

**Before the box arrives**, so the rehearsal does not stall on day one. All of
it was measured on 2026-09-09:

- **The I226 card** — bought 2026-09-11, in transit
([`hardware.md`](../hardware.md)). The spare is a
stock refurbished G4 with the onboard NIC only. If its 512 GB SSD turns out
to be a 2.5" drive, the drive carrier and a card in the second M.2 slot
contend for the same space; an M.2 SSD does not.
- **An installer for 2.9.0 or newer** — §3 step 1. The stick in transit is
whatever version it is; read it before the day.
**What the rehearsal is still missing.** This list was written on 2026-09-09
to be lined up before the box arrived. The box arrived on 2026-09-15 and the
card and the installer stick did not, so it stops being a countdown and
becomes a list of what is blocking:

- **The spare itself** — **here since 2026-09-15**, and unopened
([`hardware.md`](../hardware.md)). It is a stock refurbished G4 that the
listing says has the onboard NIC only; nothing has been read off the machine
to confirm that. **Open it before the card arrives**, because of the bullet
below: if its 512 GB SSD is a 2.5" drive, the drive carrier and a card in
the second M.2 slot contend for the same space, and an M.2 SSD does not.
That was an unknown about a box in a van and is now an unanswered question
about a box in the room — answering it while the card can still be returned
costs nothing, and answering it on the day costs the rehearsal.
- **The I226 card** — bought 2026-09-11, **still in transit**
([`hardware.md`](../hardware.md)). One of the two things the rehearsal now
waits on.
- **An installer for 2.9.0 or newer** — §3 step 1, and the other. The stick
in transit is whatever version it is; read it before the day.
- **The newest export on `oracle`**, and the age key's offline copy.
- **The numbers to check against**: `make backup-firewall ARGS=--verify-only`
prints the schema and the rule count — `24.6` and 92 rules on 2026-09-09.
Expand Down
Loading