Per-job machine classes for GitLab CI, on top of fleeting. A job names the size it needs; it gets a fresh VM of that size, used once.
include:
- remote: https://raw.githubusercontent.com/nesty92/fleetci/v0.10.0/templates/gitlab-ci.yml
build:
extends: .fleetci-4vcpu-8gb # the one line; drop it to go back
image: golang:1.27
services: [postgres:17]
script: go build ./...Stock gitlab-runner does the job as usual (image:, services:, cache,
artifacts). fleetci only decides where it runs: gitlab-runner loads
fleeting-plugin-fleetci, and its vm_isolation asks the plugin for a VM
named class-${FLEETCI_CLASS} once per job. Each class is its own fleeting
plugin, such as a cloud's, with its own warm VMs and limits.
Name classes after what they give, such as 2vcpu-4gb or 8vcpu-32gb-arm,
so a job's class says what it gets without opening the runner's config.
On AWS spot a Linux job costs about a tenth of a GitHub-hosted one; see docs/cost.md for the comparison on 104M real jobs, network and caches included.
templates/gitlab-ci.yml is a public template
with one hidden job per standard class: 1vcpu-2gb, 2vcpu-4gb,
4vcpu-8gb, 8vcpu-16gb and 16vcpu-32gb, each also with -arm.
.fleetci alone gets the runner's default class. Include it from a release
tag, as above. When your classes carry these names, that's all a project
needs.
Classes of your own go in a local file included after the template.
fleetci ci-include -extra writes one with only the classes the template
lacks:
include:
- remote: https://raw.githubusercontent.com/nesty92/fleetci/v0.10.0/templates/gitlab-ci.yml
- local: .gitlab/fleetci.yml # fleetci ci-include -extra > .gitlab/fleetci.ymlThe same file can redefine one of the template's jobs, for example to send
.fleetci-4vcpu-8gb to a class of yours named otherwise: the later include
wins.
A job extending a class the runner lacks fails when it starts, with the
runner's classes in the error; fleetci check lists which of the
template's classes the runner has. Or skip the template:
fleetci ci-include writes every class of yours into one local file, and a
misspelled class then fails when the pipeline is created.
Each hidden job sets the runner tag fleetci and the variable
FLEETCI_CLASS. The variable is what carries the class, because GitLab
never passes a job's tags to the runner.
The same classes serve GitHub Actions. Labels reach fleetci there, so the class is the label:
jobs:
pr-title:
runs-on: fleetci-1vcpu-2gb # a PR title and description check needs little
build:
runs-on: fleetci-4vcpu-8gb # class 4vcpu-8gb; runs-on: fleetci → default classWith a [github] table in the config, fleetci lists each repo's queued jobs,
takes a VM of the job's class from the same pool gitlab-runner uses, registers
a just-in-time runner with the job's labels and starts it on the VM over SSH.
The runner takes one job and exits; the VM is removed. Steps run on the VM
itself, as on GitHub's ubuntu-latest; container: and services: use its
Docker. GitLab jobs always run in a container on the VM.
A job asking for a class fleetci lacks, or for other labels besides
self-hosted, linux and its class's architecture (X64, ARM64), stays
queued and is logged. The VM image should carry actions/runner in
~/actions-runner-<version> (the images of hack/local and the
github image do); otherwise it is downloaded for each job.
Without runner_version, fleetci runs the newest runner the image has.
orgs = ["acme"] serves every repo of an organization, new ones included,
with runners registered on the org in runner_group_id (the org's default
group unless set; the group must allow the repos, and public ones only if
it allows public repos). A repo of a served org gets org runners even when
repos lists it too.
The token then needs Self-hosted runners: Read and write on the org, and Actions: Read on its repos. Or sign in as a GitHub App instead of a token: its tokens belong to no person, are renewed by fleetci every hour, and each org it is installed on has its own rate limit.
[github]
orgs = ["acme"]
app_id = 123456
app_private_key_file = "github-app.pem" # the key generated on the App's pageThe App needs the same permissions (Self-hosted runners, Actions: Read,
and Administration: Read and write for repos listed in repos), and to
be installed on each org or repo.
runner_scope = "repo" registers an org's runners on each job's repo
instead. GitHub gives a repo's runner only that repo's jobs, so the job a
runner takes is the one it was started for or another of the same repo;
with cache disks on several nodes, the VM's node then more often has that
repo's disk. The token or App needs Administration: Read and write on
the org's repos (not Self-hosted runners), runner groups don't apply, and
the org must allow repos to have their own runners (Settings → Actions →
Runners).
Without them fleetci finds jobs by listing each repo's queued runs every
10 seconds: at least two API calls per repo each time, so a token's 5,000
calls an hour last about seven idle repos, and an org's repos are listed
every 10 minutes to be polled too. With [github.webhook], GitHub tells
fleetci when a job is queued, and it gets a VM at once:
[github.webhook]
listen = ":9480"
path = "/github" # default "/"
secret_file = "webhook.secret" # the secret set on the webhook
# tls_cert_file, tls_key_file # HTTPS; otherwise put it behind a proxyPoint a webhook at it, on the org, the repos or the App, sending
Workflow jobs events as application/json, with the same secret. fleetci
checks each delivery's signature and refuses the rest. Polling goes on
every 5 minutes (poll_interval) for the repos listed in repos and those
a delivery came from in the last day, so a delivery GitHub drops is served
late, not never. A job whose runner failed or took another job is looked at
again on its own.
fleetci serve -config fleetci.toml runs the GitHub side as a process of its
own, for GitHub-only installs or to keep GitHub jobs going when gitlab-runner
or its fleetci plugin stops. Run it under a service manager that restarts it:
[Service]
ExecStart=/usr/bin/fleetci serve -config /etc/fleetci/github.toml
Restart=alwaysIts classes are its own. A class's group driven by two processes would have
each remove the other's VMs, so serve and gitlab-runner's plugin need
separate files with separate groups (on EC2 separate Auto Scaling groups),
and only serve's has [github]. When serve stops it removes its VMs.
By default actions/cache, the caches built into setup-node, setup-go
and the like, and buildx's type=gha upload to GitHub, through your NAT, with
10 GB per repo. With cache_store = "bucket" they go to a bucket of your
own instead, S3, Google Cloud Storage or Azure, set as gitlab-runner's
[runners.cache] is, so one bucket can serve both forges. No workflow
changes, and no server to run: fleetci's machine keeps the credentials and
gives each VM links to the objects its job may read or write. A bucket in
the VMs' region is the bigger win.
[github]
cache_store = "bucket"
[github.cache_bucket]
Type = "s3"
Path = "fleetci"
[github.cache_bucket.s3]
BucketName = "my-ci-caches"
BucketLocation = "eu-north-1" # AccessKey and SecretKey, or the machine's roleWith a cache disk, the disk is in front of the
bucket: saves stay on it and are uploaded after the job, which doesn't wait.
docs/cache-disk.md has the
rest. [github] cache_url instead sends them to a
github-actions-cache-server
you run. On each VM fleetci patches the runner so it keeps the
ACTIONS_RESULTS_URL it is given; both speak the v2 cache protocol, which
actions/cache v4 and later use.
Install zstd on every VM image. actions/cache compresses with zstd on
all cores where it finds zstd, and with gzip on one core where it doesn't:
fleetci lends the VM's zstd to job containers (container:), so a job in
golang or node gets it too. The cache key includes the compression, so
a VM without zstd misses what a VM with it saved.
hack/cache/up.sh runs the server on RustFS for hack/local.
A failed job's VM can be kept for a while, for whoever ran the job to get in with the SSH keys on their GitHub or GitLab account. The job's log ends with an address:
fleetci ssh <address> [ssh options] # e.g. -i ~/.ssh/id_ed25519
tailcat ssh <address> # keys from your agent or ~/.ssh/config only
You need an SSH key on your account (GitHub: Settings → SSH and GPG
keys; GitLab: User settings → SSH Keys, not a signing-only key). The keys
are read when the job starts on GitHub, when it fails on GitLab; without
one, the log says fleetci: no shell: @you has no SSH keys on … and the VM
goes as usual: add a key, then re-run the job. tailcat ssh takes no ssh
options: anything after the address is the command to run, so it offers
the keys ssh finds itself, and an IdentityFile under Host * in
~/.ssh/config leaves out the rest. Load the key with ssh-add, or add
Host tailcat-* with its IdentityFile, or use fleetci ssh <address> -i <key>. A key
fleetci refuses is in its log, by fingerprint.
On GitHub, the job asks by being re-run with debug logging (Re-run jobs
→ Enable debug logging), and only someone with write access to the repo
is let in. On GitLab, the job sets FLEETCI_SSH: "1" in its variables,
and whoever ran its pipeline is let in. The runner offers it only with:
[github.shell] # or [gitlab.shell]
hold = "10m" # how long a failed job's VM is kept; it counts against max_instances
# derp_map_url # tailcat's relays; default tailcat.dev'sNever for a pull or merge request from a fork. The address works from
anywhere, behind NAT on both ends, through
tailcat: no port to open, nothing
to set up on the VM. Its free relays are rate limited, without an SLA;
derp_map_url points at your own. The App or token needs nothing more.
Nobody gets in until the job is over. The VM is as the job left it, its
cache disk included, with Docker's images and volumes; on GitLab the job's
containers are gone. A failed job's cache disk layer is never kept, so
nothing done in the shell reaches the next job. Not on fleets of machines
yet, nor with the fleeting plugin.
On AWS, terraform-aws-gitlab-runner-autoscaler does all of this when fleetci is turned on for one of its runners. docs/runbooks has whole setups: AWS with that module, your own servers with cloud spikes, a fleet of servers, GitLab and GitHub from one runner, and GitHub only. By hand, from nothing to a first job:
1. Binaries. Each release
has plain binaries for Linux and macOS (amd64, arm64) and a checksums.txt:
v=0.10.0 arch=linux-amd64 # x-release-please-version
base=https://github.com/nesty92/fleetci/releases/download/v$v
curl -fsSLO "$base/fleetci-$arch" -fsSLO "$base/fleeting-plugin-fleetci-$arch" -fsSLO "$base/checksums.txt"
sha256sum --ignore-missing -c checksums.txt
install -m 755 "fleetci-$arch" /usr/local/bin/fleetci
install -m 755 "fleeting-plugin-fleetci-$arch" /usr/local/bin/fleeting-plugin-fleetciEach class also needs its own fleeting plugin, on the PATH or by path in
fleetci.toml. gitlab-runner fleeting install only sees the plugins named
in config.toml, not fleetci's classes, so download it from its release,
such as fleeting-plugin-aws:
curl -fsSL -o /usr/local/bin/fleeting-plugin-aws \
https://gitlab.com/gitlab-org/fleeting/plugins/aws/-/releases/v1.3.1/downloads/fleeting-plugin-aws-linux-amd64
chmod +x /usr/local/bin/fleeting-plugin-aws2. One instance group per class, such as an Auto Scaling group set up as
its fleeting plugin asks (for AWS: min and desired 0, scale-in protection,
AZRebalance suspended). Nothing else may scale it. Its VM image needs:
- Docker, and an SSH user in the
dockergroup (the class'sconnector_config.username); - for GitHub Actions jobs, libicu and ideally actions/runner unpacked in
~/actions-runner-<version>.
For fleetingd, the images have this, rebuilt weekly and on each
actions/runner release: github for GitHub Actions jobs, with the runner
and what workflows for GitHub's runners expect; gitlab, Docker alone, for
GitLab's.
fleetci waits for cloud-init to finish and for Docker to answer before it hands a VM to a job. hack/aws has a working launch template and boot script.
3. A GitLab runner registered with the tag fleetci and "Run untagged
jobs" off (project or group → Settings → CI/CD → Runners → New runner). Keep
its token in a file, such as /etc/fleetci/runner.token.
4. fleetci.toml, one class per group; see Configuration.
5. gitlab-runner's side, written by fleetci:
# a new config.toml, holding only this runner:
fleetci runner-config -global -token-file /etc/fleetci/runner.token > /etc/gitlab-runner/config.toml
# or beside other runners (then raise concurrent by the classes' max_instances):
fleetci runner-config -token-file /etc/fleetci/runner.token >> /etc/gitlab-runner/config.toml6. GitHub Actions (optional): a fine-grained token for the repos, with
Actions: Read and Administration: Read and write, in a file named by
[github] token_file; for a whole org, see A whole organization.
7. Check everything, then start gitlab-runner:
fleetci check # every class, GitHub access and config.toml; changes nothing
gitlab-runner run # or: systemctl restart gitlab-runner8. In each GitLab project, include the public template and pick
classes with extends: (see Picking a class on GitLab).
For classes the template lacks, also commit:
fleetci ci-include -extra > .gitlab/fleetci.ymlAll three commands read /etc/fleetci/fleetci.toml unless given -config
or $FLEETCI_CONFIG.
It changes nothing, so it is safe while fleetci runs, and reports one line each:
fleetci.tomlloads and is valid;- each class's plugin starts, its credentials work and its group exists, and no two classes share a group;
- the host's budget, for the classes on this machine (see Sharing this machine's CPUs and memory);
- the GitHub token or App can list each repo's jobs and runners, and each org's repos and runners; the webhook's secret and certificate load;
config.tomlhas fleetci's entry, naming each missing or wrong setting with the line to write.
It exits non-zero when anything fails.
gitlab-runner's config.toml only points at fleetci's own file
(fleetci runner-config writes this part):
[runners.autoscaler]
plugin = "fleeting-plugin-fleetci"
capacity_per_instance = 18 # GitLab jobs at once: the sum of the classes' limits
max_instances = 1
[runners.autoscaler.plugin_config]
config_file = "/etc/fleetci/fleetci.toml"
[runners.autoscaler.vm_isolation]
enabled = true
nesting_host = "unix:///fleetci/nesting.sock"
image = "class-${FLEETCI_CLASS}"# /etc/fleetci/fleetci.toml
default_class = "2vcpu-4gb"
[classes.1vcpu-2gb] # linters, PR title and description checks
plugin = "fleeting-plugin-aws"
max_instances = 6
plugin_config = { name = "fleetci-1vcpu-2gb", region = "eu-north-1" }
connector_config = { username = "ubuntu", use_external_addr = true }
[classes.2vcpu-2gb] # small checks
plugin = "fleeting-plugin-aws"
max_instances = 6
plugin_config = { name = "fleetci-2vcpu-2gb", region = "eu-north-1" }
connector_config = { username = "ubuntu", use_external_addr = true }
[classes.2vcpu-4gb]
plugin = "fleeting-plugin-aws"
max_instances = 4 # a ceiling; nothing runs until a job needs it
plugin_config = { name = "fleetci-2vcpu-4gb", region = "eu-north-1" }
connector_config = { username = "ubuntu", use_external_addr = true }
[classes.8vcpu-16gb]
plugin = "fleeting-plugin-aws"
max_instances = 2
plugin_config = { name = "fleetci-8vcpu-16gb", region = "eu-north-1" }
connector_config = { username = "ubuntu", use_external_addr = true }
# Warm VMs by time of day, as gitlab-runner's [[runners.autoscaler.policy]]:
# of the policies whose periods match, the last applies.
[[classes.8vcpu-16gb.policy]]
idle_count = 0 # default: boot per job
[[classes.8vcpu-16gb.policy]]
periods = ["* 8-18 * * mon-fri"]
timezone = "Europe/Warsaw"
idle_count = 1 # one warm in work hours…
scale_factor = 0.5 # …or half the running jobs,
scale_factor_limit = 2 # at most 2
idle_time = "20m"
[github]
repos = ["owner/name"]
token_file = "github.token" # fine-grained, Administration: write
cache_url = "http://10.0.1.5:3000/" # optional: Actions cache server, see aboveEach class needs an instance group of its own (an Auto Scaling group, say):
each class's scaler removes VMs in its group that it didn't start, so two
classes sharing one would remove each other's. fleetci refuses to start, and
fleetci check fails, when two classes name the same group. Nothing else
should scale those groups either, such as another runner.
A class without policy can set idle_count and idle_time directly, for
one policy at all times. Every class takes acquire_timeout (how long a job
waits for a VM, default 10m, or 50m with an overflow). Unknown keys are errors; relative paths are
relative to the file. Each class's plugin_config goes to its fleeting plugin
as it is.
A class's registry_mirrors, Docker Hub pull-through mirrors such as
["http://10.0.0.5:5000"], are written into each VM's Docker daemon.json
before it takes a job, so its pulls don't count against Docker Hub's limit
for the network's address. An image whose daemon.json names mirrors
already keeps its own.
fleetci measures each job on its VM from the VM's own kernel: how long its
tasks waited for a CPU, for memory or for the disk (pressure stall
information), the CPUs busy at its busiest, and its peak memory. Each job
logs a verdict, undersized, oversized, io-bound or fits, with the
size to try. It also warns when a job is outgrowing its class: its memory
peak or CPU wait growing over its earlier runs.
$ fleetci usage
FORGE JOB CLASS RUNS CPU PEAK MEM PEAK LAST
github acme/api build 2vcpu-4gb 14 2.0/2 0.5/4 GB undersized: tasks waited for a CPU 97% of the time; try 4 vCPU
gitlab g/app lint 4vcpu-8gb 31 0.9/4 0.6/8 GB oversized: at most 0.9 of 4 CPUs busy; used at most 0.6 GB of 8 GB memory; try 2 vCPU, 1 GBThe runs are kept in a SQLite database, usage_file (default usage.db in
state_dir), the last 1000 of each job.
The figures by class are Prometheus metrics too, at metrics_listen (or a
fleet controller's /metrics), with a "Job sizing" row in the Grafana
dashboards. docs/metrics.md lists the rest: jobs waiting
and running by class, the wait for a VM, warm and cold starts, overflow,
boot and readiness times, the host's budget, GitHub's API.
See docs/job-usage.md for the rules and how they were
checked.
By default each VM runs one job and is removed. A class can reuse them, as
docker-autoscaler runners do, to keep pulled images and caches:
capacity_per_instance (jobs on a VM at once) and max_use_count (jobs
before it is removed), both 1 by default. concurrent grows by
max_instances × capacity_per_instance.
Reuse is for GitLab only, whose jobs run in containers on the VM. GitHub
steps run on the VM itself, as a user who can use its Docker, so a job could
read or change another job's VM. fleetci refuses to start, and
fleetci check fails, when a file with [github] has a class that reuses
VMs. To reuse VMs for GitLab and serve GitHub too, split them: GitHub in a
fleetci serve (or another runner) with single-use classes and groups of
their own.
A fleetingd class can keep pulled images and caches without reusing VMs:
each job's VM gets a disk of that job's caches, with Docker's and
containerd's data on it, so the images, volumes and gitlab-runner's local
cache: of the job's last run are there. A runner's own S3 cache settings
still apply. The disk is a new copy-on-write layer on the job's last one,
kept only if the VM gives it back whole, so a job never sees another's
half-written disk.
[classes.small]
plugin = "fleeting-plugin-fleetingd"
plugin_config = { vm_cache_directory = "/var/lib/fleetci/cache" } # with the rest
[classes.small.cache_disk]
budget = "100GB" # all of the class's caches; refs' own, then shared, least recently used first
min_free = "20GB" # left free on that disk; short of it, jobs run without a cache disk
max_per_cache = "30GB" # one job's, trimmed to half after each job; past it, it starts empty
max_age = "168h" # a cache no job used for that long goes
reader_max_age = "48h" # sooner for merge and pull requests' ownEach job has its own disk, by its ref and name; its first run on a branch
starts from the same job's disk on the default branch. Nothing changes in
.gitlab-ci.yml: the job's cache: archives are kept on the disk and
restored from it before S3. FLEETCI_CACHE: "off" runs a job without one.
docs/cache-disk.md has cache: recipes for npm,
pnpm, Yarn, Next.js, Go, Rust, Python, PHP, Ruby and Java, and a block to
paste into a project's AGENTS.md or CLAUDE.md for coding agents.
Who can change a cache that other jobs read is fleetci's to say, not the
jobs'. A project's shared caches are written by its writers' jobs; any other
job writes a disk of its ref's own, which starts from the same job's
shared one, so a merge request can't change what the default branch runs
with. By default the writers are the jobs on protected branches and tags.
Set writers for a project that works another way:
| Rule | A job is a writer when it runs on |
|---|---|
protected |
a protected branch or tag (the default) |
default |
the default branch, protected or not |
branch:<glob> |
a branch that matches, release/* |
tag:<glob> |
a tag that matches, v* or staging-* |
all |
any branch, merge requests' too |
[gitlab.cache_disk]
writers = ["protected"] # the default; [] makes no job a writer
[[gitlab.cache_disk.projects]] # the first that matches applies
match = "group/trunk-app" # no protected branches; main deploys
writers = ["default"]
[[gitlab.cache_disk.projects]]
match = "group/services/**" # main deploys to one place, tags elsewhere
writers = ["default", "tag:staging-*", "tag:v*"]A big repository can also keep its checkout on the disk, set per project
with checkout = true: the next job fetches only what is new, 2 to 6 s
for kubernetes where a clone takes 12 s, 14 to 43 s for chromium where it
takes two and a half minutes. See Keeping the checkout.
all lets anyone who can push a branch change what the default branch's
and the tags' jobs run with: a poisoned node_modules or Docker image would
ship. It is there for projects where every branch is trusted; don't use it
otherwise. fleetci reads only what GitLab says of a job (its ref, whether it
is protected, whether it is a merge request's), never variables the job
could set.
GitHub jobs get a disk too, keyed by their repo and branch, with Docker's
data on it; the same rules go in [github.cache_disk], matching
owner/name. A pull request's job never writes the shared cache unless the
rule is all, and one from a fork never does. See
docs/cache-disk.md.
On a fleet of machines, a class takes cache_disk = true in
controller.toml, and jobs go to the worker that holds their cache; see
docs/cache-disk.md.
fleetci makes the layers with qemu-img, which must be on the host, and
needs the fleetingd fork
v0.4.0-fleetci.4
or later, which has the disk command. fleetci's releases bundle it.
docs/measurements.md has what it saved on real projects in several languages.
overflow names a class that takes a class's jobs while all of its
max_instances are in use. For example, microVMs on your own server, with
EC2 for the rest:
[classes.4vcpu-8gb]
plugin = "fleeting-plugin-fleetingd" # our fork, on this machine
max_instances = 6
overflow = "4vcpu-8gb-aws"
# plugin_config: vm_num_cpu_cores = 4, vm_memory_mb = 8192, …
[classes.4vcpu-8gb-aws]
plugin = "fleeting-plugin-aws"
max_instances = 20
plugin_config = { name = "fleetci-4vcpu-8gb", region = "eu-west-3" }
connector_config = { username = "ubuntu", use_external_addr = true }Jobs keep asking for 4vcpu-8gb, on GitLab and GitHub alike. The seventh
job running at once gets an EC2 VM, and once a job on the server ends, the
next one goes to the server again. When the overflow class is full too, the
job takes the first room in either, its own class first, for up to the
class's acquire_timeout, 50 minutes by default here. On GitLab the job
already shows as running while it waits, and the wait counts towards its job
timeout; on GitHub it stays queued.
The overflow class is a class like any other, so jobs can also ask for it by name, but it can't have an overflow of its own.
On AWS, fleetci reads each VM's instance metadata every 2 seconds, through the SSH connection it already makes to the VM: nothing runs on the VM, and no IAM is needed. A VM with a spot interruption notice takes no new job. Its running jobs go on, since fleetci never stops a job for the cloud, and the VM is removed once they end. The cloud takes the VM two minutes after the notice, so a job still running then fails, as it would anyway.
[classes.4vcpu-8gb-aws.reclaim]
watch = true # default for the AWS plugin; false turns it off
rebalance = true # also act on rebalance recommendations; default falseA rebalance recommendation comes earlier and says the VM is at higher risk. It is often not followed by an interruption, so by default it is only logged. The VM's SSH server must allow TCP forwarding, as it does by default.
The classes whose VMs run on this machine share its CPUs and memory: by
default those of fleetingd and dockervm, with each VM's size read from
their plugin_config. A job gets a VM only while its size still fits, so
a big job and many small ones share the server without each class keeping
a worst-case share. max_instances stays each class's own ceiling.
The budget is detected at start: the CPUs and memory the machine has, or
its cgroup allows, less 1 CPU and a tenth of the memory (at least 2GB) for
the host. fleetci logs it, and fleetci check prints it with what the
classes could ask for at once. [host] overrides it:
[host] # every key optional
reserve = { cpus = 4, memory = "16GB" } # keep the detection, change the reserve
cpus = 48 # or set the budget outright; more than
memory = "100GB" # the machine over-commits on purpose
nominate_after = "10m" # see below; "0s" holds room for the oldest job at once
placement = "pack" # or "spread"; see below
enabled = false # no budget: each class keeps its max_instances
[classes.mine]
plugin = "/opt/my-local-plugin" # a local plugin fleetci doesn't know:
host = true # put it on the budget, with its size
cpus = 4
memory = "8GB"A class can also set host = false to stay off the budget.
Waiting jobs are served as the Kubernetes scheduler serves pods. Room
that frees goes to the earliest waiting job it fits, and a job that doesn't
fit yet doesn't hold up later ones that do, so the server stays busy. A job
that has waited nominate_after (10 minutes) is nominated: from then on
the room that frees is held for it, and later jobs fill only what it
doesn't need, so small jobs can't starve a big one. One job is nominated at
a time, the one waiting longest; fleetci logs "job nominated". A job that
finds no room goes to its overflow, if it has one, instead of waiting.
When a class runs on more than one host with room, placement picks the
host: pack (the default) the busiest, so whole hosts stay free for large
jobs, or spread the idlest, so jobs have quieter neighbours. They are the
Kubernetes scheduler's MostAllocated and LeastAllocated scoring. On one
machine there is one host; on a fleet, placement in controller.toml
picks among the workers. Warm VMs (idle_count, or a policy's) and the VMs of a class that
reuses them hold their share all the time; fleetci refuses a budget they
don't fit, or one with no room left for a class's VM.
Several servers, at one site or many, can run jobs as one fleet.
Controllers, one or a few for when one goes, take the jobs from GitHub and
GitLab and place each on a worker with room; only they hold the forges'
credentials. The classes are defined once, in controller.toml. A new
server joins with one line, which installs fleetci from a controller and
starts it as a worker:
fleetci token create # on a controller; prints the line below
curl -fsSk --pinnedpubkey sha256//… https://ctl-1:9481/v1/install.sh | sh -s -- ctl-1 <token>Workers talk to the controllers over mutual TLS with the fleet's own CA, can update themselves one at a time, and keep their cache disks; jobs go to the worker that holds their caches. A fleet of servers has the setup, from the first controller to operating it.