Skip to content
nesty92Public

About

Per-job VM classes for GitLab and GitHub CI on fleeting

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

120 Commits

Folders and files

Repository files navigation

fleetci

Per-job machine classes for GitLab CI, on top of fleeting. A job names the size it needs; it gets a fresh VM of that size, used once.

include:
  - remote: https://raw.githubusercontent.com/nesty92/fleetci/v0.10.0/templates/gitlab-ci.yml

build:
  extends: .fleetci-4vcpu-8gb    # the one line; drop it to go back
  image: golang:1.27
  services: [postgres:17]
  script: go build ./...

Stock gitlab-runner does the job as usual (image:, services:, cache, artifacts). fleetci only decides where it runs: gitlab-runner loads fleeting-plugin-fleetci, and its vm_isolation asks the plugin for a VM named class-${FLEETCI_CLASS} once per job. Each class is its own fleeting plugin, such as a cloud's, with its own warm VMs and limits.

Name classes after what they give, such as 2vcpu-4gb or 8vcpu-32gb-arm, so a job's class says what it gets without opening the runner's config.

On AWS spot a Linux job costs about a tenth of a GitHub-hosted one; see docs/cost.md for the comparison on 104M real jobs, network and caches included.

Picking a class on GitLab

templates/gitlab-ci.yml is a public template with one hidden job per standard class: 1vcpu-2gb, 2vcpu-4gb, 4vcpu-8gb, 8vcpu-16gb and 16vcpu-32gb, each also with -arm. .fleetci alone gets the runner's default class. Include it from a release tag, as above. When your classes carry these names, that's all a project needs.

Classes of your own go in a local file included after the template. fleetci ci-include -extra writes one with only the classes the template lacks:

include:
  - remote: https://raw.githubusercontent.com/nesty92/fleetci/v0.10.0/templates/gitlab-ci.yml
  - local: .gitlab/fleetci.yml   # fleetci ci-include -extra > .gitlab/fleetci.yml

The same file can redefine one of the template's jobs, for example to send .fleetci-4vcpu-8gb to a class of yours named otherwise: the later include wins.

A job extending a class the runner lacks fails when it starts, with the runner's classes in the error; fleetci check lists which of the template's classes the runner has. Or skip the template: fleetci ci-include writes every class of yours into one local file, and a misspelled class then fails when the pipeline is created.

Each hidden job sets the runner tag fleetci and the variable FLEETCI_CLASS. The variable is what carries the class, because GitLab never passes a job's tags to the runner.

GitHub Actions

The same classes serve GitHub Actions. Labels reach fleetci there, so the class is the label:

jobs:
  pr-title:
    runs-on: fleetci-1vcpu-2gb   # a PR title and description check needs little
  build:
    runs-on: fleetci-4vcpu-8gb   # class 4vcpu-8gb; runs-on: fleetci → default class

With a [github] table in the config, fleetci lists each repo's queued jobs, takes a VM of the job's class from the same pool gitlab-runner uses, registers a just-in-time runner with the job's labels and starts it on the VM over SSH. The runner takes one job and exits; the VM is removed. Steps run on the VM itself, as on GitHub's ubuntu-latest; container: and services: use its Docker. GitLab jobs always run in a container on the VM.

A job asking for a class fleetci lacks, or for other labels besides self-hosted, linux and its class's architecture (X64, ARM64), stays queued and is logged. The VM image should carry actions/runner in ~/actions-runner-<version> (the images of hack/local and the github image do); otherwise it is downloaded for each job. Without runner_version, fleetci runs the newest runner the image has.

A whole organization

orgs = ["acme"] serves every repo of an organization, new ones included, with runners registered on the org in runner_group_id (the org's default group unless set; the group must allow the repos, and public ones only if it allows public repos). A repo of a served org gets org runners even when repos lists it too.

The token then needs Self-hosted runners: Read and write on the org, and Actions: Read on its repos. Or sign in as a GitHub App instead of a token: its tokens belong to no person, are renewed by fleetci every hour, and each org it is installed on has its own rate limit.

[github]
orgs = ["acme"]
app_id = 123456
app_private_key_file = "github-app.pem"   # the key generated on the App's page

The App needs the same permissions (Self-hosted runners, Actions: Read, and Administration: Read and write for repos listed in repos), and to be installed on each org or repo.

runner_scope = "repo" registers an org's runners on each job's repo instead. GitHub gives a repo's runner only that repo's jobs, so the job a runner takes is the one it was started for or another of the same repo; with cache disks on several nodes, the VM's node then more often has that repo's disk. The token or App needs Administration: Read and write on the org's repos (not Self-hosted runners), runner groups don't apply, and the org must allow repos to have their own runners (Settings → Actions → Runners).

Webhooks

Without them fleetci finds jobs by listing each repo's queued runs every 10 seconds: at least two API calls per repo each time, so a token's 5,000 calls an hour last about seven idle repos, and an org's repos are listed every 10 minutes to be polled too. With [github.webhook], GitHub tells fleetci when a job is queued, and it gets a VM at once:

[github.webhook]
listen = ":9480"
path = "/github"                 # default "/"
secret_file = "webhook.secret"   # the secret set on the webhook
# tls_cert_file, tls_key_file    # HTTPS; otherwise put it behind a proxy

Point a webhook at it, on the org, the repos or the App, sending Workflow jobs events as application/json, with the same secret. fleetci checks each delivery's signature and refuses the rest. Polling goes on every 5 minutes (poll_interval) for the repos listed in repos and those a delivery came from in the last day, so a delivery GitHub drops is served late, not never. A job whose runner failed or took another job is looked at again on its own.

Without gitlab-runner

fleetci serve -config fleetci.toml runs the GitHub side as a process of its own, for GitHub-only installs or to keep GitHub jobs going when gitlab-runner or its fleetci plugin stops. Run it under a service manager that restarts it:

[Service]
ExecStart=/usr/bin/fleetci serve -config /etc/fleetci/github.toml
Restart=always

Its classes are its own. A class's group driven by two processes would have each remove the other's VMs, so serve and gitlab-runner's plugin need separate files with separate groups (on EC2 separate Auto Scaling groups), and only serve's has [github]. When serve stops it removes its VMs.

Cache on your own storage

By default actions/cache, the caches built into setup-node, setup-go and the like, and buildx's type=gha upload to GitHub, through your NAT, with 10 GB per repo. With cache_store = "bucket" they go to a bucket of your own instead, S3, Google Cloud Storage or Azure, set as gitlab-runner's [runners.cache] is, so one bucket can serve both forges. No workflow changes, and no server to run: fleetci's machine keeps the credentials and gives each VM links to the objects its job may read or write. A bucket in the VMs' region is the bigger win.

[github]
cache_store = "bucket"

[github.cache_bucket]
Type = "s3"
Path = "fleetci"

[github.cache_bucket.s3]
BucketName = "my-ci-caches"
BucketLocation = "eu-north-1"   # AccessKey and SecretKey, or the machine's role

With a cache disk, the disk is in front of the bucket: saves stay on it and are uploaded after the job, which doesn't wait. docs/cache-disk.md has the rest. [github] cache_url instead sends them to a github-actions-cache-server you run. On each VM fleetci patches the runner so it keeps the ACTIONS_RESULTS_URL it is given; both speak the v2 cache protocol, which actions/cache v4 and later use.

Install zstd on every VM image. actions/cache compresses with zstd on all cores where it finds zstd, and with gzip on one core where it doesn't: fleetci lends the VM's zstd to job containers (container:), so a job in golang or node gets it too. The cache key includes the compression, so a VM without zstd misses what a VM with it saved. hack/cache/up.sh runs the server on RustFS for hack/local.

A shell into a failed job

A failed job's VM can be kept for a while, for whoever ran the job to get in with the SSH keys on their GitHub or GitLab account. The job's log ends with an address:

fleetci ssh <address> [ssh options]   # e.g. -i ~/.ssh/id_ed25519
tailcat ssh <address>                 # keys from your agent or ~/.ssh/config only

You need an SSH key on your account (GitHub: Settings → SSH and GPG keys; GitLab: User settings → SSH Keys, not a signing-only key). The keys are read when the job starts on GitHub, when it fails on GitLab; without one, the log says fleetci: no shell: @you has no SSH keys on … and the VM goes as usual: add a key, then re-run the job. tailcat ssh takes no ssh options: anything after the address is the command to run, so it offers the keys ssh finds itself, and an IdentityFile under Host * in ~/.ssh/config leaves out the rest. Load the key with ssh-add, or add Host tailcat-* with its IdentityFile, or use fleetci ssh <address> -i <key>. A key fleetci refuses is in its log, by fingerprint.

On GitHub, the job asks by being re-run with debug logging (Re-run jobs → Enable debug logging), and only someone with write access to the repo is let in. On GitLab, the job sets FLEETCI_SSH: "1" in its variables, and whoever ran its pipeline is let in. The runner offers it only with:

[github.shell]      # or [gitlab.shell]
hold = "10m"        # how long a failed job's VM is kept; it counts against max_instances
# derp_map_url      # tailcat's relays; default tailcat.dev's

Never for a pull or merge request from a fork. The address works from anywhere, behind NAT on both ends, through tailcat: no port to open, nothing to set up on the VM. Its free relays are rate limited, without an SLA; derp_map_url points at your own. The App or token needs nothing more. Nobody gets in until the job is over. The VM is as the job left it, its cache disk included, with Docker's images and volumes; on GitLab the job's containers are gone. A failed job's cache disk layer is never kept, so nothing done in the shell reaches the next job. Not on fleets of machines yet, nor with the fleeting plugin.

Install

On AWS, terraform-aws-gitlab-runner-autoscaler does all of this when fleetci is turned on for one of its runners. docs/runbooks has whole setups: AWS with that module, your own servers with cloud spikes, a fleet of servers, GitLab and GitHub from one runner, and GitHub only. By hand, from nothing to a first job:

1. Binaries. Each release has plain binaries for Linux and macOS (amd64, arm64) and a checksums.txt:

v=0.10.0 arch=linux-amd64 # x-release-please-version
base=https://github.com/nesty92/fleetci/releases/download/v$v
curl -fsSLO "$base/fleetci-$arch" -fsSLO "$base/fleeting-plugin-fleetci-$arch" -fsSLO "$base/checksums.txt"
sha256sum --ignore-missing -c checksums.txt
install -m 755 "fleetci-$arch" /usr/local/bin/fleetci
install -m 755 "fleeting-plugin-fleetci-$arch" /usr/local/bin/fleeting-plugin-fleetci

Each class also needs its own fleeting plugin, on the PATH or by path in fleetci.toml. gitlab-runner fleeting install only sees the plugins named in config.toml, not fleetci's classes, so download it from its release, such as fleeting-plugin-aws:

curl -fsSL -o /usr/local/bin/fleeting-plugin-aws \
  https://gitlab.com/gitlab-org/fleeting/plugins/aws/-/releases/v1.3.1/downloads/fleeting-plugin-aws-linux-amd64
chmod +x /usr/local/bin/fleeting-plugin-aws

2. One instance group per class, such as an Auto Scaling group set up as its fleeting plugin asks (for AWS: min and desired 0, scale-in protection, AZRebalance suspended). Nothing else may scale it. Its VM image needs:

  • Docker, and an SSH user in the docker group (the class's connector_config.username);
  • for GitHub Actions jobs, libicu and ideally actions/runner unpacked in ~/actions-runner-<version>.

For fleetingd, the images have this, rebuilt weekly and on each actions/runner release: github for GitHub Actions jobs, with the runner and what workflows for GitHub's runners expect; gitlab, Docker alone, for GitLab's.

fleetci waits for cloud-init to finish and for Docker to answer before it hands a VM to a job. hack/aws has a working launch template and boot script.

3. A GitLab runner registered with the tag fleetci and "Run untagged jobs" off (project or group → Settings → CI/CD → Runners → New runner). Keep its token in a file, such as /etc/fleetci/runner.token.

4. fleetci.toml, one class per group; see Configuration.

5. gitlab-runner's side, written by fleetci:

# a new config.toml, holding only this runner:
fleetci runner-config -global -token-file /etc/fleetci/runner.token > /etc/gitlab-runner/config.toml
# or beside other runners (then raise concurrent by the classes' max_instances):
fleetci runner-config -token-file /etc/fleetci/runner.token >> /etc/gitlab-runner/config.toml

6. GitHub Actions (optional): a fine-grained token for the repos, with Actions: Read and Administration: Read and write, in a file named by [github] token_file; for a whole org, see A whole organization.

7. Check everything, then start gitlab-runner:

fleetci check     # every class, GitHub access and config.toml; changes nothing
gitlab-runner run # or: systemctl restart gitlab-runner

8. In each GitLab project, include the public template and pick classes with extends: (see Picking a class on GitLab). For classes the template lacks, also commit:

fleetci ci-include -extra > .gitlab/fleetci.yml

All three commands read /etc/fleetci/fleetci.toml unless given -config or $FLEETCI_CONFIG.

What fleetci check checks

It changes nothing, so it is safe while fleetci runs, and reports one line each:

  • fleetci.toml loads and is valid;
  • each class's plugin starts, its credentials work and its group exists, and no two classes share a group;
  • the host's budget, for the classes on this machine (see Sharing this machine's CPUs and memory);
  • the GitHub token or App can list each repo's jobs and runners, and each org's repos and runners; the webhook's secret and certificate load;
  • config.toml has fleetci's entry, naming each missing or wrong setting with the line to write.

It exits non-zero when anything fails.

Configuration

gitlab-runner's config.toml only points at fleetci's own file (fleetci runner-config writes this part):

[runners.autoscaler]
  plugin = "fleeting-plugin-fleetci"
  capacity_per_instance = 18  # GitLab jobs at once: the sum of the classes' limits
  max_instances = 1
  [runners.autoscaler.plugin_config]
    config_file = "/etc/fleetci/fleetci.toml"
  [runners.autoscaler.vm_isolation]
    enabled = true
    nesting_host = "unix:///fleetci/nesting.sock"
    image = "class-${FLEETCI_CLASS}"
# /etc/fleetci/fleetci.toml
default_class = "2vcpu-4gb"

[classes.1vcpu-2gb]            # linters, PR title and description checks
plugin = "fleeting-plugin-aws"
max_instances = 6
plugin_config = { name = "fleetci-1vcpu-2gb", region = "eu-north-1" }
connector_config = { username = "ubuntu", use_external_addr = true }

[classes.2vcpu-2gb]            # small checks
plugin = "fleeting-plugin-aws"
max_instances = 6
plugin_config = { name = "fleetci-2vcpu-2gb", region = "eu-north-1" }
connector_config = { username = "ubuntu", use_external_addr = true }

[classes.2vcpu-4gb]
plugin = "fleeting-plugin-aws"
max_instances = 4            # a ceiling; nothing runs until a job needs it
plugin_config = { name = "fleetci-2vcpu-4gb", region = "eu-north-1" }
connector_config = { username = "ubuntu", use_external_addr = true }

[classes.8vcpu-16gb]
plugin = "fleeting-plugin-aws"
max_instances = 2
plugin_config = { name = "fleetci-8vcpu-16gb", region = "eu-north-1" }
connector_config = { username = "ubuntu", use_external_addr = true }

# Warm VMs by time of day, as gitlab-runner's [[runners.autoscaler.policy]]:
# of the policies whose periods match, the last applies.
[[classes.8vcpu-16gb.policy]]
idle_count = 0                       # default: boot per job

[[classes.8vcpu-16gb.policy]]
periods = ["* 8-18 * * mon-fri"]
timezone = "Europe/Warsaw"
idle_count = 1                       # one warm in work hours…
scale_factor = 0.5                   # …or half the running jobs,
scale_factor_limit = 2               # at most 2
idle_time = "20m"

[github]
repos = ["owner/name"]
token_file = "github.token"          # fine-grained, Administration: write
cache_url = "http://10.0.1.5:3000/"  # optional: Actions cache server, see above

Each class needs an instance group of its own (an Auto Scaling group, say): each class's scaler removes VMs in its group that it didn't start, so two classes sharing one would remove each other's. fleetci refuses to start, and fleetci check fails, when two classes name the same group. Nothing else should scale those groups either, such as another runner.

A class without policy can set idle_count and idle_time directly, for one policy at all times. Every class takes acquire_timeout (how long a job waits for a VM, default 10m, or 50m with an overflow). Unknown keys are errors; relative paths are relative to the file. Each class's plugin_config goes to its fleeting plugin as it is.

A class's registry_mirrors, Docker Hub pull-through mirrors such as ["http://10.0.0.5:5000"], are written into each VM's Docker daemon.json before it takes a job, so its pulls don't count against Docker Hub's limit for the network's address. An image whose daemon.json names mirrors already keeps its own.

Is each job's class the right size?

fleetci measures each job on its VM from the VM's own kernel: how long its tasks waited for a CPU, for memory or for the disk (pressure stall information), the CPUs busy at its busiest, and its peak memory. Each job logs a verdict, undersized, oversized, io-bound or fits, with the size to try. It also warns when a job is outgrowing its class: its memory peak or CPU wait growing over its earlier runs.

$ fleetci usage
FORGE   JOB             CLASS      RUNS  CPU PEAK  MEM PEAK  LAST
github  acme/api build  2vcpu-4gb  14    2.0/2     0.5/4 GB  undersized: tasks waited for a CPU 97% of the time; try 4 vCPU
gitlab  g/app lint      4vcpu-8gb  31    0.9/4     0.6/8 GB  oversized: at most 0.9 of 4 CPUs busy; used at most 0.6 GB of 8 GB memory; try 2 vCPU, 1 GB

The runs are kept in a SQLite database, usage_file (default usage.db in state_dir), the last 1000 of each job. The figures by class are Prometheus metrics too, at metrics_listen (or a fleet controller's /metrics), with a "Job sizing" row in the Grafana dashboards. docs/metrics.md lists the rest: jobs waiting and running by class, the wait for a VM, warm and cold starts, overflow, boot and readiness times, the host's budget, GitHub's API. See docs/job-usage.md for the rules and how they were checked.

Reusing VMs

By default each VM runs one job and is removed. A class can reuse them, as docker-autoscaler runners do, to keep pulled images and caches: capacity_per_instance (jobs on a VM at once) and max_use_count (jobs before it is removed), both 1 by default. concurrent grows by max_instances × capacity_per_instance.

Reuse is for GitLab only, whose jobs run in containers on the VM. GitHub steps run on the VM itself, as a user who can use its Docker, so a job could read or change another job's VM. fleetci refuses to start, and fleetci check fails, when a file with [github] has a class that reuses VMs. To reuse VMs for GitLab and serve GitHub too, split them: GitHub in a fleetci serve (or another runner) with single-use classes and groups of their own.

A cache disk per job

A fleetingd class can keep pulled images and caches without reusing VMs: each job's VM gets a disk of that job's caches, with Docker's and containerd's data on it, so the images, volumes and gitlab-runner's local cache: of the job's last run are there. A runner's own S3 cache settings still apply. The disk is a new copy-on-write layer on the job's last one, kept only if the VM gives it back whole, so a job never sees another's half-written disk.

[classes.small]
plugin = "fleeting-plugin-fleetingd"
plugin_config = { vm_cache_directory = "/var/lib/fleetci/cache" }  # with the rest

[classes.small.cache_disk]
budget = "100GB"        # all of the class's caches; refs' own, then shared, least recently used first
min_free = "20GB"       # left free on that disk; short of it, jobs run without a cache disk
max_per_cache = "30GB"  # one job's, trimmed to half after each job; past it, it starts empty
max_age = "168h"        # a cache no job used for that long goes
reader_max_age = "48h"  # sooner for merge and pull requests' own

Each job has its own disk, by its ref and name; its first run on a branch starts from the same job's disk on the default branch. Nothing changes in .gitlab-ci.yml: the job's cache: archives are kept on the disk and restored from it before S3. FLEETCI_CACHE: "off" runs a job without one.

docs/cache-disk.md has cache: recipes for npm, pnpm, Yarn, Next.js, Go, Rust, Python, PHP, Ruby and Java, and a block to paste into a project's AGENTS.md or CLAUDE.md for coding agents.

Who can change a cache that other jobs read is fleetci's to say, not the jobs'. A project's shared caches are written by its writers' jobs; any other job writes a disk of its ref's own, which starts from the same job's shared one, so a merge request can't change what the default branch runs with. By default the writers are the jobs on protected branches and tags. Set writers for a project that works another way:

Rule A job is a writer when it runs on
protected a protected branch or tag (the default)
default the default branch, protected or not
branch:<glob> a branch that matches, release/*
tag:<glob> a tag that matches, v* or staging-*
all any branch, merge requests' too
[gitlab.cache_disk]
writers = ["protected"]          # the default; [] makes no job a writer

[[gitlab.cache_disk.projects]]   # the first that matches applies
match = "group/trunk-app"        # no protected branches; main deploys
writers = ["default"]

[[gitlab.cache_disk.projects]]
match = "group/services/**"      # main deploys to one place, tags elsewhere
writers = ["default", "tag:staging-*", "tag:v*"]

A big repository can also keep its checkout on the disk, set per project with checkout = true: the next job fetches only what is new, 2 to 6 s for kubernetes where a clone takes 12 s, 14 to 43 s for chromium where it takes two and a half minutes. See Keeping the checkout.

all lets anyone who can push a branch change what the default branch's and the tags' jobs run with: a poisoned node_modules or Docker image would ship. It is there for projects where every branch is trusted; don't use it otherwise. fleetci reads only what GitLab says of a job (its ref, whether it is protected, whether it is a merge request's), never variables the job could set.

GitHub jobs get a disk too, keyed by their repo and branch, with Docker's data on it; the same rules go in [github.cache_disk], matching owner/name. A pull request's job never writes the shared cache unless the rule is all, and one from a fork never does. See docs/cache-disk.md.

On a fleet of machines, a class takes cache_disk = true in controller.toml, and jobs go to the worker that holds their cache; see docs/cache-disk.md.

fleetci makes the layers with qemu-img, which must be on the host, and needs the fleetingd fork v0.4.0-fleetci.4 or later, which has the disk command. fleetci's releases bundle it. docs/measurements.md has what it saved on real projects in several languages.

Your machines first, the cloud for spikes

overflow names a class that takes a class's jobs while all of its max_instances are in use. For example, microVMs on your own server, with EC2 for the rest:

[classes.4vcpu-8gb]
plugin = "fleeting-plugin-fleetingd"   # our fork, on this machine
max_instances = 6
overflow = "4vcpu-8gb-aws"
# plugin_config: vm_num_cpu_cores = 4, vm_memory_mb = 8192, …

[classes.4vcpu-8gb-aws]
plugin = "fleeting-plugin-aws"
max_instances = 20
plugin_config = { name = "fleetci-4vcpu-8gb", region = "eu-west-3" }
connector_config = { username = "ubuntu", use_external_addr = true }

Jobs keep asking for 4vcpu-8gb, on GitLab and GitHub alike. The seventh job running at once gets an EC2 VM, and once a job on the server ends, the next one goes to the server again. When the overflow class is full too, the job takes the first room in either, its own class first, for up to the class's acquire_timeout, 50 minutes by default here. On GitLab the job already shows as running while it waits, and the wait counts towards its job timeout; on GitHub it stays queued.

The overflow class is a class like any other, so jobs can also ask for it by name, but it can't have an overflow of its own.

Spot VMs the cloud takes back

On AWS, fleetci reads each VM's instance metadata every 2 seconds, through the SSH connection it already makes to the VM: nothing runs on the VM, and no IAM is needed. A VM with a spot interruption notice takes no new job. Its running jobs go on, since fleetci never stops a job for the cloud, and the VM is removed once they end. The cloud takes the VM two minutes after the notice, so a job still running then fails, as it would anyway.

[classes.4vcpu-8gb-aws.reclaim]
watch = true      # default for the AWS plugin; false turns it off
rebalance = true  # also act on rebalance recommendations; default false

A rebalance recommendation comes earlier and says the VM is at higher risk. It is often not followed by an interruption, so by default it is only logged. The VM's SSH server must allow TCP forwarding, as it does by default.

Sharing this machine's CPUs and memory

The classes whose VMs run on this machine share its CPUs and memory: by default those of fleetingd and dockervm, with each VM's size read from their plugin_config. A job gets a VM only while its size still fits, so a big job and many small ones share the server without each class keeping a worst-case share. max_instances stays each class's own ceiling.

The budget is detected at start: the CPUs and memory the machine has, or its cgroup allows, less 1 CPU and a tenth of the memory (at least 2GB) for the host. fleetci logs it, and fleetci check prints it with what the classes could ask for at once. [host] overrides it:

[host]                                   # every key optional
reserve = { cpus = 4, memory = "16GB" }  # keep the detection, change the reserve
cpus    = 48                             # or set the budget outright; more than
memory  = "100GB"                        #   the machine over-commits on purpose
nominate_after = "10m"                   # see below; "0s" holds room for the oldest job at once
placement = "pack"                       # or "spread"; see below
enabled = false                          # no budget: each class keeps its max_instances

[classes.mine]
plugin = "/opt/my-local-plugin"          # a local plugin fleetci doesn't know:
host   = true                            #   put it on the budget, with its size
cpus   = 4
memory = "8GB"

A class can also set host = false to stay off the budget.

Waiting jobs are served as the Kubernetes scheduler serves pods. Room that frees goes to the earliest waiting job it fits, and a job that doesn't fit yet doesn't hold up later ones that do, so the server stays busy. A job that has waited nominate_after (10 minutes) is nominated: from then on the room that frees is held for it, and later jobs fill only what it doesn't need, so small jobs can't starve a big one. One job is nominated at a time, the one waiting longest; fleetci logs "job nominated". A job that finds no room goes to its overflow, if it has one, instead of waiting.

When a class runs on more than one host with room, placement picks the host: pack (the default) the busiest, so whole hosts stay free for large jobs, or spread the idlest, so jobs have quieter neighbours. They are the Kubernetes scheduler's MostAllocated and LeastAllocated scoring. On one machine there is one host; on a fleet, placement in controller.toml picks among the workers. Warm VMs (idle_count, or a policy's) and the VMs of a class that reuses them hold their share all the time; fleetci refuses a budget they don't fit, or one with no room left for a class's VM.

Several servers as one fleet

Several servers, at one site or many, can run jobs as one fleet. Controllers, one or a few for when one goes, take the jobs from GitHub and GitLab and place each on a worker with room; only they hold the forges' credentials. The classes are defined once, in controller.toml. A new server joins with one line, which installs fleetci from a controller and starts it as a worker:

fleetci token create        # on a controller; prints the line below
curl -fsSk --pinnedpubkey sha256//… https://ctl-1:9481/v1/install.sh | sh -s -- ctl-1 <token>

Workers talk to the controllers over mutual TLS with the fleet's own CA, can update themselves one at a time, and keep their cache disks; jobs go to the worker that holds their caches. A fleet of servers has the setup, from the first controller to operating it.

About

Per-job VM classes for GitLab and GitHub CI on fleeting

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages