Skip to content

About

Pipeline to automatically extract and parse data from Satisfactory

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

16 Commits

Folders and files

Repository files navigation

Satisfactory Assets Extractor

An unattended pipeline that keeps a data + icon set for the game Satisfactory in sync with Steam. On a schedule it detects new game builds, extracts the shipped documentation and icon textures, normalizes them, and hands the result off to a co-located API.

It runs headless on Linux (developed against Ubuntu Server 24.04).

What it does

For each branch (stable / experimental), once per run:

  1. Detect - query Steam for the current build id (anonymous, no login).
  2. Download - steamcmd-update the game client for that branch.
  3. Version - read the human game version (e.g. 1.2.3.1) + engine version from the install.
  4. Docs diff - locate the Docs file and skip the rest if it hasn't changed.
  5. Parse - run the satisfactory-tools/docs-parser package (once per variant: default and -ficsmas), producing the item / recipe / building / schematic data and the list of referenced icon textures.
  6. Extract - pull those textures out of the UE5 pak files with CUE4Parse and normalize each to 256×256 and 64×64 PNGs.
  7. Content-address + dedup - name each image by a hash of its contents, so identical icons are stored once and unchanged images are never re-uploaded.
  8. Rewrite + publish - replace every icon reference in the parsed data with its image id, wrap it as { "data": …, "metadata": {} }, and drop the images + versioned JSON into the API's data directory, then trigger the API import. The API decides whether the build joins the current SFTools version (only cosmetic changes such as icons, names or descriptions - plans keep working) or starts a new one (anything the planner uses changed). That decision, the planner changes behind it and the command to override it end up in the notification.

State (per-branch build id / docs hash, published image ids, run history) lives in MariaDB. Runs are locked per-branch, guarded for disk space and extraction failures, and report success/failure to a log file and (optionally) Discord.

Architecture

Three cooperating components, decoupled by a CLI + JSON/file contract:

Component Tech Role
orchestrator/ PHP 8.5 (Symfony Console) The pipeline: version check, download, docs diff, publish, state, scheduling, notifications. Shells out to the other two.
extractor-net/ .NET 10 + CUE4Parse + SkiaSharp Decodes UE5 textures and writes normalized PNGs + per-asset content hashes.
docs parser satisfactory-tools/docs-parser (Composer) Parses the game Docs into structured data. Driven as a console command.
mod-extractor-net/ .NET 10 + CUE4Parse Reads the blueprint classes of mods (and base-game classes) and writes their values the way the Docs file does.

The heavy lifting that must be .NET (CUE4Parse) is isolated in extractor-net/; everything else is PHP.

Repository layout

orchestrator/     PHP orchestrator (bin/extractor, src/, config/)
extractor-net/    .NET image extractor (Program.cs, Extractor.csproj)
world-extractor-net/  .NET world extractor: resource nodes, collectibles, map tiles
mod-extractor-net/    .NET mod class extractor, used by mod:import
systemd/          systemd units
data/             runtime working dir (git-ignored): downloads, artifacts, images

Requirements

  • Linux, steamcmd, a Steam account that owns Satisfactory (the Docs + icon paks ship only with the game client, not the free dedicated server)
  • PHP 8.5 (pdo_mysql, curl, iconv, gd) + Composer
  • .NET 10 SDK/runtime
  • MariaDB or equivalent
  • Ample disk (~30 GB per branch for the game install)

Setup

Run everything below from the repository root unless noted.

1. Build the components

# PHP orchestrator (installs the docs-parser package + Symfony)
cd orchestrator && composer install && cd ..

# .NET image extractor
cd extractor-net && dotnet build -c Release && cd ..

# .NET world extractor (resource nodes, collectibles, map tiles)
cd world-extractor-net && dotnet build -c Release && cd ..

# .NET mod class extractor (mod:import)
cd mod-extractor-net && dotnet build -c Release && cd ..

The .NET tools are not rebuilt by git pull; rebuild them after every update that touches them.

2. Create the database + user (MariaDB/MySQL)

sudo mariadb <<'SQL'
CREATE DATABASE IF NOT EXISTS sf_extractor CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
CREATE USER IF NOT EXISTS 'sf_extractor'@'localhost' IDENTIFIED BY 'change-me';
GRANT ALL PRIVILEGES ON sf_extractor.* TO 'sf_extractor'@'localhost';
FLUSH PRIVILEGES;
SQL

3. Configure

cp orchestrator/config/config.example.yaml orchestrator/config/config.yaml

Edit orchestrator/config/config.yaml (it's commented throughout). At minimum set:

  • database.* - DSN / user / password matching step 2
  • steam.username - the Steam account that owns Satisfactory
  • extraction.dotnet - path to the .NET 10 runtime (e.g. dotnet) and extraction.dll - path to the built extractor-net/bin/Release/net10.0/Extractor.dll
  • publish.api_data_dir - where the co-located API reads data + images
  • paths.data - working dir for downloads/artifacts (needs ~30 GB per branch)
  • (optional) notifications.discord.webhook_url for success/failure pings

Then create the state tables:

php orchestrator/bin/extractor db:migrate

4. Steam login (one-time)

The Docs + icon paks ship only with the game client, so downloads need an owning account. Log in once interactively so steamcmd caches the session (later non-interactive runs reuse it):

steamcmd +login <username>    # enter password + Steam Guard, then `quit`

5. First run

php orchestrator/bin/extractor run --branch=stable

6. Schedule (hourly systemd timer)

systemd/sf-extractor@.service is a template - copy the units, then edit the three CHANGE_ME/placeholder lines in the installed service to match your host:

  • User= - the account that owns the cached steamcmd session + .NET runtime
  • WorkingDirectory= - absolute path to this repo's orchestrator/
  • ExecStart= - the php binary path if it isn't /usr/bin/php
sudo cp systemd/sf-extractor@.service systemd/sf-extractor@.timer /etc/systemd/system/
sudoedit /etc/systemd/system/sf-extractor@.service   # set User=, WorkingDirectory=, php path
sudo systemctl daemon-reload
sudo systemctl enable --now sf-extractor@stable.timer   # instance name = branch

(You can instead edit the copies in systemd/ before cp - either works.) Only stable is scheduled by default; add experimental (disk permitting) with systemctl enable --now sf-extractor@experimental.timer.

Commands

php orchestrator/bin/extractor check                        # new build? (no credentials/download)
php orchestrator/bin/extractor run --branch=stable          # run one branch
php orchestrator/bin/extractor run --branch=all             # both branches (default)
php orchestrator/bin/extractor run --branch=stable --force  # ignore the "unchanged" check
php orchestrator/bin/extractor db:migrate                   # create/verify state tables
php orchestrator/bin/extractor world --branch=stable        # re-extract + publish the world/map of an install

run extracts the world (resource nodes with locations, collectibles, map tiles) of every new build and publishes it content-addressed into the API data dir (maps/<id>/, world/{source,collectibles}-<hash>.json); the references go into each variant's metadata, which the API's publish keeps. A failed world extraction only warns - the build is published without a map. world does the same for an install that is already published and prints the API command (world:attach) that stamps it into existing versions.

check needs no Steam credentials and works even before the DB exists (it just skips the comparison). run requires the DB.

Operating the scheduled unit:

systemctl start sf-extractor@stable.service      # trigger a run now
journalctl -u sf-extractor@stable.service -e     # the last run's logs
systemctl list-timers 'sf-extractor@*'           # next scheduled run

Failures are written to notifications.log_file and (if configured) a Discord webhook.

Importing mods

php orchestrator/bin/extractor mod:import Easy_Recipes                  # latest release from ficsit.app
php orchestrator/bin/extractor mod:import Easy_Recipes --mod-version 1.4.4 --public
php orchestrator/bin/extractor mod:import --file Easy_Recipes.smod      # a local .smod file
php orchestrator/bin/extractor mod:import RefinedPower --dry-run        # only write data files + report
php orchestrator/bin/extractor mod:selftest                             # check the converter against the base game

mod:import reads a content mod (and every mod it needs) against a branch's install and stores it as an SFTools mod version through the API (mods.command, the API's mods:import; the owner is the API's modImport.ownerUuid). Mods are private unless --public is given.

  1. The mod and its required dependencies (except SML) are resolved on ficsit.app (api.ficsit.app/v2/query, no account needed) and their Windows .smod files downloaded to data/mods/downloads/. --with <file.smod> supplies a dependency from disk.
  2. mod-extractor-net dumps every blueprint class of the mods with the values its class default object sets. Old mods (legacy .pak, e.g. UE 5.3) are read with their own engine version.
  3. A class default object only stores what differs from its parent, so each class is laid over its parent: a base-game class's Docs entry, another mod class, or the C++ defaults. The C++ defaults are not in the paks; they are inferred by comparing base-game blueprints with their Docs entries. Recipe and descriptor names, generator fuels and sink points are derived the way the game derives them. Buildings named differently from their descriptor are renamed so the parser pairs them.
  4. The base Docs is parsed with and without the mod's entries (dependencies are in both); what the mod adds or changes is the mod version's data. Recipes using unknown items are dropped, icons are published like a build's, and the data is handed to the API with links to the dependency versions.

Mods made with ContentLib ship JSON files in a ContentLib/ folder (some have no paks at all), which ContentLib turns into content when the game starts. They are converted to Docs entries the way ContentLib reads them: new items, recipes and schematics (named after their file), UnlockedBy on recipes, visual kits (PNG icons from ContentLib/Icons/ are scaled here, like extracted textures) and item/recipe/schematic patches, which change existing classes. A changed base-game or dependency class replaces that class when the mod is parsed, so it comes out in the mod's data. CDOs/ files (direct property edits) are applied to the properties the Docs list for that class. Names resolve the way ContentLib resolves them (_C optional, case ignored). Class names DocsParser cannot read (Build_X_SF+_C, Recipe_X™_C) are renamed, with every reference, right before parsing.

Classes built on a mod's own C++ classes have no readable parent: the paks store it only as a hash of /Script/<Module>.<Class>, which is matched against the identifiers in the mod DLLs to name it. Their standard game type is guessed from what the blueprint sets (ingredients → recipe, stack size or Desc_ → item, power or inventories or Build_ → manufacturer, …); a base blueprint that shows nothing takes its children's type. Values the C++ class sets in code (a generator's output, for one) are not in the files, so guessed classes come out with standard defaults.

The report lists, per mod, whether it ships code (DLLs) or is blueprint-only, what was imported, which classes were left out and why, the guessed types (grouped by C++ class), types SFTools does not read, and content the files cannot show (ContentLib, conditional schematics, replaced base-game files). --class-map replaces a guess: per blueprint (Build_X_C=FGBuildableGeneratorFuel) or per C++ class for every blueprint built on it (FFPlotRecipe=FGRecipe). Everything a run produced stays in data/artifacts/mods/<time>-<mod>/ (report.json, data/*.json).

Whatever the files cannot show can be written by hand in a mod adapter (orchestrator/src/Mods/Adapters/, registered in ModAdapters). An adapter is written for one mod version (since()) and used for it and every later version; older versions are imported without it. It can add class-map entries (--class-map still wins), change or add the mod's Docs entries before they are parsed (patchEntries, through ModEntries), and change the parsed data (patchData). The report names the adapter used. FicsitFarmingAdapter adds the Seed Extractor's recipes, which the mod picks in code.

mod:selftest rebuilds every base-game class the same way and compares the parsed result with the parse of the real Docs; only vehicles (whose values live in vehicle classes the Docs do not list) and a few one-off base-game quirks should differ.

License

MIT - see LICENSE. Copyright (c) 2026 greeny.

Credits

Built on CUE4Parse, SkiaSharp, and satisfactory-tools/docs-parser. Not affiliated with Coffee Stain Studios.

About

Pipeline to automatically extract and parse data from Satisfactory

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages