Repository navigation
feat: local spark regtest network - #37
Conversation
talosmachina
left a comment
There was a problem hiding this comment.
1 finding (1 P2). Adds an opt-in spark profile: three Spark operators, the SSP with two LDK nodes, LNURL, data-sync, mempool.space and a Core 31 node peered with bitcoind, driven by ./spark. Reviewed 1ec24b9, full tier, static: this PR was not built or run here, since the sandboxed runner could not confirm the author's permission and failed closed.
What I checked, and 4 candidates I ruled out
Read in full: spark, spark-regtest/compose.yml, every script under spark-regtest/scripts/, all four dockerfiles, both patches, both nginx configs, spark.env, docs/spark-regtest.md, the README and docker-compose.yml hunks.
Ran: docker compose config --services with and without --profile spark (the default stack gains no service), and the bind-mount ownership probe behind the finding.
CI: no checks on this branch.
Ruled out
- Host port collisions: 23049, 23081-23083, 23090 and 23535-23537 are unused by every other service in
docker-compose.yml; 60401 and the ZMQ ports stay unpublished. - Unauthenticated SSP internal API: 59050 is not published, only 59049.
- Fork of the shared chain:
miner.shwaits inwait_for_chain_syncfor a peer and forbitcoind's height before mining. bitcoindRPC from spark containers:polaruser/polarpassmatch therpcauthline and./bitcoin-cli.- Cold reader: a second model tried to refute the finding and reproduced it instead (rootful Docker, Compose v5.6.1).
Merge confidence: 4/5, one P2 on Linux hosts only, and the stack was not exercised here.
coreyphillips
left a comment
There was a problem hiding this comment.
Two independent reviews.
needs changing before merge
- Reset eventually exhausts funding on the retained chain (spark:89-90)
worth doing, does not block
- Underfunded miner wallet stalls setup silently, and the suggested fix fails (spark-regtest/scripts/miner.sh:42-58)
./spark upshows no progress while the operators generate keys (spark:66-75)- The chain API wait bypasses the startup timeout (spark-regtest/scripts/ready.sh:91-92)
- App setup omits the required companion patches (README.md:692)
nits
- Docs say
./spark resetissues a new config, but the config is fixed (docs/spark-regtest.md:75) spark.envkeeps upstream's nix comment and an unusedDATA_SYNC_VERSION(spark-regtest/spark.env:4-6)
coreyphillips
left a comment
There was a problem hiding this comment.
Rechecked after 1 new commit (7c57263). Nothing blocking remains.
addressed
- Underfunded miner wallet stalls setup silently, and the suggested fix fails (spark-regtest/scripts/miner.sh:42-58). Fixed:
ready.sh:48-52now reports when the wallet holds under 1 BTC. A stubbed run printed it at 0.5 BTC and showed the pool count at 5 BTC.miner.sh:56-58now gives a recovery that works instead of./bitcoin-cli fund, and the fixed keys inlib.sh:58-73keep the early coinbases across a reset. ./spark upshows no progress while the operators generate keys (spark:66-75). Fixed:spark:67-78now builds, startsspark-readyalone, follows its log, then runsup -dfor the rest. I replayed that order in a scratch project where a dependency turns healthy after 12 s. The progress lines streamed during the 15 s thatup -dblocked.- Docs say
./spark resetissues a new config, but the config is fixed (docs/spark-regtest.md:75). Fixed:docs/spark-regtest.md:82-83now says the config is built from fixed keys and stays the same across./spark resetunlessSPARK_PUBLIC_HOSTchanges, which matchesinit.sh. spark.envkeeps upstream's nix comment and an unusedDATA_SYNC_VERSION(spark-regtest/spark.env:4-6). Fixed: The nix comment andDATA_SYNC_VERSIONare gone (grep finds neither), andspark.env:4now says the keys are repeated in compose.yml. If you want one copy later, aninclude:entry in docker-compose.yml can takeenv_file: spark-regtest/spark.envfor interpolation, so the inline keys could become${...}references.- Reset eventually exhausts funding on the retained chain (spark:89-90). Fixed:
lib.sh:58-73imports fixed descriptors fromSPARK_WALLET_TPRV(spark.env:8) withtimestamp: 0. The tprv's checksum is valid and it derives from 32 bytes of 0x07. On a scratch Core 31 node I ranensure_wallet, mined 200 blocks to it and spent 4,000 BTC, then deleted the wallet directory and ranensure_walletagain. The balance came back the same, change included: 1049.99994439 BTC. - The chain API wait bypasses the startup timeout (spark-regtest/scripts/ready.sh:91-92). Fixed:
ready.sh:97-103now checks the readiness deadline inside the chain API wait. A stubbed run with both markers present, a failing chain-tip endpoint andREADY_TIMEOUT_SECONDS=2exited 1 after 3 s with "the chain API (...) is behind the chain". - App setup omits the required companion patches (README.md:692). Fixed:
README.md:692anddocs/spark-regtest.md:67-72now say the apps read these settings only once the settings changes land on bitkit-android#1697 and bitkit-ios#1128. They also say what a dev build needs patched in until then. This comes before the funding and exit steps, and the README anchor resolves to that section.
new since the last review
worth doing, does not block
- An interrupted key import leaves a blank wallet that crash-loops spark-miner (spark-regtest/scripts/lib.sh:60-72)
coreyphillips
left a comment
There was a problem hiding this comment.
Rechecked after 1 new commit (27153f8). Nothing blocking remains.
addressed
- An interrupted key import leaves a blank wallet that crash-loops spark-miner (spark-regtest/scripts/lib.sh:60-72). Fixed:
spark-regtest/scripts/lib.sh:60-64now creates or loadsdefault, then skips the import only whenlistdescriptorsreturns descriptors. I tested it on a scratch Core 31 regtest node. With a blankdefaulteither loaded or only on disk,ensure_walletimported both descriptors, the 2,500 BTC balance came back, andgetnewaddressworked. A rerun skipped the import. The crash loop is gone. The rescan-phase gap is filed separately as a new finding.
new since the last review
worth doing, does not block
- A rescan cut short after the import leaves the wallet permanently short of coins (spark-regtest/scripts/lib.sh:63-64)
coreyphillips
left a comment
There was a problem hiding this comment.
Rechecked after 1 new commit (5ef4f09). Nothing blocking remains.
addressed
- A rescan cut short after the import leaves the wallet permanently short of coins (spark-regtest/scripts/lib.sh:63-64). Fixed:
spark-regtest/scripts/lib.sh:76-77now runsrescanblockchain 0on every start, whether or not the descriptors already exist. I ran the branch'sensure_walletagainst a scratch Core 31 node with 3,150 blocks paying the fixed keys (14,949.98 BTC). After adocker stopduring the import's scan, the reloaded wallet held 1,850 BTC, and the nextensure_walletbrought it back to 14,949.98 BTC. A stop duringrescanblockchain, and anabortrescan, both made the call fail withRescan aborted.and exit 1, so underset -ethe miner exits and its restart rescans. With the tip three days past the coinbases,rescanblockchain 0still found them, even though the import timestamp isnow.
new since the last review
nits
- The import with timestamp "now" still scans the chain before the full rescan (spark-regtest/scripts/lib.sh:71)
|
Merged: approved by @coreyphillips on 5ef4f09, every review thread answered, no required checks on this repository. |
Refs:
Description
sparkcompose profile, so a local Spark regtest network runs on bitkit-docker's chain for local development, offline E2E runs and emergency-exit tests. Shared CI stays on Lightspark's hosted regtest.breez/spark@b54845c, the last public operator, since Breez's current operator image is private. A small patch makes it skip theRETURNEDtree node status newer SDKs ask for, without which wallet sync fails.breez/spark-sdk@7b6124227, the last commit built on that operator (newer SSPs cannot talk to it), with upstream'sce29ca045backported so the SSP acceptssilent_payment_proof, which Spark SDK 0.28 asks for.spark-bitcoind(Bitcoin Core 31.0), peered with the existingbitcoind, since Core 26.0 relays none of Spark's v3 packages with P2A anchors. Both nodes stay on one chain; its miner waits for that chain before mining and tops its wallet up frombitcoind's. The wallet's keys are fixed, so it keeps its coins across./spark reset../sparkCLI:up,down,reset,config,mine,mine-to,mature,send,tx,address,mempool,invoice,pay, andoperators stop|startfor uncooperative-operator exits.docs/spark-regtest.mdand the README how to start the profile, point Android and iOS dev builds at it, fund wallets and run an exit test.SPARK_LOCAL_CONFIG_B64andSPARK_LOCAL_CHAIN_URL(dev and regtest builds only), land as small PRs stacked on the Spark PRs above. Until then a dev build needs those two values patched in.spark-regtest/spark.envand the fixed seeds incompose.ymlare upstream's public regtest fixtures frombreez/spark-sdkregtest/local(MIT, attributed inspark-regtest/README.md). They hold no value outside this network.Out of Scope
Design
N/A — no UI changes.
Preview
A Bitkit dev build on Spark SDK 0.28.0, connected to this stack.
1-receive2x.mp4
2-app-pays2x.mp4
3-exit-operators-down2x.mp4
QA Notes
Journeys
N/A
Manual Tests
./spark upon a fresh project → ready in about 10 minutes: 12,000 keyshares per operator, the SSP's leaf pool stocked, Alice's channel open../spark pay <invoice> 100000to the app's Spending invoice → "Received Instant Bitcoin", 100,000 sats../spark invoice 10000→ "Bitcoin Sent"; Alice lists the paymentINBOUNDSUCCEEDED, 10,000,000 msat.breez-sdk-spark/0.28.0../spark operators stop, the proxy answers 502):./spark sendpays the funding address, confirmed in one block;./spark mine;./spark mempooland./spark tx;spark-bitcoindandbitcoindreport the same best block after Spark's blocks.Unknown field "silent_payment_proof").Automated Checks
N/A