Skip to content

Make CI scripts more homogeneous - #220

Merged
tnull merged 5 commits into
lightningdevkit:mainfrom
tnull:2023-12-homogenous-ci
Dec 21, 2023
Merged

tnull merged 5 commits into
lightningdevkit:mainfrom
tnull:2023-12-homogenous-ci

Conversation

@tnull

@tnull tnull commented Dec 18, 2023 •

Copy link
Copy Markdown
Collaborator

Since we now have a number of different integration tests (CLN, VSS), we move them all to the tests folder and align their naming scheme. As most of our Rust tests could also be considered blackbox/integration tests, we also move them from functional_tests.rs to tests/integration_tests_rust.rs to clean up our test directory structure.

@tnull
tnull force-pushed the 2023-12-homogenous-ci branch from 1b18719 to 193591e Compare December 18, 2023 14:47
@tnull tnull changed the title Make CI script names more homogeneous Make CI scripts more homogeneous Dec 18, 2023
@tnull
tnull force-pushed the 2023-12-homogenous-ci branch 9 times, most recently from 79a958d to e9caf8f Compare December 19, 2023 10:09
@tnull
tnull requested a review from G8XSU December 19, 2023 10:27
@tnull
tnull force-pushed the 2023-12-homogenous-ci branch from e9caf8f to 6b4bb42 Compare December 19, 2023 17:58

@G8XSU G8XSU left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lgtm !
Ci is failing.

Comment thread src/test_utils.rs Resolved
@tnull
tnull force-pushed the 2023-12-homogenous-ci branch 2 times, most recently from ef09c6f to c6f90e1 Compare December 20, 2023 10:24
@tnull

tnull commented Dec 20, 2023

Copy link
Copy Markdown
Collaborator Author

Lgtm ! Ci is failing.

Yeah, now went the same route as upstream for the MSRV/1.63 CI: download bitcoind/electrs manually and use them for our tests via a cfg(no_download) flag, which spares us a whole bunch of pinning, also the most recent ones leading to said CI failure.

@@ -0,0 +1,43 @@
# Our Esplora-based tests require `electrs` and `bitcoind`

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't we do this as part of setup_bitcoind_and_electrsd() in test code?

so that our CI and local tests work consistently and in same manner.

This will avoid another flag.
Moreover, the auto-download feature of bitcoind doesn't work on my mac.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't we do this as part of setup_bitcoind_and_electrsd() in test code?

Electrsd and bitcoind already do this in Rust but pull in quite a bit of annoying unnecessary dev dependencies to do so. Rather than constantly pinning new dependencies (and even risking to mask some actual dependency bump requirements), we can just use CI's curl and be done with it.

so that our CI and local tests work consistently and in same manner.

We leave the auto-download feature as the default behavior, so nothing changes for local testing.

Moreover, the auto-download feature of bitcoind doesn't work on my mac.

I assume this is due to some local anti-virus software or similar, which we probably can't make work in the build scripts. If manual downloads work for you, you can use the newly added download script, or just point BITCOIND_EXE and ELECTRS_EXE env. variables to some preinstalled binaries before running with the no_download cfg flag.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I am fine either ways, feel free to continue with current approach.
I was hoping that if we do it ourselves we will just need zip and reqwest and we can use fixed lower versions of those crates.

@G8XSU G8XSU left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Feel free to squash fixups.

.. as all of our tests in `functional_tests.rs` are essentially
integration tests.
Previously, we used the auto-download feature of the
`electrsd`/`bitcoind` crates. While convenient, they unnecessarily
introduced a lot of dependecies (`zip`, `zstd`, `time`, etc.) to our
test environment which needed pinning for the MSRV of 1.63.

Here, we introduce a new `no_download` config flag allowing us to
disable this auto-download feature in CI, where we now opt to download
the corresponding binaries manually. We keep the default-auto-download
as a convenience feature for running tests locally though.
@tnull
tnull force-pushed the 2023-12-homogenous-ci branch from c6f90e1 to 4b8220b Compare December 21, 2023 18:17
@tnull

tnull commented Dec 21, 2023

Copy link
Copy Markdown
Collaborator Author

Feel free to squash fixups.

Squashed without further changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants