Skip to content

Improve CI/CD for the repo.  #57

Open

Description

@tok-kkk

Object

We want to have a test suite build into our CI system which helps us reduce the amount of manual testing. The test suite should cover most of the CLI command and tests on different VPSs and systems.

Design

Testing Accounts

We should create separate accounts or sub-users of our main account and limit the service access. This ensures the safety of our VPS main account. We also need to make sure the account credential is private. (i.e. set up as Github secrets or environment variables)

Testing cases

CI should be triggered on every new commit. Both CircleCI and GitHub Actions allow you to skip certain commits if you have [skip ci] in the title or description. source.It should cover both positive and negative test cases. And tests against different VPS should be running parallel if that's possible. It should test the following cases:

  • Test the CLI binary is working and returns the expected version.
  • up a couple of nodes with different options and destroy them with different options.
  • Deploy a node, testing following cases before destroy
    • check start, stop and restart commands are able to control the darknode status.
    • check list command show the correct info of the darknode
    • check update command is able to compare the versions and do an update when outdated.
    • try resize the darknode to an upper tier and a lower tier and check the darknode status
    • check exec command with different options

Discussion

  • Github Actions VS CircleCI
    I personally favour CircleCI as we already have everything set up in CircleCI and it has some features which Github Actions doesn't have. Github Action is also good in terms of github integration, the setup should be more easy and friendly than CircleCi. We can achieve the same result with Github Actions, but with more developing time.

  • Not sure how can we test the ssh command in CI.

Activity

  1. added this to the v3.1.0 milestone on Apr 15, 2020
  2. self-assigned this
    on Apr 15, 2020
  3. loongy commented on Apr 16, 2020

    @loongy
    Contributor

    Are these tests going to be run by GitHub Actions, or are they expected to be run by the user on-demand? We should specify that the VPS accounts we use for this must be separated from the main VPS accounts we use so that compromising the CI will not compromise the main accounts.

  4. loongy commented on Apr 16, 2020

    @loongy
    Contributor

    This issue should probably describe the exact commands we will be testing, in what order. For example: we will do darknode up, then darknode version, then darknode down (obviously we want a few different workflows, many of which will be a bit more complex than this).

  5. changed the title [-]Integration tests for CLI[/-] [+][WIP] Integration tests for CLI[/+] on Apr 16, 2020
  6. changed the title [-][WIP] Integration tests for CLI[/-] [+][WIP]Integration tests for CLI[/+] on Apr 16, 2020
  7. loongy commented on Apr 16, 2020

    @loongy
    Contributor

    @tok-kkk let's proceed with GitHub Actions, just because it allows us to keep as much as possible on GitHub, and we can cut down on an external service. GitHub Actions also seems to be a little more reliable recently. RE: the testing ssh command, we can omit that from testing in these tests.

  8. loongy commented on Apr 16, 2020

    @loongy
    Contributor

    CI should be triggered on every new commit.

    This might be a bit extreme, since these tests are probably going to take a decent amount of time to actually execute in full. I would recommend that we only run them on attempts to merge into master (eg. PRs into master).

  9. changed the title [-][WIP]Integration tests for CLI[/-] [+]Integration tests for CLI[/+] on Apr 21, 2020
  10. changed the title [-]Integration tests for CLI[/-] [+]Improve CI/CD for the repo. [/+] on May 22, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

featureNew feature or requestwipWork in progress

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions