Skip to content

Autobuild fails intermittently on Ubuntu 20.04 #676

New issue
New issue
Closed
Closed
Autobuild fails intermittently on Ubuntu 20.04#676
Labels
bugSomething isn't working

Description

@jasonleenaylor
jasonleenaylor
opened on Aug 3, 2021

We have noticed that in our CodeQL github action the autobuild fails a little less than 10% of the time.

When it succeeds the dotnet build --no-incremental /p:UseSharedCompilation=false completes without error, but when it fails the logs show a successful build but there is a non 0 exit code.
The fallback to MSBuild doesn't work for our project so the autobuild eventually fails.

So far the job has always succeeded when we re-run it.

Here is one example of the failure in our project:
https://github.com/sillsdev/TheCombine/runs/3204859583#step:4:157

Activity

  1. rdmarsh2
    added
    bugSomething isn't working
    on Aug 5, 2021

  2. rdmarsh2 commented on Aug 5, 2021

    @rdmarsh2
    rdmarsh2
    on Aug 5, 2021

    Hi, I've passed this on to our C# team. In the meantime, would you mind adding this snippet to your CodeQL workflow after the finalize step to upload some extra logs as Actions artifacts for failed jobs?

    - uses: actions/upload-artifact@v2
  if: ${{ failure() }}
  with:
    name: tracer-logs
    path: ${{ runner.temp }}/*.log
  3. jasonleenaylor
    added 2 commits that reference this issue on Aug 5, 2021

    Add failure logs to the artifacts if CodeQL autobuild fails

    bd08f97

    Add failure logs to the artifacts if CodeQL autobuild fails (#1292)

    30c353f

  4. jasonleenaylor commented on Aug 11, 2021

    @jasonleenaylor
    jasonleenaylor
    on Aug 11, 2021
    Author

    @rdmarsh2 We now have a failure with the trace logs to view here:
    https://github.com/sillsdev/TheCombine/pull/1300/checks?check_run_id=3295392201

    Hmm, re-running it seems to have removed the Artifact, but I have a copy if anyone wants it. It doesn't actually appear to have anything useful to my eyes.

  5. jasonleenaylor commented on Aug 13, 2021

    @jasonleenaylor
    jasonleenaylor
    on Aug 13, 2021
    Author

    @rdmarsh2 We had another failure today: https://github.com/sillsdev/TheCombine/pull/1307/checks?check_run_id=3325019354 Would anyone on the C# team have any more suggestions on how to diagnose this?

  6. criemen commented on Aug 17, 2021

    @criemen
    criemen
    on Aug 17, 2021
    Contributor

    @jasonleenaylor I was able to reproduce this by building your project in a loop, after ~40min I got the crash.
    I will discuss this with the team and get back to you how we will address this problem.

  7. jasonleenaylor commented on Sep 3, 2021

    @jasonleenaylor
    jasonleenaylor
    on Sep 3, 2021
    Author

    @criemen We just had this pop up again today. I appreciate that you all made the effort, any updates?

  8. johnthagen commented on Sep 3, 2021

    @johnthagen
    johnthagen
    on Sep 3, 2021

    The latest example from today: https://github.com/sillsdev/TheCombine/runs/3505515280

  9. criemen commented on Sep 6, 2021

    @criemen
    criemen
    on Sep 6, 2021
    Contributor

    @jasonleenaylor I am sorry for not getting back to you, I simply forgot to update the issue after discussing this internally.

    We are currently working on a rewrite of some parts of our build tracing infrastructure that will allow us to completly get rid of the piece of software that is crashing right now.
    Unfortunately, that release is still 1-2 months out, and there's no workaround for this except to re-run the failing workflow.

  10. jasonleenaylor commented on Sep 17, 2021

    @jasonleenaylor
    jasonleenaylor
    on Sep 17, 2021
    Author

    @criemen I hope the rewrite is going smoothly. We hit it again today.

  11. jasonleenaylor commented on Oct 12, 2021

    @jasonleenaylor
    jasonleenaylor
    on Oct 12, 2021
    Author

    @criemen This popped up again for us so I came to check in. Am I right in assuming that the discussion in PR #770 is of interest to us? 🤞

  12. criemen commented on Oct 21, 2021

    @criemen
    criemen
    on Oct 21, 2021
    Contributor

    I am sorry to hear!
    We are still developing a solution to this problem. It is underway, it's just taking some time.
    Unfortunately, the new-style tracing refered to in #770 is not enough (but indeed a part) of that solution.

  13. criemen commented on Apr 22, 2022

    @criemen
    criemen
    on Apr 22, 2022
    Contributor

    I just wanted to pop in and say we've not forgotten you!
    Addressing this issue got postponed for making CodeQL compatible with Windows 11, but now we're back on track.
    Hopefully, we'll be able to ship a fix for this issue around June/July.

  14. jasonleenaylor commented on Jul 1, 2022

    @jasonleenaylor
    jasonleenaylor
    on Jul 1, 2022
    Author

    @criemen We've had a bunch of failures recently. How is it going?

  15. criemen commented on Jul 1, 2022

    @criemen
    criemen
    on Jul 1, 2022
    Contributor

    @jasonleenaylor If everything goes well, we'll roll out the component that replaces the crashing part end of July, and we'll then be able to disable the crashing component in ca. mid August.

  16. criemen commented on Aug 8, 2022

    @criemen
    criemen
    on Aug 8, 2022
    Contributor

    All the code for this is now in-place for the upcoming 2.10.3 release of the CodeQL CLI. Once that hits actions, the crashes you're observing will stop 🎉

    Closing this issue now, if you after the release of 2.10.3 observe more problems, please open a new issue.

  17. criemen
    closed this as completedon Aug 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions