Skip to content

Unstable 4.19 and main branch smoketests #8931

Closed
Closed
Unstable 4.19 and main branch smoketests#8931
Assignees
DaanHooglandsureshanaparti
Labels
Severity:CriticalCritical bug
Milestone
4.19.1.0

Description

@yadvr
yadvr
opened on Apr 17, 2024

Smoketests seen failing both in GitHub Actions jobs and in BO/smoketests in latest 4.19 and main branches.

ISSUE TYPE
  • Bug Report
COMPONENT NAME
GitHub Actions failures, Smoketests failures

Activity

  1. yadvr
    added
    Severity:CriticalCritical bug
    on Apr 17, 2024
  2. yadvr
    assigned
    DaanHoogland
    and
    sureshanaparti
    on Apr 17, 2024
  3. yadvr
    added this to the 4.19.1.0 milestone on Apr 17, 2024

  4. weizhouapache commented on Apr 17, 2024

    @weizhouapache
    weizhouapache
    on Apr 17, 2024
    Member

    it looks an issue with codecov account
    should we talk to apache infra ro codecov ?

    @rohityadavcloud @DaanHoogland

  5. DaanHoogland commented on Apr 18, 2024

    @DaanHoogland
    DaanHoogland
    on Apr 18, 2024
    Contributor

    this set of failures in smoke tests have become common in the last week(s): #8839 (comment), #8911 (comment)

  6. weizhouapache commented on Apr 18, 2024

    @weizhouapache
    weizhouapache
    on Apr 18, 2024
    Member

    it looks the issue has been addressed in #8930 which fixes a regression in #8800
    @vishesh92 may confirm it

  7. DaanHoogland commented on Apr 18, 2024

    @DaanHoogland
    DaanHoogland
    on Apr 18, 2024
    Contributor

    it looks an issue with codecov account should we talk to apache infra ro codecov ?

    @rohityadavcloud @DaanHoogland

    created https://issues.apache.org/jira/browse/INFRA-25725

  8. DaanHoogland commented on Apr 19, 2024

    @DaanHoogland
    DaanHoogland
    on Apr 19, 2024
    Contributor

    another set of tests that fails at times is vm lifecycle

  9. weizhouapache commented on Apr 19, 2024

    @weizhouapache
    weizhouapache
    on Apr 19, 2024
    Member

    another set of tests that fails at times is vm lifecycle

    @DaanHoogland
    yes, I am checking
    looks like intermittent failures.

  10. DaanHoogland commented on Apr 19, 2024

    @DaanHoogland
    DaanHoogland
    on Apr 19, 2024
    Contributor

    another set of tests that fails at times is vm lifecycle

    @DaanHoogland yes, I am checking looks like intermittent failures.

    It is not just your PR, @weizhouapache . I've seen this a lot lately.

  11. weizhouapache commented on Apr 19, 2024

    @weizhouapache
    weizhouapache
    on Apr 19, 2024
    Member

    another set of tests that fails at times is vm lifecycle

    @DaanHoogland yes, I am checking looks like intermittent failures.

    It is not just your PR, @weizhouapache . I've seen this a lot lately.

    @DaanHoogland
    I've found a potential issue which might cause it.
    let me fix it

  12. weizhouapache commented on Apr 19, 2024

    @weizhouapache
    weizhouapache
    on Apr 19, 2024
    Member

    ok, it seems to be caused by the host stuck at Connecting state
    it should be fixed in #8502
    cc @vishesh92 @DaanHoogland

    2024-04-18 17:29:24,294 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-50:ctx-bbc166a8) (logid:d2a04252) Failed to handle host connection:
com.cloud.exception.ConnectionException: Unable to acquire lock on host db16f280-1a2a-4094-bc46-f8c5384c2125
    at com.cloud.agent.manager.AgentManagerImpl.sendReadyAndGetAttache(AgentManagerImpl.java:1153)
    at com.cloud.agent.manager.AgentManagerImpl.handleConnectedAgent(AgentManagerImpl.java:1166)
    at com.cloud.agent.manager.AgentManagerImpl$HandleAgentConnectTask.runInContext(AgentManagerImpl.java:1250)
    at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
    at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
    at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
    at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
    at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)

  13. weizhouapache commented on Apr 19, 2024

    @weizhouapache
    weizhouapache
    on Apr 19, 2024
    Member

    hope #8948 helps

  14. sureshanaparti
    added this to Apache CloudStack 4.19.1on Apr 23, 2024

  15. yadvr commented on Apr 25, 2024

    @yadvr
    yadvr
    on Apr 25, 2024
    MemberAuthor

    Let's do a new round of smoketests and close this if branches are stable enough cc @sureshanaparti @weizhouapache @DaanHoogland

    4.18 - #7345
    4.19 - #8601
    main/4.20 - #8602

  16. weizhouapache commented on Apr 25, 2024

    @weizhouapache
    weizhouapache
    on Apr 25, 2024
    Member

    it looks the only issue now is

    test_01_events_resource Error 423.26 test_events_resource.py

  17. DaanHoogland commented on Apr 25, 2024

    @DaanHoogland
    DaanHoogland
    on Apr 25, 2024
    Contributor

    yes a new one, test_event_resource.py started failing in several PR smoke test runs:
    #8889 (comment)
    #8891 (comment)
    #8809 (comment)
    #8937 (comment)

  18. weizhouapache commented on Apr 26, 2024

    @weizhouapache
    weizhouapache
    on Apr 26, 2024
    Member

    I updated the integration test

    diff --git a/test/integration/smoke/test_events_resource.py b/test/integration/smoke/test_events_resource.py
index 660cbd37bce..4bbab66532f 100644
--- a/test/integration/smoke/test_events_resource.py
+++ b/test/integration/smoke/test_events_resource.py
@@ -16,7 +16,6 @@
 # under the License.
 """ BVT tests for Events Resource
 """
-import json
 import os
 import tempfile
 import time
@@ -185,7 +184,7 @@ class TestEventsResource(cloudstackTestCase):
         for event in events:
             if event.type.startswith("VM.") or (event.type.startswith("NETWORK.") and not event.type.startswith("NETWORK.ELEMENT")) or event.type.startswith("VOLUME.") or event.type.startswith("ACCOUNT.") or event.type.startswith("DOMAIN.") or event.type.startswith("TEMPLATE."):
                 if event.resourceid is None or event.resourcetype is None:
-                    self.debug("Failed event:: %s" % json.dumps(event, indent=2))
+                    self.debug("Failed event:: %s" % event)
                     self.fail("resourceid or resourcetype for the event not found!")
                 else:
                     self.debug("Event %s at %s:: Resource Type: %s, Resource ID: %s" % (event.type, event.created, event.resourcetype, event.resourceid))

    and got the following error (resourceid is not found)
    test_01_events_resource (tests.smoke.test_events_resource.TestEventsResource): DEBUG: Failed event:: {id : '4a7bbf38-35ee-42d3-ada6-09ffc20b702c', username : 'admin', type : 'VOLUME.DESTROY', level : 'INFO', description : 'Successfully completed destroying a volume', account : 'admin', domainid : '37ff2485-cc1e-11ee-a4eb-1e00db000197', domain : 'ROOT', resourcetype : 'Volume', created : '2024-04-26T13:24:11+0000', state : 'Completed'}

    it looks the error test_01_events_resource is caused by #8800 (merged on 2024-04-12)

    cc @vishesh92

  19. sureshanaparti
    moved this to Todo in Apache CloudStack 4.19.1on May 23, 2024
  20. sureshanaparti
    moved this from Todo to In Progress in Apache CloudStack 4.19.1on May 23, 2024
  21. sureshanaparti
    added this to Apache CloudStack BugFest - Issueson Jun 3, 2024
  22. sureshanaparti
    moved this to Todo in Apache CloudStack BugFest - Issueson Jun 3, 2024
  23. DaanHoogland
    moved this from Todo to Dev In Progress in Apache CloudStack BugFest - Issueson Jun 3, 2024

  24. DaanHoogland commented on Jun 18, 2024

    @DaanHoogland
    DaanHoogland
    on Jun 18, 2024
    Contributor

    test_safe_shutdown seems the only one remaining

  25. sureshanaparti commented on Jun 29, 2024

    @sureshanaparti
    sureshanaparti
    on Jun 29, 2024
    Contributor

    test_safe_shutdown fixed in #9276, failure not seen in recent tests.

  26. sureshanaparti
    closed this as completedon Jun 29, 2024
  27. github-project-automation
    moved this from In Progress to Done in Apache CloudStack 4.19.1on Jun 29, 2024
  28. github-project-automation
    moved this from Dev In Progress to Done in Apache CloudStack BugFest - Issueson Jun 29, 2024
  29. DaanHoogland
    removed this from Apache CloudStack BugFest - Issueson Dec 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Severity:CriticalCritical bug

Type

No type

Projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions