Skip to content

bogus usage jobs may be created when usage is behind in processing #12424

Closed
Bug
#12616
Closed
bogus usage jobs may be created when usage is behind in processing#12424
Bug
#12616
Assignees
abh1sar
Milestone
4.20.3

Description

@DaanHoogland
DaanHoogland
opened on Jan 14, 2026

problem

During trouble shooting missing data, when usage is behind it was found that a lot of usage_jobs are created that won’t have a period to process:

2026-01-13 01:00:38,566 DEBUG [cloud.usage.UsageManagerImpl] (Usage-HB-1:null) (logid:) it's been 86438558 ms since last usage job and 66261443 ms until next job, scheduling an immediate job to catch up (aggregation duration is 1440 minutes)
2026-01-13 01:00:38,568 DEBUG [cloud.usage.UsageManagerImpl] (Usage-HB-1:null) (logid:) Scheduling Usage job...
2026-01-13 01:00:38,570 INFO  [cloud.usage.UsageManagerImpl] (Usage-Job-1:null) (logid:) starting usage job...
2026-01-13 01:00:38,606 INFO  [cloud.usage.UsageManagerImpl] (Usage-Job-1:null) (logid:) Parsing usage records between Mon Jan 12 01:00:00 CET 2026 and Tue Jan 13 00:59:59 CET 2026
2026-01-13 01:01:38,581 DEBUG [cloud.usage.UsageManagerImpl] (Usage-HB-1:null) (logid:) it's been 86498581 ms since last usage job and 66201420 ms until next job, scheduling an immediate job to catch up (aggregation duration is 1440 minutes)
2026-01-13 01:01:38,581 DEBUG [cloud.usage.UsageManagerImpl] (Usage-HB-1:null) (logid:) Scheduling Usage job...
2026-01-13 01:02:38,559 DEBUG [cloud.usage.UsageManagerImpl] (Usage-HB-1:null) (logid:) it's been 86558559 ms since last usage job and 66141442 ms until next job, scheduling an immediate job to catch up (aggregation duration is 1440 minutes)
2026-01-13 01:02:38,559 DEBUG [cloud.usage.UsageManagerImpl] (Usage-HB-1:null) (logid:) Scheduling Usage job...
2026-01-13 01:03:38,558 DEBUG [cloud.usage.UsageManagerImpl] (Usage-HB-1:null) (logid:) it's been 86618558 ms since last usage job and 66081443 ms until next job, scheduling an immediate job to catch up (aggregation duration is 1440 minutes)
2026-01-13 01:03:38,558 DEBUG [cloud.usage.UsageManagerImpl] (Usage-HB-1:null) (logid:) Scheduling Usage job...
2026-01-13 01:04:38,559 DEBUG [cloud.usage.UsageManagerImpl] (Usage-HB-1:null) (logid:) it's been 86678559 ms since last usage job and 66021442 ms until next job, scheduling an immediate job to catch up (aggregation duration is 1440 minutes)
2026-01-13 01:04:38,559 DEBUG [cloud.usage.UsageManagerImpl] (Usage-HB-1:null) (logid:) Scheduling Usage job...
2026-01-13 01:05:38,558 DEBUG [cloud.usage.UsageManagerImpl] (Usage-HB-1:null) (logid:) it's been 86738559 ms since last usage job and 65961442 ms until next job, scheduling an immediate job to catch up (aggregation duration is 1440 minutes)
2026-01-13 01:05:38,558 DEBUG [cloud.usage.UsageManagerImpl] (Usage-HB-1:null) (logid:) Scheduling Usage job...
2026-01-13 01:06:02,952 DEBUG [cloud.usage.UsageManagerImpl] (Usage-Job-1:null) (logid:) Creating networking offering: 18 for Vm: 75432 for account: 239496
2026-01-13 01:06:03,028 DEBUG [cloud.usage.UsageManagerImpl] (Usage-Job-1:null) (logid:) create volume with id : 163459 for account: 239496
2026-01-13 01:06:03,497 DEBUG [cloud.usage.UsageManagerImpl] (Usage-Job-1:null) (logid:) deleting network offering: 18 from Vm: 96981

the HeartBeat thread should be clever about existing jobs and not create the extra jobs.

versions

ACS 4.19

The steps to reproduce the bug

  1. have an environment running for a while
  2. prune intermediate and usage tables a couple of months
  3. set the events of those months to unprocessed
  4. start usage
  5. observe the a job starts and multiple other jobs are created.
  6. the extra jobs seem to have an interval of -1 second and fail (of course)

What to do about it?

make sure the heartbeat job does not create jobs while jobs are being processed/scheduled already.

Activity

  1. DaanHoogland
    added the
    Bug
    issue type on Jan 14, 2026
  2. DaanHoogland
    added this to Apache CloudStack BugFest - Issueson Jan 14, 2026
  3. DaanHoogland
    moved this to Todo in Apache CloudStack BugFest - Issueson Jan 14, 2026
  4. DaanHoogland
    added this to the 4.20.3 milestone on Jan 14, 2026
  5. nvazquez
    moved this from Todo to Discuss in Apache CloudStack BugFest - Issueson Jan 15, 2026
  6. abh1sar
    self-assigned this
    on Jan 28, 2026

  7. abh1sar commented on Jan 29, 2026

    @abh1sar
    abh1sar
    on Jan 29, 2026
    Contributor

    I was able to reproduce the issue (With some code orchestration to speed up the process)

    Steps:

    1. Set usage.stats.job.aggregation.range to 5 minutes
    Image
    1. Put a sleep in UsageManagerImpl.parse(UsageJobVO job, long startDateMillis, long endDateMillis) for 5 minutes (we need a long running usage parsing job)
    2. stop the cloudstack-usage service.
    3. start it after ~12 minutes
    4. The heartbeat code sees that it has been long since last job was scheduled, so it schedules it.
    5. Usage job scheduled sleeps for 5 minutes
    6. Heartbeat code runs every minute and logs the Usage job starting every one minute (It doesn't actually start new jobs, just updates the last running job with the new heartbeat)
    2026-01-29 11:31:58,846 INFO  [cloud.usage.UsageServer] (main:[]) (logid:) UsageServer ready...                                                                                                                                                                                                                                     
2026-01-29 11:32:13,827 DEBUG [usage.UsageManagerImpl.Heartbeat] (Usage-HB-1:[]) (logid:) it's been 733825 ms since last usage job and 63166176 ms until next job, scheduling an immediate job to catch up (aggregation duration is 5 minutes)                                                                                      
2026-01-29 11:32:13,828 DEBUG [cloud.usage.UsageManagerImpl_EnhancerByCloudStack_7373c433] (Usage-HB-1:[]) (logid:) Scheduling Usage job...                                                                                                                                                                                         
2026-01-29 11:32:13,829 INFO  [cloud.usage.UsageManagerImpl_EnhancerByCloudStack_7373c433] (Usage-Job-1:[]) (logid:) starting usage job...                                                                                                                                                                                          
2026-01-29 11:32:13,881 INFO  [cloud.usage.UsageManagerImpl_EnhancerByCloudStack_7373c433] (Usage-Job-1:[]) (logid:) Parsing usage records between [2026-01-29T11:20:00+0000] and [2026-01-29T11:31:59+0000].                                                                                                                       
2026-01-29 11:33:13,808 DEBUG [usage.UsageManagerImpl.Heartbeat] (Usage-HB-1:[]) (logid:) it's been 793808 ms since last usage job and 63106193 ms until next job, scheduling an immediate job to catch up (aggregation duration is 5 minutes)                                                                                      
2026-01-29 11:33:13,808 DEBUG [cloud.usage.UsageManagerImpl_EnhancerByCloudStack_7373c433] (Usage-HB-1:[]) (logid:) Scheduling Usage job...                                                                                                                                                                                         
2026-01-29 11:34:13,806 DEBUG [usage.UsageManagerImpl.Heartbeat] (Usage-HB-1:[]) (logid:) it's been 853806 ms since last usage job and 63046195 ms until next job, scheduling an immediate job to catch up (aggregation duration is 5 minutes)                                                                                      
2026-01-29 11:34:13,806 DEBUG [cloud.usage.UsageManagerImpl_EnhancerByCloudStack_7373c433] (Usage-HB-1:[]) (logid:) Scheduling Usage job...                                                                                                                                                                                         
2026-01-29 11:35:13,816 DEBUG [usage.UsageManagerImpl.Heartbeat] (Usage-HB-1:[]) (logid:) it's been 913816 ms since last usage job and 62986185 ms until next job, scheduling an immediate job to catch up (aggregation duration is 5 minutes)                                                                                      
2026-01-29 11:35:13,816 DEBUG [cloud.usage.UsageManagerImpl_EnhancerByCloudStack_7373c433] (Usage-HB-1:[]) (logid:) Scheduling Usage job...                                                                                                                                                                                         
2026-01-29 11:36:13,804 DEBUG [usage.UsageManagerImpl.Heartbeat] (Usage-HB-1:[]) (logid:) it's been 973804 ms since last usage job and 62926197 ms until next job, scheduling an immediate job to catch up (aggregation duration is 5 minutes)                                                                                      
2026-01-29 11:36:13,805 DEBUG [cloud.usage.UsageManagerImpl_EnhancerByCloudStack_7373c433] (Usage-HB-1:[]) (logid:) Scheduling Usage job...                                                                                                                                                                                         
2026-01-29 11:37:13,804 DEBUG [usage.UsageManagerImpl.Heartbeat] (Usage-HB-1:[]) (logid:) it's been 1033804 ms since last usage job and 62866197 ms until next job, scheduling an immediate job to catch up (aggregation duration is 5 minutes)                                                                                     
2026-01-29 11:37:13,804 DEBUG [cloud.usage.UsageManagerImpl_EnhancerByCloudStack_7373c433] (Usage-HB-1:[]) (logid:) Scheduling Usage job...      

    mysql> mysql> select * from usage_job where end_millis=0;
+------+------------------------------------------------------+--------+----------+-----------+--------------+------------+-----------+------------+----------+---------+---------------------+
| id   | host                                                 | pid    | job_type | scheduled | start_millis | end_millis | exec_time | start_date | end_date | success | heartbeat           |
+------+------------------------------------------------------+--------+----------+-----------+--------------+------------+-----------+------------+----------+---------+---------------------+
| 1811 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 393299 |        0 |         0 |            0 |          0 |         0 | NULL       | NULL     |    NULL | 2026-01-29 11:34:13 |
+------+------------------------------------------------------+--------+----------+-----------+--------------+------------+-----------+------------+----------+---------+---------------------+
1 row in set (0.00 sec)

mysql> select * from usage_job where end_millis=0;
+------+------------------------------------------------------+--------+----------+-----------+--------------+------------+-----------+------------+----------+---------+---------------------+
| id   | host                                                 | pid    | job_type | scheduled | start_millis | end_millis | exec_time | start_date | end_date | success | heartbeat           |
+------+------------------------------------------------------+--------+----------+-----------+--------------+------------+-----------+------------+----------+---------+---------------------+
| 1811 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 393299 |        0 |         0 |            0 |          0 |         0 | NULL       | NULL     |    NULL | 2026-01-29 11:34:13 |
+------+------------------------------------------------------+--------+----------+-----------+--------------+------------+-----------+------------+----------+---------+---------------------+
1 row in set (0.00 sec)

mysql> 
mysql> select * from usage_job where end_millis=0;
+------+------------------------------------------------------+--------+----------+-----------+--------------+------------+-----------+------------+----------+---------+---------------------+
| id   | host                                                 | pid    | job_type | scheduled | start_millis | end_millis | exec_time | start_date | end_date | success | heartbeat           |
+------+------------------------------------------------------+--------+----------+-----------+--------------+------------+-----------+------------+----------+---------+---------------------+
| 1811 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 393299 |        0 |         0 |            0 |          0 |         0 | NULL       | NULL     |    NULL | 2026-01-29 11:35:13 |
+------+------------------------------------------------------+--------+----------+-----------+--------------+------------+-----------+------------+----------+---------+---------------------+
1 row in set (0.00 sec)

mysql> select * from usage_job where end_millis=0;
+------+------------------------------------------------------+--------+----------+-----------+--------------+------------+-----------+------------+----------+---------+---------------------+
| id   | host                                                 | pid    | job_type | scheduled | start_millis | end_millis | exec_time | start_date | end_date | success | heartbeat           |
+------+------------------------------------------------------+--------+----------+-----------+--------------+------------+-----------+------------+----------+---------+---------------------+
| 1811 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 393299 |        0 |         0 |            0 |          0 |         0 | NULL       | NULL     |    NULL | 2026-01-29 11:36:13 |
+------+------------------------------------------------------+--------+----------+-----------+--------------+------------+-----------+------------+----------+---------+---------------------+
1 row in set (0.00 sec)

mysql> select * from usage_job where end_millis=0;
+------+------------------------------------------------------+--------+----------+-----------+--------------+------------+-----------+------------+----------+---------+---------------------+
| id   | host                                                 | pid    | job_type | scheduled | start_millis | end_millis | exec_time | start_date | end_date | success | heartbeat           |
+------+------------------------------------------------------+--------+----------+-----------+--------------+------------+-----------+------------+----------+---------+---------------------+
| 1812 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 393299 |        0 |         0 |            0 |          0 |         0 | NULL       | NULL     |    NULL | 2026-01-29 11:37:16 |
+------+------------------------------------------------------+--------+----------+-----------+--------------+------------+-----------+------------+----------+---------+---------------------+
1 row in set (0.00 sec)

    Please note that this doesn't create multiple usage_job entry, the last active usage_job is updated.
    But this will queue lots of jobs to the _executor. The _executor is single threaded. So the first actual job will finish and then all the queue jobs will be run one after another, but not end up doing anything.

    As a fix, Heartbeat code should check if there is a job already running and not try to schedule it again.

  8. DaanHoogland
    marked usage jobs are sometimes created for no apparent reason #12503 as a duplicate of this issue on Jan 29, 2026

  9. abh1sar commented on Feb 4, 2026

    @abh1sar
    abh1sar
    on Feb 4, 2026
    Contributor

    I could also see lot's of failed -1 second jobs as a side-effect of the above

    mysql> select * from usage_job where id>3140;       
+------+------------------------------------------------------+--------+----------+-----------+---------------+---------------+-----------+---------------------+---------------------+---------+---------------------+
| id   | host                                                 | pid    | job_type | scheduled | start_millis  | end_millis    | exec_time | start_date          | end_date            | success | heartbeat           |
+------+------------------------------------------------------+--------+----------+-----------+---------------+---------------+-----------+---------------------+---------------------+---------+---------------------+
| 3141 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 393299 |        0 |         0 | 1770085920000 | 1770086219999 |    300810 | 2026-02-03 02:32:00 | 2026-02-03 02:36:59 |       1 | 2026-02-03 02:42:13 |
| 3142 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 393299 |        0 |         0 | 1770086220000 | 1770086519999 |    300870 | 2026-02-03 02:37:00 | 2026-02-03 02:41:59 |       1 | 2026-02-03 02:47:13 |
| 3143 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 393299 |        0 |         0 | 1770086520000 | 1770086819999 |    300879 | 2026-02-03 02:42:00 | 2026-02-03 02:46:59 |       1 | 2026-02-03 02:52:13 |
| 3144 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 393299 |        0 |         0 | 1770086820000 | 1770087119999 |    300785 | 2026-02-03 02:47:00 | 2026-02-03 02:51:59 |       1 | 2026-02-03 02:57:13 |
| 3146 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770087120000 | 1770087659999 |   3602755 | 2026-02-03 02:52:00 | 2026-02-03 03:00:59 |       1 | 2026-02-03 04:01:30 |
| 3147 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770087660000 | 1770091259999 |     11428 | 2026-02-03 03:01:00 | 2026-02-03 04:00:59 |       1 | 2026-02-03 04:01:33 |
| 3148 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         2 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:44 |
| 3149 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:44 |
| 3150 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:44 |
| 3151 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:44 |
| 3152 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         2 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:44 |
| 3153 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:44 |
| 3154 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         2 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3155 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3156 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3157 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3158 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3159 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3160 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3161 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3162 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3163 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3164 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3165 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3166 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3167 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3168 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3169 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3170 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         2 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3171 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3172 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         2 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3173 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3174 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3175 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3176 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3177 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3178 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         2 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3179 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3180 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3181 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3182 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         2 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3183 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3184 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3185 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3186 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3187 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         2 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3188 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3189 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3190 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:46 |
| 3191 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:46 |
| 3192 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         2 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:46 |
| 3193 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:46 |
| 3194 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:46 |
| 3195 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         2 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:46 |
| 3196 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:46 |
| 3197 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:46 |
| 3198 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:46 |
| 3199 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:46 |
| 3200 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:46 |
| 3201 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:46 |
| 3202 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:46 |
| 3203 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:46 |
| 3204 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:46 |
| 3205 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:46 |
| 3206 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:46 |
| 3207 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 |             0 |             0 |         0 | NULL                | NULL                |    NULL | 2026-02-03 04:01:46 |
+------+------------------------------------------------------+--------+----------+-----------+---------------+---------------+-----------+---------------------+---------------------+---------+---------------------+
  10. DaanHoogland
    linked a pull request that will close this issueUsage: Heartbeat should not schedule usage job when a job is already running #12616on Feb 11, 2026

  11. DaanHoogland commented on Feb 13, 2026

    @DaanHoogland
    DaanHoogland
    on Feb 13, 2026
    ContributorAuthor

    fixed in #12616

  12. DaanHoogland
    closed this as completedon Feb 13, 2026
  13. github-project-automation
    moved this from Discuss to Done in Apache CloudStack BugFest - Issueson Feb 13, 2026
  14. DaanHoogland
    removed this from Apache CloudStack BugFest - Issueson Mar 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Bug

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions