Skip to content

Heartbeat retention 60 days - #1079

Merged
AchoArnold merged 4 commits into
mainfrom
heartbeat-retention-60-days
Nov 7, 2026
Merged

AchoArnold merged 4 commits into
mainfrom
heartbeat-retention-60-days

Conversation

@AchoArnold

Copy link
Copy Markdown
Member

No description provided.

AchoArnold and others added 2 commits November 1, 2026 14:48
Add a single-field TTL index on heartbeat timestamps while preserving the existing query index.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: f27d8f52-9620-4ee2-a564-ec4870a5f840
Clear the queued-check marker when heartbeat history is missing.
A new heartbeat restarts inactive monitoring without duplicating active checks.
Preserve online/offline notifications and cover recovery with regression tests.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: f27d8f52-9620-4ee2-a564-ec4870a5f840
@codacy-production

codacy-production Bot commented Nov 7, 2026 •

Copy link
Copy Markdown

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 13 complexity · 0 duplication

Metric Results
Complexity 13
Duplication 0

View in Codacy

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

@greptile-apps

greptile-apps Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 3/5

[Medium impact] The changes affect heartbeat history and checks within the MongoDB-backed monitoring workflow.

Fix the duplicate-check and lost-check races before merging.

Findings

  1. P1 Duplicate checks keep running ▶
  2. P1 Returning phones can lose checks ▶
  3. P2 Queue errors keep phones offline ▶

Summary

Adds a 60-day MongoDB TTL index for heartbeat history.

  • MongoDB removes heartbeat records after they reach 60 days old.
  • Heartbeat checks stop for empty history and restart after a new heartbeat.

The restart and stop paths need coordination to prevent duplicate or missing checks. A queue error also skips updating the phone's online state.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Heartbeat arrives] --> B[Save heartbeat]
  B --> C{Check inactive?}
  C -->|Yes| D[Schedule check]
  C -->|No| E[Update phone online]
  D -->|Success| E
  D -->|Error| F[Return before online update]
  G[Check runs] --> H{History exists?}
  H -->|Yes| I[Check phone and schedule next check]
  H -->|No| J[Clear QueueID and stop]
Loading

Reviews (1) · Last reviewed commit: "fix(api): restart stopped checks when he..." · Reviewed by Greptile

Comment thread api/pkg/services/heartbeat_service.go Outdated
Comment on lines +127 to +128
if monitor.QueueID == "" || monitor.RequiresCheck() {
if err = service.scheduleHeartbeatCheck(ctx, heartbeat.Timestamp, &HeartbeatMonitorParams{

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Duplicate checks keep running

Two overlapping heartbeats can both load an inactive monitor before either saves its new QueueID. Both then schedule a check. Each check calls Monitor, which schedules another check without checking whether it still owns QueueID. This creates lasting duplicate checks and missed-heartbeat notifications.

Claim the restart atomically, and add a test with overlapping Store calls.

Comment on lines +309 to +314
if stacktrace.GetCode(err) == repositories.ErrCodeNotFound {
ctxLogger.Info(fmt.Sprintf("heartbeat history does not exist for userID [%s] and owner [%s], stopping check", params.UserID, params.Owner))
if err = service.monitorRepository.UpdateQueueID(ctx, monitor.ID, ""); err != nil {
return service.tracer.WrapErrorSpan(span, stacktrace.Propagatef(err, "cannot clear heartbeat check queue for monitor [%s]", monitor.ID))
}
return nil

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Returning phones can lose checks

After Last returns not found, Store can save a heartbeat and read the still-populated, recently updated QueueID, so it does not restart checks. This branch then clears QueueID and exits. If the phone goes offline again before sending another heartbeat, no task remains to detect it.

Coordinate stopping and restarting checks so a heartbeat arriving in this window cannot be missed.

Comment thread api/pkg/services/heartbeat_service.go Resolved
@AchoArnold
AchoArnold merged commit 06930f1 into main Nov 7, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant