Skip to content

Preserve idle samples with zero wall time and no CPU time rollup - #199

Merged
szegedi merged 1 commit into
mainfrom
szegedi/keep-idle
Feb 27, 2025
Merged

szegedi merged 1 commit into
mainfrom
szegedi/keep-idle

Conversation

@szegedi

@szegedi szegedi commented Feb 25, 2025 •

Copy link
Copy Markdown
Collaborator

What does this PR do?:
Changes previous behavior where Node.js profiler dropped idle samples from its final pprof file output. It now keeps them and no longer rolls up their CPU time to the next non-idle sample. It also set their wall time to zero.

Motivation:
Keeping idle samples this way has several benefits:

  • It gives us evidence that sampling happened.
    • We can use them to break event loop blocking strides that would falsely look continuous if we discarded such samples.
    • We can detect reductions in sampling rate and thus deterioration of profiling quality. It’s currently unclear how would we communicate this to the user.
  • It gives us a more realistic view of CPU times. Even a 0,5% of CPU usage while idle will add up to 50% CPU usage over 100 samples (normally, 1s), and previously this accrued to whatever is the next non-idle sample, artificially inflating its CPU use.

Additional Notes:

A bit of background on idle samples

In Node.js context, idle samples are samples with essentially no stack traces; V8 profiler records a single-frame sample with function name (idle) when no JS code is running at the sampling moment. This is due to Node.js not being just a generic JS runtime but one that only invokes code as IO and timer event callbacks. The top level JS program that runs when you invoke it with node index.js mostly just sets up some event listeners (a HTTP server typically) and exits. There's not necessarily an active execution (with a stack) at any given time, like for ordinary threaded language runtimes that have a concept of some code always being on stack, just maybe blocked on a lock or IO. Of course Node has that too, it's just that the underlying event loop is in C and as such doesn't register in the V8 profiler until the loop calls into V8 to run an event callback – the V8 callback entry point is always our profiling event horizon.

The why and how of keeping idle samples

Instead of dropping idle samples, we should keep them. We should also mark their wall time as zero, so they don’t show up as wall time and would really serve mostly as evidence that sampling happened at that time. We should also remove them as wall events from the timeline. However, we would also keep any CPU time ascribed to them. Previously, we would roll their CPU time into the next non-idle sample. This means that even with an idle event loop, we’ll probably see some microseconds of CPU time every 10 miliseconds as belonging to the sample that represents both the libuv thread loop idling as well as the sampling mechanism itself.

Jira: PROF-11371

… next

- Fix optionality of various elements of the time profile context
@szegedi
szegedi requested a review from nsavoire as a code owner February 25, 2025 13:21
@pr-commenter

pr-commenter Bot commented Feb 25, 2025

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2025-02-25 13:22:27

Comparing candidate commit 89afc6d in PR branch szegedi/keep-idle with baseline commit 3be8293 in branch main.

Found 0 performance improvements and 1 performance regressions! Performance is the same for 91 metrics, 28 unstable metrics.

scenario:profiler-light-load-no-wall-profiler-18

  • 🟥 cpu_user_time [+1.727ms; +9.024ms] or [+2.009%; +10.493%]

@github-actions

Copy link
Copy Markdown

Overall package size

Self size: 9.55 MB
Deduped: 9.93 MB
No deduping: 9.93 MB

Dependency sizes | name | version | self size | total size | |------|---------|-----------|------------| | source-map | 0.7.4 | 226 kB | 226 kB | | pprof-format | 2.1.0 | 111.69 kB | 111.69 kB | | p-limit | 3.1.0 | 7.75 kB | 13.78 kB | | delay | 5.0.0 | 11.17 kB | 11.17 kB | | node-gyp-build | 3.9.0 | 8.81 kB | 8.81 kB |

🤖 This report was automatically generated by heaviest-objects-in-the-universe

@szegedi szegedi added the semver-minor Usually minor non-breaking improvements label Feb 25, 2025

@nsavoire nsavoire left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice improvement !

@szegedi
szegedi merged commit 51951b7 into main Feb 27, 2025
@szegedi
szegedi deleted the szegedi/keep-idle branch February 27, 2025 10:12
szegedi added a commit that referenced this pull request Mar 13, 2025
…ext (#199)

- Preserve idle samples with zero wall time and no cpu time rollup to next
- Fix optionality of various elements of the time profile context
@szegedi szegedi mentioned this pull request Mar 13, 2025
szegedi added a commit that referenced this pull request Mar 17, 2025
…ext (#199)

- Preserve idle samples with zero wall time and no cpu time rollup to next
- Fix optionality of various elements of the time profile context
szegedi added a commit that referenced this pull request Nov 9, 2026
OtelProfilesBuilder collects any number of normalized profiles, such as
the time and heap profiles of one collection cycle, into one
ExportProfilesServiceRequest of pprof-format/otel. All of them share the
request's dictionary, as the OTel format intends. Each profile is added
through a ProfileVisitor from visitor(), so whatever drives the visit,
the normalizer or a profiler emitting samples itself, needs no knowledge
of OTel. serializeTimeProfileOtel() and serializeHeapProfileOtel() are
shorthands for a request with a single profile.

- An OTel profile has a single sample type, so each added profile
  becomes one OTel profile per sample type (sample count, wall and CPU
  time; in-use and allocated objects and space). When every sampling
  event is a single timestamped observation, as with samplePerContext,
  the sample count is folded into the profile of the period type (wall
  time) instead, provided no other sample adds an observation to that
  profile. It then gets an observation for every sampling event, even
  one whose value is zero, so the count is its number of observations. Idle samples, which deliberately have zero wall time
  (see #199), thereby stay in the wall profile as timestamped evidence
  that sampling happened, while their CPU time stays in the CPU profile.
  No count profile then has to repeat every sample just to carry 1s.
  When some sampling events are summed, the count keeps its profile, so
  no count is lost.
- The profiles' time range starts at the start of collection, so sample
  timestamps fall within [time_unix_nano, time_unix_nano +
  duration_nano), as the format asks.
- Frames become locations, with columns, and the normalizer's stacks
  become the stack table. Functions are identified by what they encode
  as, display name and resolved file, so one script mapping to several
  original files keeps them apart. Functions, locations, stacks,
  attributes and links are deduplicated across all profiles. Every
  dictionary table starts with its zero value.
- Labels become interned attributes: strings, integers, or doubles for
  fractional numbers, which pprof labels can't hold. An optional
  extractLink callback can turn some labels into a link to a span; the
  builder validates the ID lengths, copies the IDs, and treats a link
  whose trace ID or span ID is all zeros, which is invalid, as no link.
- Samples with the same stack, attributes and link are merged.
  Timestamped samples accumulate a value and a timestamp per
  observation; others are summed. Other zero values, such as zero CPU
  time, are left out of their sample type's profile.
- Frames whose source map is declared but missing locally, the
  candidates for server-side unminification, get a
  datadog.frame.missing_source_map location attribute. The profiles
  with such frames get a datadog.profile.has_missing_source_maps
  attribute, which the builder learns from the visitor's end()
  callback. pprof carries this as a profile comment, but OTel's
  pprof.profile.comment is reserved for human-readable text. No
  semantic convention exists for this, so the keys are provisional
  until the Datadog backend confirms them.

Visitors run synchronously, and build() can be called after any number
of completed visits; a visit that throws leaves partial data, so the
builder should then be discarded. The builder isn't exported from the
package yet, so that the API can settle first.

pprof-format/otel first shipped in pprof-format 2.4.0, so that is now
the minimum version.
szegedi added a commit that referenced this pull request Nov 9, 2026
OtelProfilesBuilder collects any number of normalized profiles, such as
the time and heap profiles of one collection cycle, into one
ExportProfilesServiceRequest of pprof-format/otel. All of them share the
request's dictionary, as the OTel format intends. Each profile is added
through a ProfileVisitor from visitor(), so whatever drives the visit,
the normalizer or a profiler emitting samples itself, needs no knowledge
of OTel. serializeTimeProfileOtel() and serializeHeapProfileOtel() are
shorthands for a request with a single profile.

- An OTel profile has a single sample type, so each added profile
  becomes one OTel profile per sample type (sample count, wall and CPU
  time; in-use and allocated objects and space). When every sampling
  event is a single timestamped observation, as with samplePerContext,
  the sample count is folded into the profile of the period type (wall
  time) instead, provided no other sample adds an observation to that
  profile. It then gets an observation for every sampling event, even
  one whose value is zero, so the count is its number of observations. Idle samples, which deliberately have zero wall time
  (see #199), thereby stay in the wall profile as timestamped evidence
  that sampling happened, while their CPU time stays in the CPU profile.
  No count profile then has to repeat every sample just to carry 1s.
  When some sampling events are summed, the count keeps its profile, so
  no count is lost.
- The profiles' time range starts at the start of collection, so sample
  timestamps fall within [time_unix_nano, time_unix_nano +
  duration_nano), as the format asks.
- Frames become locations, with columns, and the normalizer's stacks
  become the stack table. Functions are identified by what they encode
  as, display name and resolved file, so one script mapping to several
  original files keeps them apart. Functions, locations, stacks,
  attributes and links are deduplicated across all profiles. Every
  dictionary table starts with its zero value.
- Labels become interned attributes: strings, integers, or doubles for
  fractional numbers, which pprof labels can't hold. An optional
  extractLink callback can turn some labels into a link to a span; the
  builder validates the ID lengths, copies the IDs, and treats a link
  whose trace ID or span ID is all zeros, which is invalid, as no link.
- Samples with the same stack, attributes and link are merged.
  Timestamped samples accumulate a value and a timestamp per
  observation; others are summed. Other zero values, such as zero CPU
  time, are left out of their sample type's profile.
- Frames whose source map is declared but missing locally, the
  candidates for server-side unminification, get a
  datadog.frame.missing_source_map location attribute. The profiles
  with such frames get a datadog.profile.has_missing_source_maps
  attribute, which the builder learns from the visitor's end()
  callback. pprof carries this as a profile comment, but OTel's
  pprof.profile.comment is reserved for human-readable text. No
  semantic convention exists for this, so the keys are provisional
  until the Datadog backend confirms them.

Visitors run synchronously, and build() can be called after any number
of completed visits; a visit that throws leaves partial data, so the
builder should then be discarded. The builder isn't exported from the
package yet, so that the API can settle first.

pprof-format/otel first shipped in pprof-format 2.4.0, so that is now
the minimum version.
szegedi added a commit that referenced this pull request Nov 9, 2026
OtelProfilesBuilder collects any number of normalized profiles, such as
the time and heap profiles of one collection cycle, into one
ExportProfilesServiceRequest of pprof-format/otel. All of them share the
request's dictionary, as the OTel format intends. Each profile is added
through a ProfileVisitor from visitor(), so whatever drives the visit,
the normalizer or a profiler emitting samples itself, needs no knowledge
of OTel. serializeTimeProfileOtel() and serializeHeapProfileOtel() are
shorthands for a request with a single profile.

- An OTel profile has a single sample type, so each added profile
  becomes one OTel profile per sample type (sample count, wall and CPU
  time; in-use and allocated objects and space). When every sampling
  event is a single timestamped observation, as with samplePerContext,
  the sample count is folded into the profile of the period type (wall
  time) instead, provided no other sample adds an observation to that
  profile. It then gets an observation for every sampling event, even
  one whose value is zero, so the count is its number of observations. Idle samples, which deliberately have zero wall time
  (see #199), thereby stay in the wall profile as timestamped evidence
  that sampling happened, while their CPU time stays in the CPU profile.
  No count profile then has to repeat every sample just to carry 1s.
  When some sampling events are summed, the count keeps its profile, so
  no count is lost.
- The profiles' time range starts at the start of collection, so sample
  timestamps fall within [time_unix_nano, time_unix_nano +
  duration_nano), as the format asks.
- Frames become locations, with columns, and the normalizer's stacks
  become the stack table. Functions are identified by what they encode
  as, display name and resolved file, so one script mapping to several
  original files keeps them apart. Functions, locations, stacks,
  attributes and links are deduplicated across all profiles. Every
  dictionary table starts with its zero value.
- Labels become interned attributes: strings, integers, or doubles for
  fractional numbers, which pprof labels can't hold. An optional
  extractLink callback can turn some labels into a link to a span; the
  builder validates the ID lengths, copies the IDs, and treats a link
  whose trace ID or span ID is all zeros, which is invalid, as no link.
- Samples with the same stack, attributes and link are merged.
  Timestamped samples accumulate a value and a timestamp per
  observation; others are summed. Other zero values, such as zero CPU
  time, are left out of their sample type's profile.
- Frames whose source map is declared but missing locally, the
  candidates for server-side unminification, get a
  datadog.frame.missing_source_map location attribute. The profiles
  with such frames get a datadog.profile.has_missing_source_maps
  attribute, which the builder learns from the visitor's end()
  callback. pprof carries this as a profile comment, but OTel's
  pprof.profile.comment is reserved for human-readable text. No
  semantic convention exists for this, so the keys are provisional
  until the Datadog backend confirms them.

Visitors run synchronously, and build() can be called after any number
of completed visits; a visit that throws leaves partial data, so the
builder should then be discarded. The builder isn't exported from the
package yet, so that the API can settle first.

pprof-format/otel first shipped in pprof-format 2.4.0, so that is now
the minimum version.
szegedi added a commit that referenced this pull request Nov 9, 2026
OtelProfilesBuilder collects any number of normalized profiles, such as
the time and heap profiles of one collection cycle, into one
ExportProfilesServiceRequest of pprof-format/otel. All of them share the
request's dictionary, as the OTel format intends. Each profile is added
through a ProfileVisitor from visitor(), so whatever drives the visit,
the normalizer or a profiler emitting samples itself, needs no knowledge
of OTel. serializeTimeProfileOtel() and serializeHeapProfileOtel() are
shorthands for a request with a single profile.

- An OTel profile has a single sample type, so each added profile
  becomes one OTel profile per sample type (sample count, wall and CPU
  time; in-use and allocated objects and space). When every sampling
  event is a single timestamped observation, as with samplePerContext,
  the sample count is folded into the profile of the period type (wall
  time) instead, provided no other sample adds an observation to that
  profile. It then gets an observation for every sampling event, even
  one whose value is zero, so the count is its number of observations. Idle samples, which deliberately have zero wall time
  (see #199), thereby stay in the wall profile as timestamped evidence
  that sampling happened, while their CPU time stays in the CPU profile.
  No count profile then has to repeat every sample just to carry 1s.
  When some sampling events are summed, the count keeps its profile, so
  no count is lost.
- The profiles' time range starts at the start of collection, so sample
  timestamps fall within [time_unix_nano, time_unix_nano +
  duration_nano), as the format asks.
- Frames become locations, with columns, and the normalizer's stacks
  become the stack table. Functions are identified by what they encode
  as, display name and resolved file, so one script mapping to several
  original files keeps them apart. Functions, locations, stacks,
  attributes and links are deduplicated across all profiles. Every
  dictionary table starts with its zero value.
- Labels become interned attributes: strings, integers, or doubles for
  fractional numbers, which pprof labels can't hold. An optional
  extractLink callback can turn some labels into a link to a span; the
  builder validates the ID lengths, copies the IDs, and treats a link
  whose trace ID or span ID is all zeros, which is invalid, as no link.
- Samples with the same stack, attributes and link are merged.
  Timestamped samples accumulate a value and a timestamp per
  observation; others are summed. Other zero values, such as zero CPU
  time, are left out of their sample type's profile.
- Frames whose source map is declared but missing locally, the
  candidates for server-side unminification, get a
  datadog.frame.missing_source_map location attribute. The profiles
  with such frames get a datadog.profile.has_missing_source_maps
  attribute, which the builder learns from the visitor's end()
  callback. pprof carries this as a profile comment, but OTel's
  pprof.profile.comment is reserved for human-readable text. No
  semantic convention exists for this, so the keys are provisional
  until the Datadog backend confirms them.

Visitors run synchronously, and build() can be called after any number
of completed visits; a visit that throws leaves partial data, so the
builder should then be discarded. The builder isn't exported from the
package yet, so that the API can settle first.

pprof-format/otel first shipped in pprof-format 2.4.0, so that is now
the minimum version.
szegedi added a commit that referenced this pull request Nov 12, 2026
OtelProfilesBuilder collects any number of normalized profiles, such as
the time and heap profiles of one collection cycle, into one
ExportProfilesServiceRequest of pprof-format/otel. All of them share the
request's dictionary, as the OTel format intends. Each profile is added
through a ProfileVisitor from visitor(), so whatever drives the visit,
the normalizer or a profiler emitting samples itself, needs no knowledge
of OTel. serializeTimeProfileOtel() and serializeHeapProfileOtel() are
shorthands for a request with a single profile.

- An OTel profile has a single sample type, so each added profile
  becomes one OTel profile per sample type (sample count, wall and CPU
  time; in-use and allocated objects and space). When every sampling
  event is a single timestamped observation, as with samplePerContext,
  the sample count is folded into the profile of the period type (wall
  time) instead, provided no other sample adds an observation to that
  profile. It then gets an observation for every sampling event, even
  one whose value is zero, so the count is its number of observations. Idle samples, which deliberately have zero wall time
  (see #199), thereby stay in the wall profile as timestamped evidence
  that sampling happened, while their CPU time stays in the CPU profile.
  No count profile then has to repeat every sample just to carry 1s.
  When some sampling events are summed, the count keeps its profile, so
  no count is lost.
- The profiles' time range starts at the start of collection, so sample
  timestamps fall within [time_unix_nano, time_unix_nano +
  duration_nano), as the format asks.
- Frames become locations, with columns, and the normalizer's stacks
  become the stack table. Functions are identified by what they encode
  as, display name and resolved file, so one script mapping to several
  original files keeps them apart. Functions, locations, stacks,
  attributes and links are deduplicated across all profiles. Every
  dictionary table starts with its zero value.
- Labels become interned attributes: strings, integers, or doubles for
  fractional numbers, which pprof labels can't hold. An optional
  extractLink callback can turn some labels into a link to a span; the
  builder validates the ID lengths, copies the IDs, and treats a link
  whose trace ID or span ID is all zeros, which is invalid, as no link.
- A sample whose stack has no frames, such as a timeline event, refers
  to the dictionary's zero-value stack.
- Samples with the same stack, attributes and link are merged.
  Timestamped samples accumulate a value and a timestamp per
  observation; others are summed. Other zero values, such as zero CPU
  time, are left out of their sample type's profile.
- Frames whose source map is declared but missing locally, the
  candidates for server-side unminification, get a
  datadog.frame.missing_source_map location attribute. The profiles
  with such frames get a datadog.profile.has_missing_source_maps
  attribute, which the builder learns from the visitor's end()
  callback. pprof carries this as a profile comment, but OTel's
  pprof.profile.comment is reserved for human-readable text. No
  semantic convention exists for this, so the keys are provisional
  until the Datadog backend confirms them.

visitor() takes options for the OTel profiles that the visit adds:
attributes, and an original payload with its format. Attribute values
can be structured, e.g. key-value lists, and are interned by a canonical
form of their value, so repeating per-collection data such as the kind
of snapshot or endpoint counts on each profile costs an index each. The
original payload can carry a document the backend reads while a
representation in attributes is agreed; the proto requires its two
fields to be set together, so they're set as one.

Visitors run synchronously, and build() can be called after any number
of completed visits; a visit that throws leaves partial data, so the
builder should then be discarded. The builder, its options and the
shorthands are exported from the package.

pprof-format/otel first shipped in pprof-format 2.4.0, so that is now
the minimum version.
szegedi added a commit that referenced this pull request Nov 13, 2026
OtelProfilesBuilder collects any number of normalized profiles, such as
the time and heap profiles of one collection cycle, into one
ExportProfilesServiceRequest of pprof-format/otel. All of them share the
request's dictionary, as the OTel format intends. Each profile is added
through a ProfileVisitor from visitor(), so whatever drives the visit,
the normalizer or a profiler emitting samples itself, needs no knowledge
of OTel. serializeTimeProfileOtel() and serializeHeapProfileOtel() are
shorthands for a request with a single profile.

- An OTel profile has a single sample type, so each added profile
  becomes one OTel profile per sample type (sample count, wall and CPU
  time; in-use and allocated objects and space). When every sampling
  event is a single timestamped observation, as with samplePerContext,
  the sample count is folded into the profile of the period type (wall
  time) instead, provided no other sample adds an observation to that
  profile. It then gets an observation for every sampling event, even
  one whose value is zero, so the count is its number of observations.
  Idle samples, which deliberately have zero wall time (see #199),
  thereby stay in the wall profile as timestamped evidence that sampling
  happened, while their CPU time stays in the CPU profile. No count
  profile then has to repeat every sample just to carry 1s. When some
  sampling events are summed, the count keeps its profile, so no count
  is lost.
- The profiles' time range starts at the start of collection, so sample
  timestamps fall within [time_unix_nano, time_unix_nano +
  duration_nano), as the format asks. All profiles of a request get the
  union of their ranges, so one collection cycle's profiles cover the
  same period, including a heap profile, which is a snapshot at one
  instant.
- Frames become locations, with columns, and the normalizer's stacks
  become the stack table. Functions are identified by what they encode
  as, display name and resolved file, so one script mapping to several
  original files keeps them apart. Functions, locations, stacks,
  attributes and links are deduplicated across all profiles. Every
  dictionary table starts with its zero value.
- Labels become interned attributes: strings, integers, or doubles for
  fractional numbers, which pprof labels can't hold. An optional
  extractLink callback can turn some labels into a link to a span; the
  builder validates the ID lengths, copies the IDs, and treats a link
  whose trace ID or span ID is all zeros, which is invalid, as no link.
- A sample whose stack has no frames, such as a timeline event, refers
  to the dictionary's zero-value stack.
- Samples with the same stack, attributes and link are merged.
  Timestamped samples accumulate a value and a timestamp per
  observation; others are summed. Other zero values, such as zero CPU
  time, are left out of their sample type's profile.
- Frames whose source map is declared but missing locally, the
  candidates for server-side unminification, get a
  datadog.frame.missing_source_map location attribute. The profiles
  with such frames get a datadog.profile.has_missing_source_maps
  attribute, which the builder learns from the visitor's end()
  callback. pprof carries this as a profile comment, but OTel's
  pprof.profile.comment is reserved for human-readable text. No
  semantic convention exists for this, so the keys are provisional
  until the Datadog backend confirms them.

visitor() takes options for the OTel profiles that the visit adds:
attributes, and an original payload with its format. Attribute values
can be structured, e.g. key-value lists, and are interned by a canonical
form of their value, so repeating per-collection data such as the kind
of snapshot or endpoint counts on each profile costs an index each. The
original payload can carry a document the backend reads while a
representation in attributes is agreed; the proto requires its two
fields to be set together, so they're set as one.

Visitors run synchronously, and build() can be called after any number
of completed visits; a visit that throws leaves partial data, so the
builder should then be discarded. The builder, its options and the
shorthands are exported from the package.

pprof-format/otel first shipped in pprof-format 2.4.0, so that is now
the minimum version.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

semver-minor Usually minor non-breaking improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants