Add compression mode to replay event

It would be good to know how compression works etc. for replay events. There are a few cases:

Compression is turned off via useCompression: false
Compression is enabled and used
Compression is enabled but we fall back to uncompressed mode due to e.g. worker loading issues

I propose to add a field compressionMode to the replay event payload:

type CompressionMode = 'disabled' | 'compressed' | 'fallback';

type ReplayEvent {
 // ... existing fields
  compressionMode: CompressionMode
}

This way, we can later run analysis how frequently compression fails, which can inform decisions about the compression strategy etc.

We can also eventually use this information to give users tips for improving their replay setup - if we detect a project has only events with fallback compression mode, we can prompt the user to either fix their CSP settings or disable compression, as an example.

Should this be on the event, context, or the replay recording header (where we currently send segment ID?)

@cmanallen

Honestly, no preference here. I'd say on the event would be the easiest to set for us, not sure how it is on the ingestion side of things cc @cmanallen & @JoshFerge

@mydea

@mydea @billyvg There are pros and cons to either event. I think on the replay-event is best. Will the value be static for the entire replay or will some segments be compressed and others not?

@mydea

@mydea @billyvg There are pros and cons to either event. I think on the replay-event is best. Will the value be static for the entire replay or will some segments be compressed and others not?

I'd say as of now the value should be static, but it's not technically necessary/sure that it will be. In theory, it could change, e.g. if we decide to change the logic for what happens on compression errors (currently, we stop, but theoretically we could revert to sending uncompressed events, ...).

Similar to: https://github.com/getsentry/sentry-dotnet/blob/ad080333c2b104393aeb46ccd004147d2d645f1c/src/Sentry/SentryOptions.cs#L401

i would prefer adding to the replay recording headers payload and logging in our consumer, as it is quite a bit of work to make changes to our replay events consumer and add a clickhouse migration.

Honestly I'm fine putting it wherever you think it works best :) we can also put it into the event context if that's easier (not sure if/how we can query from there though?)

i would prefer adding to the replay recording headers payload and logging in our consumer, as it is quite a bit of work to make changes to our replay events consumer and add a clickhouse migration.

What would we need to add to the SDK header? Can we have a proposal on this issue? Seems like we can resolve this issu easily with this approach.
We could also add a flag to indicate whether default masking is on or off? Could help with: #7104 where the motivation is to understand how often folks go on manual PII stripping mode for the recording

@mydea

This way, we can later run analysis how frequently compression fails, which can inform decisions about the compression strategy etc.

@mydea How are you defining failure here? Do you mean decompression in Relay or compression failures in the SDK? If decompression fails in Relay how does that inform compression strategy?

@bruno-garcia

@bruno-garcia Any metrics calculations do not need to be logged. If we want to know what % of replays are masked by default it needs to go into Clickhouse or some other OLAP database.

@cmanallen

@cmanallen it is trivial to pipe GCP logs to bigquery which aggregation queries can be ran on.

clickhouse is not a great usecase for this as we have to manage the schema, migrations, etc. logging and BQ sink is much less operationally heavy and easy to do.

@mydea

This way, we can later run analysis how frequently compression fails, which can inform decisions about the compression strategy etc.

@mydea How are you defining failure here? Do you mean decompression in Relay or compression failures in the SDK? If decompression fails in Relay how does that inform compression strategy?

This is client-side failure - we detect if we cannot load the web worker, and then fall back to uncompressed mode. Basically, the logic would be something like:

const mode = useCompression ? 'compressed' : 'disabled';

if (useCompression) {
  try {
    await tryLoadCompressionWorker();
  } catch(error) {
    mode = 'fallback';
    // fall back to uncompressed payloads
  }
}

@mydea

@mydea And with this data do you only want to know how often compression fails? Or would you like to correlate this data with projects, browsers, devices, sdk-versions, etc?

I think it would be helpful to be able to correlate this with esp. browser, device & sdk version information (e.g. to find out something like "compression fails esp. often for Safari on iOS").

To be clear, what we want to be able to take away from this is:

How frequently is this disabled? --> users have to manually do this! --> If this is very often, we should investigate why users disable this, as the goal should be that mostly it "just works" with compression.
How often does the web worker not work even though it is enabled? --> If this is very often, we should look into ways to improve this - why is it failing?

@mydea

@mydea I would say the replay-event is the correct place to store this compression value. We do not have to ingest this into Clickhouse. If we're permitted, we could log in Relay or in our Snuba consumer (or just forward the message to a consumer owned wholly be replays and do whatever we want) and use @JoshFerge's suggestion to ingest this data into BigQuery. We can draw conclusions from there.

Relatedly, BQ would be a good place to run our maintenance queries (missing/duplicate segments query, etc.). We could run some really intensive stuff on it.

Let's take josh's recommendation and add it to the replay recording headers?

@mydea

@mydea @billyvg unless this is determined super critical in the next week or two, i think the plan is to hold off until we make some refactors to the backend, which would make doing something like this very easy for us.

@mydea

@mydea @billyvg unless this is determined super critical in the next week or two, i think the plan is to hold off until we make some refactors to the backend, which would make doing something like this very easy for us.

That's def. fine, just ping in here when there has been progress and we want to continue looking into this 👍

Navigation Menu

Repository navigation

Add compression mode to replay event #7140

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Add compression mode to replay event #7140

Description

Activity

billyvg commented on Feb 15, 2023

mydea commented on Feb 16, 2023

cmanallen commented on Feb 17, 2023

mydea commented on Feb 20, 2023

bruno-garcia commented on Feb 21, 2023

JoshFerge commented on Feb 21, 2023

mydea commented on Feb 22, 2023

bruno-garcia commented on Feb 27, 2023

cmanallen commented on Feb 28, 2023

cmanallen commented on Feb 28, 2023

JoshFerge commented on Feb 28, 2023

JoshFerge commented on Feb 28, 2023

mydea commented on Mar 1, 2023

cmanallen commented on Mar 1, 2023

mydea commented on Mar 1, 2023

cmanallen commented on Mar 1, 2023

billyvg commented on Mar 1, 2023

JoshFerge commented on Mar 2, 2023

mydea commented on Mar 3, 2023

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions