From 039e0e93f35c18b97e6907991e970bd1850edd87 Mon Sep 17 00:00:00 2001 From: Taylor Caldwell Date: Mon, 14 Sep 2026 14:57:57 -0700 Subject: [PATCH 1/2] Rewrite batch-compliance docs for the api.x.com byte-path flow The three narrative pages still documented the legacy GCS flow: storage.googleapis.com upload/download URLs, resumable uploads, and an event vocabulary that no longer matches results. - introduction: complete the endpoint table (upload/download byte paths, cancel), replace the event tables with the real action/reason contract including rehydrate/tweet_edited and scrub_geo/geo_scrubbed. - quickstart: api.x.com signed URLs with expiry fields and lifetimes, one-active-job-per-type note, upload semantics (single PUT, 200, 403 on expired token), NDJSON results with real examples incl. error records and the deleted_at alias, event-priority rule, cancel section. - integrate: full replacement of the GCS resumable-upload guide with an integration guide covering signed-URL lifecycle, job lifecycle, upload limits (1 GiB single PUT, no resumable mode), result-interpretation rules, the problem+json error catalog, and operational best practices. --- .../compliance/batch-compliance/integrate.mdx | 127 ++++++++++++++++-- .../batch-compliance/introduction.mdx | 33 +++-- .../batch-compliance/quickstart.mdx | 91 +++++++++---- 3 files changed, 201 insertions(+), 50 deletions(-) diff --git a/x-api/compliance/batch-compliance/integrate.mdx b/x-api/compliance/batch-compliance/integrate.mdx index a546cbedf..e9abf4aea 100644 --- a/x-api/compliance/batch-compliance/integrate.mdx +++ b/x-api/compliance/batch-compliance/integrate.mdx @@ -2,28 +2,131 @@ title: Integration guide sidebarTitle: Integration guide keywords: ["batch compliance integration", "compliance batch integration", "compliance jobs integration", "batch compliance setup"] -description: "When using the Batch compliance endpoints, developers can batch upload large amounts of. Reference for the X API v2 standard tier covering batch compliance." +description: "Key concepts, URL lifecycle, error handling, and best practices for integrating the X API v2 Batch Compliance endpoints." --- import { Button } from '/snippets/button.mdx'; -## Working with resumable uploads +The Batch Compliance endpoints let you upload large datasets of Post IDs or +user IDs and learn what action is needed to keep your datasets aligned with +user intent and the current state of content on X. This guide covers the +concepts the [quickstart](/x-api/compliance/batch-compliance/quickstart) +glosses over: URL lifecycle, job lifecycle, error handling, and operational +best practices. -When using the Batch compliance endpoints, developers can batch upload large amounts of X data and understand what action is needed to ensure that their datasets reflect user intent and the current state of the content on X. Uploading large amounts of data to a remote server is a relatively straightforward operation when systems and connectivity are stable and reliable. However, this may not always be the case. Some environments may impose a connection timeout, effectively cutting the connection between your app and the upload server after a set amount of time; you may also encounter connection issues, for example when trying to upload a large file from your laptop over a wi-fi connection. In these circumstances, it’s desirable to upload smaller portions of that file at a time, rather than having one single continuous connection. +## Signed URLs -X's batch compliance endpoints rely on Google Cloud Storage to process large files. This type of storage is optimized for various applications; Cloud Storage supports a technique to manage large files called resumable uploads. +Creating a job returns two signed, single-purpose URLs on `api.x.com`: -If the upload goes wrong at any point, Google Cloud Storage is able to resume the operation from where it was left off. +| URL | Method | Purpose | Lifetime | +|:----|:-------|:--------|:---------| +| `upload_url` | PUT | Submit your ID file for this job | ~15 minutes from job creation | +| `download_url` | GET | Retrieve this job's results | ~7 days from job creation | -### Creating a resumable job +Each URL embeds a token that authorizes exactly one operation on exactly one +job until its expiry. Treat the URLs as secrets — anyone holding them can +perform that operation until they expire. Do not log them or embed them in +client-side code. -#### Step one: +The exact expiry instants are returned as `upload_expires_at` and +`download_expires_at` on the job object. If the upload window lapses before +you use it, cancel the job (`DELETE /2/compliance/jobs/:id`) and create a +new one — upload URLs are not refreshable. -First, you will have to create a compliance job and specify whether you will be uploading Post IDs or user IDs (using the type parameter). Additionally, add resumable to the body and set it to true. Make sure to replace the $APP\_ACCESS\_TOKEN below with your App only Access Token below. +## Job lifecycle ``` -curl --request POST \ - 'https://api.x.com/2/compliance/jobs' --header 'Authorization: Bearer $APP_ACCESS_TOKEN --header 'Content-Type: application/json' --data-raw '{ - "type": "tweets", - \ No newline at end of file +created ──upload──▶ in_progress ──processing──▶ complete + │ │ │ + └──── cancel ─────────┘ download + (or expiry) (~7 days) +``` + +- **One active job per type.** While a `tweets` job is `created` or + `in_progress`, creating another `tweets` job returns `409 Conflict`. A + `users` job can run concurrently with a `tweets` job. +- **Processing is asynchronous.** Poll `GET /2/compliance/jobs/:id` with + backoff (e.g. every 30–60 seconds). Processing time scales with dataset + size. +- **Jobs expire.** A job left in `created` without an upload expires. Jobs + can also end `failed`; both free the active-job slot. + +## Uploads + +The upload is a single `PUT` of a plain-text body, one numeric ID per line, +up to 1 GiB. There is no resumable/chunked mode: if a transfer is +interrupted, retry the whole PUT while the upload URL is valid, or cancel +the job and start over. For very large datasets, split IDs across multiple +sequential jobs — results are per-ID, so partitioning is safe. + +Invalid lines (non-numeric, malformed) don't fail the job; each comes back +in the results as an error record: + +```json +{"id": "not_a_valid_id", "error": "invalid_id"} +``` + +## Interpreting results + +Results are newline-delimited JSON. Three rules cover the contract: + +1. **Absence means clean.** An ID missing from the results has no + compliance event; your stored copy requires no action. +2. **`action` tells you what to do**: `delete` (remove your stored copy), + `rehydrate` (re-fetch the Post — it was edited and your copy is stale), + or `scrub_geo` (remove stored location data for the Post). +3. **One event per ID.** When several events apply, you receive the + highest-impact one: `delete` outranks `rehydrate`, which outranks + `scrub_geo`. + +Deletion events additionally carry `deleted_at` (an alias of `redacted_at`) +for compatibility with consumers of the legacy results format. `redacted_at` +can be `null` for events that predate timestamp tracking; the obligation to +act is the same. + +## Error handling + +Errors use RFC 7807 problem+json bodies. + +| Status | When | Handling | +|:-------|:-----|:---------| +| `400` | Malformed request (bad `type`, bad body) | Fix the request | +| `401` | Missing/invalid Bearer on the JSON endpoints, or unusable byte-path token | Check credentials / recreate the job | +| `403` | Valid but expired or wrong-purpose byte-path token; App not enrolled for the endpoint | Recreate the job; check your App's access level | +| `409` | A job of this type is already active | Wait for it, or cancel it | +| `413` | Upload exceeds 1 GiB | Split the dataset across jobs | + +## Best practices + +- **Run compliance jobs on a schedule.** X requires stored content to + reflect current compliance state; a recurring (e.g. daily or weekly) job + over your stored IDs, acting on every returned event, is the intended + integration pattern. Pair batch jobs with the + [compliance streams](/x-api/compliance/streams/introduction) if you need + real-time signals between runs. +- **Upload immediately after creating a job.** The ~15-minute upload window + is comfortably enough for automation but not for manual workflows with + long pauses. +- **Store job IDs with your submissions.** `GET /2/compliance/jobs` lists + recent jobs, but correlating submissions to results is your side's + responsibility. +- **Download results promptly** and persist them; the download URL lapses + after ~7 days and results are not retrievable afterward. +- **Act on `rehydrate` via the Posts lookup endpoints** — re-fetching + returns the current revision of an edited Post along with its edit + history. + +## Next steps + + + + Create your first compliance job + + + Real-time compliance events + + + Full endpoint documentation + + diff --git a/x-api/compliance/batch-compliance/introduction.mdx b/x-api/compliance/batch-compliance/introduction.mdx index 191422aca..bd101d718 100644 --- a/x-api/compliance/batch-compliance/introduction.mdx +++ b/x-api/compliance/batch-compliance/introduction.mdx @@ -44,6 +44,9 @@ The Batch Compliance endpoints let you upload datasets of Post IDs or user IDs a | POST | [`/2/compliance/jobs`](/x-api/compliance/create-compliance-job) | Create a new compliance job | | GET | [`/2/compliance/jobs/:id`](/x-api/compliance/get-compliance-job-by-id) | Get job status | | GET | [`/2/compliance/jobs`](/x-api/compliance/get-compliance-jobs) | List all jobs | +| PUT | [`upload_url`](/x-api/compliance/upload-compliance-job-submission) | Upload your ID dataset (URL returned at job creation) | +| GET | [`download_url`](/x-api/compliance/download-compliance-job-results) | Download results (URL returned at job creation) | +| DELETE | [`/2/compliance/jobs/:id`](/x-api/compliance/cancel-compliance-job) | Cancel a job awaiting upload or processing | --- @@ -58,24 +61,28 @@ The Batch Compliance endpoints let you upload datasets of Post IDs or user IDs a ## Compliance events +Results carry an `action` — what you should do with your stored copy — and a +`reason` — why. + ### Post events -| Event | Description | -|:------|:------------| -| `deleted` | Post was deleted by user | -| `bounced` | Post failed compliance check | -| `protected` | Account became protected | -| `suspended` | Account was suspended | -| `scrub_geo` | Geo data was removed | +| Action | Reason | Description | +|:-------|:-------|:------------| +| `delete` | `deleted` | Post was deleted | +| `delete` | `protected` | Author's account became protected | +| `delete` | `deactivated` | Author's account was deactivated | +| `delete` | `suspended` | Author's account was suspended | +| `rehydrate` | `tweet_edited` | Post was edited; re-fetch it to get the current revision | +| `scrub_geo` | `geo_scrubbed` | Author removed geo data from their Posts; delete stored geo information | ### User events -| Event | Description | -|:------|:------------| -| `deleted` | Account was deleted | -| `suspended` | Account was suspended | -| `protected` | Account became protected | -| `deactivated` | Account was deactivated | +| Action | Reason | Description | +|:-------|:-------|:------------| +| `delete` | `deleted` | Account was deleted | +| `delete` | `protected` | Account became protected | +| `delete` | `deactivated` | Account was deactivated | +| `delete` | `suspended` | Account was suspended | --- diff --git a/x-api/compliance/batch-compliance/quickstart.mdx b/x-api/compliance/batch-compliance/quickstart.mdx index 9ef254d47..966dc1a9f 100644 --- a/x-api/compliance/batch-compliance/quickstart.mdx +++ b/x-api/compliance/batch-compliance/quickstart.mdx @@ -66,18 +66,30 @@ Before you begin, you'll need: "type": "tweets", "name": "my-compliance-job", "status": "created", - "upload_url": "https://storage.googleapis.com/...", - "download_url": "https://storage.googleapis.com/...", + "upload_url": "https://api.x.com/2/compliance/jobs/1234567890/upload?token=...", + "upload_expires_at": "2024-01-15T10:15:00.000Z", + "download_url": "https://api.x.com/2/compliance/jobs/1234567890/download?token=...", + "download_expires_at": "2024-01-22T10:00:00.000Z", "created_at": "2024-01-15T10:00:00.000Z" } } ``` - Save the `upload_url` and `download_url` for the next steps. + Save the `upload_url` and `download_url` for the next steps. Both are + signed, single-purpose URLs tied to this job: the `upload_url` expires + about **15 minutes** after job creation, and the `download_url` about + **7 days** after. If an upload URL expires before you use it, cancel the + job and create a new one. + + + You can have one active (`created` or `in_progress`) job per job type at + a time. Creating a second one returns `409 Conflict` until the first + completes, fails, or is canceled. + - Create a text file with one ID per line: + Create a plain-text file with one numeric ID per line (up to 1 GiB): ``` 1234567890 @@ -86,17 +98,22 @@ Before you begin, you'll need: 1234567893 ``` - Save as `ids.txt`. + Save as `ids.txt`. Lines that are not valid IDs are reported in the + results as error records rather than failing the whole job. - Upload the file to the provided `upload_url`: + Upload the file to the provided `upload_url` with a single PUT. The URL's + signed token is the credential — no other authentication is required: ```bash curl -X PUT "UPLOAD_URL_FROM_RESPONSE" \ -H "Content-Type: text/plain" \ --data-binary @ids.txt ``` + + A successful upload returns `200 OK` and the job moves to + `in_progress`. An expired token returns `403` with a problem+json body. @@ -119,20 +136,27 @@ Before you begin, you'll need: - Once status is `complete`, download from the `download_url`: + Once status is `complete`, download from the `download_url` (like the + upload URL, its signed token is the credential): ```bash - curl "DOWNLOAD_URL_FROM_RESPONSE" -o results.json + curl "DOWNLOAD_URL_FROM_RESPONSE" -o results.ndjson ``` - **Result format** (one JSON object per line): + **Result format** (newline-delimited JSON, one object per line): ```json - {"id": "1234567890", "action": "delete", "created_at": "2024-01-10T12:00:00.000Z", "redacted_at": "2024-01-12T08:30:00.000Z", "reason": "deleted"} - {"id": "1234567891", "action": "delete", "created_at": "2024-01-10T12:00:00.000Z", "redacted_at": "2024-01-13T14:20:00.000Z", "reason": "suspended"} + {"id": "1234567890", "action": "delete", "created_at": "2024-01-10T12:00:00.000Z", "redacted_at": "2024-01-12T08:30:00.000Z", "deleted_at": "2024-01-12T08:30:00.000Z", "reason": "deleted"} + {"id": "1234567891", "action": "rehydrate", "created_at": "2024-01-10T12:00:00.000Z", "redacted_at": "2024-01-13T14:20:00.000Z", "reason": "tweet_edited"} + {"id": "1234567892", "action": "scrub_geo", "created_at": "2018-05-27T10:00:00.000Z", "redacted_at": "2019-06-20T02:00:00.000Z", "reason": "geo_scrubbed"} + {"id": "not_a_valid_id", "error": "invalid_id"} ``` - Only IDs with compliance events appear in the results. IDs not in the results are still valid. + Only IDs with compliance events appear in the results. **An ID absent + from the results has no compliance event — your stored copy is fine.** + Invalid submitted lines come back as `{"id": ..., "error": "invalid_id"}` + records. `redacted_at` can be `null` when the original event predates + timestamp tracking. @@ -142,25 +166,30 @@ Before you begin, you'll need: - | Action | Reason | Description | - |:-------|:-------|:------------| - | `delete` | `deleted` | Post was deleted | - | `delete` | `bounced` | Post failed compliance check | - | `delete` | `protected` | Account became protected | - | `delete` | `suspended` | Account was suspended | - | `delete` | `scrub_geo` | Geo data was removed | + | Action | Reason | What to do | + |:-------|:-------|:-----------| + | `delete` | `deleted` | Delete your stored copy — the Post was deleted | + | `delete` | `protected` | Delete your stored copy — the author's account became protected | + | `delete` | `deactivated` | Delete your stored copy — the author's account was deactivated | + | `delete` | `suspended` | Delete your stored copy — the author's account was suspended | + | `rehydrate` | `tweet_edited` | Re-fetch the Post — it was edited and your copy is a stale revision | + | `scrub_geo` | `geo_scrubbed` | Delete stored geo data for the Post — the author removed location data | - | Action | Reason | Description | - |:-------|:-------|:------------| - | `delete` | `deleted` | Account was deleted | - | `delete` | `suspended` | Account was suspended | - | `delete` | `protected` | Account became protected | - | `delete` | `deactivated` | Account was deactivated | + | Action | Reason | What to do | + |:-------|:-------|:-----------| + | `delete` | `deleted` | Delete stored data for the account — it was deleted | + | `delete` | `protected` | Delete stored data — the account became protected | + | `delete` | `deactivated` | Delete stored data — the account was deactivated | + | `delete` | `suspended` | Delete stored data — the account was suspended | +When multiple events apply to the same ID, the results carry the +highest-impact one: `delete` outranks `rehydrate`, which outranks +`scrub_geo`. + --- ## List all jobs @@ -174,6 +203,18 @@ curl "https://api.x.com/2/compliance/jobs?type=tweets" \ --- +## Cancel a job + +Cancel a job that is awaiting upload or still processing (frees the +one-active-job slot for its type): + +```bash +curl -X DELETE "https://api.x.com/2/compliance/jobs/1234567890" \ + -H "Authorization: Bearer $BEARER_TOKEN" +``` + +--- + ## Next steps From 6319b29d349ece3603f56e327f222bdbc75a8388 Mon Sep 17 00:00:00 2001 From: Taylor Caldwell Date: Tue, 15 Sep 2026 17:00:02 -0700 Subject: [PATCH 2/2] Byte-path requests require the Bearer token Decision: upload/download URLs require the App's Authorization header in addition to the signed URL token, replacing the legacy GCS model where pre-signed URLs were self-authorizing. Migrating integrations must add the header; a migration note now flags this in both the quickstart and the integration guide. --- x-api/compliance/batch-compliance/integrate.mdx | 10 +++++++--- .../compliance/batch-compliance/quickstart.mdx | 17 +++++++++++++---- 2 files changed, 20 insertions(+), 7 deletions(-) diff --git a/x-api/compliance/batch-compliance/integrate.mdx b/x-api/compliance/batch-compliance/integrate.mdx index e9abf4aea..d023ab60a 100644 --- a/x-api/compliance/batch-compliance/integrate.mdx +++ b/x-api/compliance/batch-compliance/integrate.mdx @@ -24,9 +24,13 @@ Creating a job returns two signed, single-purpose URLs on `api.x.com`: | `upload_url` | PUT | Submit your ID file for this job | ~15 minutes from job creation | | `download_url` | GET | Retrieve this job's results | ~7 days from job creation | -Each URL embeds a token that authorizes exactly one operation on exactly one -job until its expiry. Treat the URLs as secrets — anyone holding them can -perform that operation until they expire. Do not log them or embed them in +Each URL embeds a token that binds it to exactly one operation on exactly one +job until its expiry. Requests to these URLs must also carry your +`Authorization: Bearer` header — the same App credential that created the job. +This differs from the previous Google Cloud Storage-based flow, where +pre-signed URLs were used without an Authorization header; if you are +migrating an existing integration, add the header to your upload and download +requests. Still treat the URLs as secrets: do not log them or embed them in client-side code. The exact expiry instants are returned as `upload_expires_at` and diff --git a/x-api/compliance/batch-compliance/quickstart.mdx b/x-api/compliance/batch-compliance/quickstart.mdx index 966dc1a9f..0a33a8766 100644 --- a/x-api/compliance/batch-compliance/quickstart.mdx +++ b/x-api/compliance/batch-compliance/quickstart.mdx @@ -103,15 +103,22 @@ Before you begin, you'll need: - Upload the file to the provided `upload_url` with a single PUT. The URL's - signed token is the credential — no other authentication is required: + Upload the file to the provided `upload_url` with a single PUT, + authenticating with the same Bearer token used to create the job: ```bash curl -X PUT "UPLOAD_URL_FROM_RESPONSE" \ + -H "Authorization: Bearer $BEARER_TOKEN" \ -H "Content-Type: text/plain" \ --data-binary @ids.txt ``` + + Unlike the previous Google Cloud Storage-based flow, upload and download + URLs now require your `Authorization` header in addition to the signed + token embedded in the URL. + + A successful upload returns `200 OK` and the job moves to `in_progress`. An expired token returns `403` with a problem+json body. @@ -137,10 +144,12 @@ Before you begin, you'll need: Once status is `complete`, download from the `download_url` (like the - upload URL, its signed token is the credential): + upload, this requires your Bearer token): ```bash - curl "DOWNLOAD_URL_FROM_RESPONSE" -o results.ndjson + curl "DOWNLOAD_URL_FROM_RESPONSE" \ + -H "Authorization: Bearer $BEARER_TOKEN" \ + -o results.ndjson ``` **Result format** (newline-delimited JSON, one object per line):