Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
865a4f5
SK-3075: add Detect V2 to flowvault and skyvault SDKs
saileshwar-skyflow Oct 26, 2026
05b2e1d
SK-3075: fix generated version fallback and refresh API contract base…
saileshwar-skyflow Oct 26, 2026
8774fe4
SK-3075: make common detect tests unittest-compatible
saileshwar-skyflow Oct 26, 2026
e1c3e28
SK-3075: document Detect V2 for skyvault
promptless[bot] Oct 26, 2026
a976075
SK-3075: fix API reference link in skyvault samples README
promptless[bot] Oct 26, 2026
fad619e
SK-3075: split detect DataSourceType into per-SDK DeidentifySourceTyp…
saileshwar-skyflow Nov 2, 2026
80d0d18
SK-3075: rename the Detect README section heading to "Detect"
saileshwar-skyflow Nov 3, 2026
56a66ec
SK-3076 add v1-deprecation warning (#288)
skyflow-bharti Nov 3, 2026
22d6b35
SK-3075: fix auth, version, and skyvault data-plane after codegen regen
saileshwar-skyflow Nov 3, 2026
1ccf503
SK-3075: refresh API contract baselines for DeidentifySourceType/Reid…
saileshwar-skyflow Nov 3, 2026
d4e8d51
Merge remote-tracking branch 'origin/main' into saileshwar/SK-3075-ad…
saileshwar-skyflow Nov 3, 2026
4c58420
SK-3075: update skyvault unit tests for the new-codegen data-plane an…
saileshwar-skyflow Nov 3, 2026
e13cffd
SK-3075: split detect RedactionLevel into a per-SDK rule (match Java)
saileshwar-skyflow Nov 3, 2026
893a976
SK-3075: fix common detect test for the RedactionLevel split
saileshwar-skyflow Nov 3, 2026
28905b7
SK-3075: make the skyvault sdist self-contained
saileshwar-skyflow Nov 3, 2026
12bdad5
SK-3075: regenerate skyvault files client for multipart file upload
saileshwar-skyflow Nov 4, 2026
deaa724
SK-3075: restore working skyvault generated version.py
saileshwar-skyflow Nov 4, 2026
4bb4129
SK-3075: re-trigger PR CI on the restored generated version.py
saileshwar-skyflow Nov 4, 2026
9ef1103
SK-3075: merge latest Detect V2 source branch into docs branch
promptless[bot] Nov 4, 2026
f58da00
SK-3075: use DeidentifySourceType and ReidentifySourceType in skyvaul…
promptless[bot] Nov 4, 2026
7e82842
SK-3284: add detect V1 to V2 migration guide for skyvault python sdk
saileshwar-skyflow Nov 4, 2026
688c432
SK-3075: document Detect V2 in skyvault and flowvault READMEs and sam…
saileshwar-skyflow Nov 5, 2026
1862ec5
Merge remote-tracking branch 'origin/saileshwar/SK-3075-add-detect-v2…
promptless[bot] Nov 5, 2026
0f77402
SK-3075: align flowvault Detect V2 with Java and add detect to flowva…
saileshwar-skyflow Nov 5, 2026
5831909
SK-3075: expose includePrefix / prefixOverride on Detect Entity and C…
saileshwar-skyflow Nov 6, 2026
fb84c3f
Merge remote-tracking branch 'origin/saileshwar/SK-3075-add-detect-v2…
promptless[bot] Nov 6, 2026
1ccbea8
SK-3075: clarify that PollOptions wait_time caps backoff wait, not el…
promptless[bot] Nov 6, 2026
1e35b3a
SK-3075: Merge flowvault-release/26.10.0 into docs branch
promptless[bot] Nov 6, 2026
e511fe6
SK-3075: document Detect token prefix options and flowvault Detect di…
promptless[bot] Nov 6, 2026
99a4ea2
SK-3075: link the renamed Detect migration guide and drop the Detect …
promptless[bot] Nov 9, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
444 changes: 325 additions & 119 deletions docs/api_reference.md

Large diffs are not rendered by default.

31 changes: 25 additions & 6 deletions flowvault/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1005,7 +1005,7 @@ detect = skyflow_client.detect()
- **Role.** The service account needs the Detect Invoker, Detect Manager or Vault Owner role on the vault. Detect Invoker can de-identify data and write vault tokens into the vault; Detect Manager adds the ability to create, update and delete Detect configurations. Re-identifying values stored as `VAULT_TOKEN` is governed like detokenization: the caller also needs a policy granting the Detokenization action on the destination, unless it is a Vault Owner.
- **Configuration.** Pass `configuration_id` (the id of a stored Detect configuration) or an inline `configuration` (a `DetectConfiguration`). Both are optional: with neither set the SDK sends only the vault id and the run uses the vault's default Detect settings; sending both is rejected by the API with HTTP 400.
- **Destination for vault tokens.** An entity de-identified with `VAULT_TOKEN` stores its value in the token group named by `destination`. That token group must already exist in the vault; Detect writes into it but does not create it.
- **API reference.** Every Detect request, response and configuration class is listed in the [API Reference](../docs/api_reference.md).
- **API reference.** Every Detect request, response and configuration class is listed in the [API Reference](../docs/api_reference.md). That reference is written for skyvault; its [Detect helper classes](../docs/api_reference.md#detect-helper-classes) section lists where flowvault differs.

## deidentify_string

Expand Down Expand Up @@ -1060,9 +1060,9 @@ Returns a `DeidentifyStringResponse` with `processed_text`, `entities` (each a `
| Class | Fields | Notes |
|-------|--------|-------|
| `Detect` | `entities`, `object_entities`, `restrict`, `skip`, `custom_types`, `return_entities` | `restrict` / `skip` are regexes always / never treated as sensitive. `return_entities` is `EXCLUDE_SENSITIVE_DATA` (server default), `ALL` or `NONE`. |
| `Entity` | `entity_type`, `deidentification_type`, `destination`, `transformation` | `entity_type` defaults to `ALL`. `deidentification_type` is `ENTITY_UNIQUE_COUNTER` (server default), `ENTITY_ONLY`, `MASK` or `VAULT_TOKEN`. `destination` (a token group name) is used only with `VAULT_TOKEN`. |
| `Entity` | `entity_type`, `deidentification_type`, `destination`, `transformation` | `entity_type` defaults to `ALL`. `deidentification_type` is `ENTITY_UNIQUE_COUNTER` (server default), `ENTITY_ONLY`, `MASK` or `VAULT_TOKEN`. `destination` (a token group name) is used only with `VAULT_TOKEN`. `include_prefix` and `prefix_override` are described below the table. |
| `Transformation` / `ShiftDates` | `shift_dates`; `min_days`, `max_days` | Shifts detected dates by a random number of days within the range. |
| `CustomType` | `label`, `match` | A regex-defined entity type; `label` is reported as the entity type of its matches. |
| `CustomType` | `label`, `match`, `deidentification_type`, `destination`, `include_prefix`, `prefix_override` | A regex-defined entity type; `label` is reported as the entity type of its matches. The other fields work as they do on `Entity`. |
| `ObjectEntity` | `entity_type`, `deidentification_type` | Visual objects in images: `FACE`, `LICENSE_PLATE`, `LOGO`, `SIGNATURE` or `ALL`, handled with `REDACT` or `UNREDACT`. File inputs only. |
| `FileMapping` | `source`, `destination`, `output_file_name_suffix`, `entities_destination`, `object_entities_destination` | Where the processed file and entity files are written. File inputs only. |
| `Media` | `audio`, `document`, `image` | Media-type options. File inputs only. |
Expand Down Expand Up @@ -1104,14 +1104,33 @@ configuration = DetectConfiguration(
)
```

**Token prefixes.** With `VAULT_TOKEN`, a generated token starts by default with the entity type, for example `[NAME_<token>]`. To change this, set `include_prefix` and `prefix_override` on an `Entity` or `CustomType`:
Comment thread
promptless[bot] marked this conversation as resolved.
Resolved

- `prefix_override='NAAM'` produces `[NAAM_<token>]`.
- `include_prefix=IncludePrefix.NO` returns the raw vault token. A token without its prefix can't be re-identified later, so use it only when you won't need the original value back.

You can't combine `prefix_override` with `include_prefix=IncludePrefix.NO`. When you set neither, the server default applies. Import `IncludePrefix` from `skyflow.utils.enums`.

```python
from skyflow.vault.detect import Entity
from skyflow.utils.enums import EntityType, DeidentificationType

Entity(
entity_type=EntityType.NAME,
deidentification_type=DeidentificationType.VAULT_TOKEN,
destination='<TOKEN_GROUP_NAME>',
prefix_override='NAAM',
)
```

## deidentify_file

De-identifies a file. The API is asynchronous: the submit call returns a run id and the result is retrieved from the run. Build the request with `DeidentifyFileRequest`.

Supply the file in exactly one of three ways: `file_path`, `file`, or `value` plus a `DeidentifySourceType` `data_source` (`BASE64`, the `SKYFLOW_ID` of a file record in the vault, or `PRESIGNED_URL`). A local `file` / `file_path` is read when the request is sent, base64-encoded, and its `data_format` inferred from the extension. Passing more than one of them, or none, fails client-side validation before any network call. `configuration_id` / `configuration` follow the same rules as for strings (see [Before you start](#before-you-start)).
Supply the file in exactly one of three ways: `file_path`, `file`, or `value` plus a `DeidentifySourceType` `data_source` (`BASE64` or `PRESIGNED_URL`; flowvault doesn't accept a Skyflow ID as the source). A local `file` / `file_path` is read when the request is sent, base64-encoded, and its `data_format` inferred from the extension. Passing more than one of them, or none, fails client-side validation before any network call. `configuration_id` / `configuration` follow the same rules as for strings (see [Before you start](#before-you-start)).
Comment thread
promptless[bot] marked this conversation as resolved.
Resolved

- **Submit only** (default): one call, returns `run_id`. `status` is `None` and nothing is written to disk — retrieve the result later with [`get_run`](#get_run).
- **Submit and poll**: pass `poll_options`. The SDK polls with exponential backoff (1, 2, 4, 8, 16, then 32 seconds) until the run is `SUCCESS`, `FAILED` or `UNKNOWN`, or until `wait_time` (default 60 s, max 300) or `max_attempts` (default and max 23) is used up. If the budget runs out the response carries the run id with `IN_PROGRESS` or `QUEUED`, so you can resume with `get_run`. `output_directory` is valid only with `poll_options`; setting it without them fails client-side validation.
- **Submit and poll**: pass `poll_options`. The SDK polls with exponential backoff (1, 2, 4, 8, 16, then 32 seconds) until the run is `SUCCESS`, `FAILED` or `UNKNOWN`, or until `wait_time` (default 60 s, max 300) or `max_attempts` (default and max 23) is used up. `wait_time` caps the combined sleep between status checks, not the elapsed time, so polling can run longer than `wait_time`. If the budget runs out the response carries the run id with `IN_PROGRESS` or `QUEUED`, so you can resume with `get_run`. `output_directory` is valid only with `poll_options`; setting it without them fails client-side validation.
Comment thread
promptless[bot] marked this conversation as resolved.
Resolved

```python
from skyflow.vault.detect import DeidentifyFileRequest, PollOptions
Expand Down Expand Up @@ -1149,7 +1168,7 @@ else: # IN_PROGRESS or QUEUED: budget ran out, resume
- `/data/processed/processed-entities-patient-notes.json` — the detected entities
- `/data/processed/processed-object-entities-patient-notes.json` — object entities, when the run produced them

`output_directory` is created if it does not exist. **If you poll a local file and omit `output_directory`, these files are written to the current working directory** — often your project root or a server's working directory — so set `output_directory` explicitly. A request built from a `SKYFLOW_ID` or `PRESIGNED_URL` value has no local name to derive from, and presigned-URL outputs are not fetched, so in those cases read `response.output` and write the files yourself.
`output_directory` is created if it does not exist. **If you poll a local file and omit `output_directory`, these files are written to the current working directory** — often your project root or a server's working directory — so set `output_directory` explicitly. A request built from `data_source` and `value` has no local file name to derive from, and the SDK doesn't fetch presigned-URL outputs. In those cases, read `response.output` and write the files yourself.

**Supported formats and limits.** `data_format` accepts every `FileDataFormat`: `bmp`, `csv`, `dcm`, `doc`, `docx`, `gif`, `jpeg`, `jpg`, `json`, `jsonl`, `mp3`, `pdf`, `png`, `ppt`, `pptx`, `tif`, `tiff`, `txt`, `wav`, `xls`, `xlsx`, `xml`, `zip`. The `VAULT_TOKEN` de-identification type is supported only for `txt`, `csv`, `json`, `jsonl`, `xml`, `docx`, `xlsx` and `pptx`.

Expand Down
46 changes: 43 additions & 3 deletions skyvault/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -569,6 +569,9 @@ Detect configurations are created in Studio or through the management API — th
detect = skyflow_client.detect()
```

> [!NOTE]
> skyflow 3.0.0 replaces the `deidentify_text`, `reidentify_text`, and `get_detect_run` methods and their request and response classes with `deidentify_string`, `reidentify_string`, and `get_run`. `deidentify_file` keeps its name but takes a new request. To upgrade existing code, see the [Detect migration guide](../docs/migrate_detect.md).
Comment thread
promptless[bot] marked this conversation as resolved.
Resolved

### Before you start

- **Role.** The service account needs the Detect Invoker, Detect Manager or Vault Owner role on the vault. Detect Invoker can de-identify data and write vault tokens into the vault; Detect Manager adds the ability to create, update and delete Detect configurations. Re-identifying values stored as `VAULT_TOKEN` is governed like detokenization: the caller also needs a policy granting the Detokenization action on the destination table and columns, unless it is a Vault Owner.
Expand Down Expand Up @@ -622,16 +625,19 @@ response = skyflow_client.detect().deidentify_string(

Returns a `DeidentifyStringResponse` with `processed_text`, `entities` (each a `DetectedEntity` with `token`, `value`, `location`, `entity_type`, `entity_scores`; never `None`) and `metrics`. `value` is populated only when the configuration sets `return_entities` to `ReturnEntitiesType.ALL`; the default `EXCLUDE_SENSITIVE_DATA` returns an empty `value`, so raw sensitive data does not come back unless you ask for it.

> [!TIP]
> See the full example in the samples directory: [deidentify_string.py](samples/detect_api/deidentify_string.py)

### Detect configuration

`DetectConfiguration` is the inline form of a stored configuration and accepts the same options. `detect` applies to string and file inputs; `file_mapping` and `media` apply to file inputs only.

| Class | Fields | Notes |
|-------|--------|-------|
| `Detect` | `entities`, `object_entities`, `restrict`, `skip`, `custom_types`, `return_entities` | `restrict` / `skip` are regexes always / never treated as sensitive. `return_entities` is `EXCLUDE_SENSITIVE_DATA` (server default), `ALL` or `NONE`. |
| `Entity` | `entity_type`, `deidentification_type`, `destination`, `transformation` | `entity_type` defaults to `ALL`. `deidentification_type` is `ENTITY_UNIQUE_COUNTER` (server default), `ENTITY_ONLY`, `MASK` or `VAULT_TOKEN`. `destination` (`table.column`) is used only with `VAULT_TOKEN`. |
| `Entity` | `entity_type`, `deidentification_type`, `destination`, `transformation` | `entity_type` defaults to `ALL`. `deidentification_type` is `ENTITY_UNIQUE_COUNTER` (server default), `ENTITY_ONLY`, `MASK` or `VAULT_TOKEN`. `destination` (`table.column`) is used only with `VAULT_TOKEN`. `include_prefix` and `prefix_override` are described below the table. |
| `Transformation` / `ShiftDates` | `shift_dates`; `min_days`, `max_days` | Shifts detected dates by a random number of days within the range. |
| `CustomType` | `label`, `match` | A regex-defined entity type; `label` is reported as the entity type of its matches. |
| `CustomType` | `label`, `match`, `deidentification_type`, `destination`, `include_prefix`, `prefix_override` | A regex-defined entity type; `label` is reported as the entity type of its matches. The other fields work as they do on `Entity`. |
| `ObjectEntity` | `entity_type`, `deidentification_type` | Visual objects in images: `FACE`, `LICENSE_PLATE`, `LOGO`, `SIGNATURE` or `ALL`, handled with `REDACT` or `UNREDACT`. File inputs only. |
| `FileMapping` | `source`, `destination`, `output_file_name_suffix`, `entities_destination`, `object_entities_destination` | Where the processed file and entity files are written. File inputs only. |
| `Media` | `audio`, `document`, `image` | Media-type options. File inputs only. |
Expand Down Expand Up @@ -673,14 +679,33 @@ configuration = DetectConfiguration(
)
```

**Token prefixes.** With `VAULT_TOKEN`, a generated token starts by default with the entity type, for example `[NAME_<token>]`. To change this, set `include_prefix` and `prefix_override` on an `Entity` or `CustomType`:
Comment thread
promptless[bot] marked this conversation as resolved.
Resolved

- `prefix_override='NAAM'` produces `[NAAM_<token>]`.
- `include_prefix=IncludePrefix.NO` returns the raw vault token. A token without its prefix can't be re-identified later, so use it only when you won't need the original value back.

You can't combine `prefix_override` with `include_prefix=IncludePrefix.NO`. When you set neither, the server default applies. Import `IncludePrefix` from `skyflow.utils.enums`.

```python
from skyflow.vault.detect import Entity
from skyflow.utils.enums import EntityType, DeidentificationType

Entity(
entity_type=EntityType.NAME,
deidentification_type=DeidentificationType.VAULT_TOKEN,
destination='persons.name',
prefix_override='NAAM',
)
```

### De-identify File: `.deidentify_file(request)`

De-identifies a file. The API is asynchronous: the submit call returns a run id and the result is retrieved from the run. Build the request with `DeidentifyFileRequest`.

Supply the file in exactly one of three ways: `file_path`, `file`, or `value` plus a `DeidentifySourceType` `data_source` (`BASE64`, the `SKYFLOW_ID` of a file record in the vault, or `PRESIGNED_URL`). A local `file` / `file_path` is read when the request is sent, base64-encoded, and its `data_format` inferred from the extension. Passing more than one of them, or none, fails client-side validation before any network call. `configuration_id` / `configuration` follow the same rules as for strings (see [Before you start](#before-you-start)).

- **Submit only** (default): one call, returns `run_id`. `status` is `None` and nothing is written to disk — retrieve the result later with [`get_run`](#get-run-get_runrequest).
- **Submit and poll**: pass `poll_options`. The SDK polls with exponential backoff (1, 2, 4, 8, 16, then 32 seconds) until the run is `SUCCESS`, `FAILED` or `UNKNOWN`, or until `wait_time` (default 60 s, max 300) or `max_attempts` (default and max 23) is used up. If the budget runs out the response carries the run id with `IN_PROGRESS` or `QUEUED`, so you can resume with `get_run`. `output_directory` is valid only with `poll_options`; setting it without them fails client-side validation.
- **Submit and poll**: pass `poll_options`. The SDK polls with exponential backoff (1, 2, 4, 8, 16, then 32 seconds) until the run is `SUCCESS`, `FAILED` or `UNKNOWN`, or until `wait_time` (default 60 s, max 300) or `max_attempts` (default and max 23) is used up. `wait_time` caps the combined sleep between status checks, not the elapsed time, so polling can run longer than `wait_time`. If the budget runs out the response carries the run id with `IN_PROGRESS` or `QUEUED`, so you can resume with `get_run`. `output_directory` is valid only with `poll_options`; setting it without them fails client-side validation.
Comment thread
promptless[bot] marked this conversation as resolved.
Resolved

```python
from skyflow.vault.detect import DeidentifyFileRequest, PollOptions
Expand Down Expand Up @@ -742,6 +767,9 @@ skyflow_client.detect().deidentify_file(

Returns a `DeidentifyFileResponse` with `run_id`, `status` (a `DetectRunStatus`, `None` when not polled), `output_type`, `output` (a list of `FileOutput` with `processed_file`, `processed_file_type`, `processed_file_extension`), `metrics` and `message`.

> [!TIP]
> See the full example in the samples directory: [deidentify_file.py](samples/detect_api/deidentify_file.py)

### Get Run: `.get_run(request)`

Retrieves the status and, once complete, the outputs of a Detect file run. Build the request with `GetRunRequest`, initialized with the `run_id` returned from a prior `deidentify_file` call. `output` is empty and `output_type` / `metrics` are `None` until the run completes.
Expand All @@ -763,6 +791,9 @@ else: # QUEUED, IN_PROGRESS, UNKNOWN: poll again late

Returns a `GetRunResponse` with the same fields as `DeidentifyFileResponse`.

> [!TIP]
> See the full example in the samples directory: [get_run.py](samples/detect_api/get_run.py)

### Re-identify String: `.reidentify_string(request)`

Synchronous. Re-identification looks tokens up in the vault, so it applies only to values de-identified with `VAULT_TOKEN`. Placeholders produced by `ENTITY_UNIQUE_COUNTER`, `ENTITY_ONLY` or `MASK` are not stored and cannot be re-identified.
Expand Down Expand Up @@ -790,6 +821,9 @@ print(response.processed_text)

Returns a `ReidentifyStringResponse` with `processed_text` and `metrics`.

> [!TIP]
> See the full example in the samples directory: [reidentify_string.py](samples/detect_api/reidentify_string.py)

### Re-identify File: `.reidentify_file(request)`
Comment thread
promptless[bot] marked this conversation as resolved.
Resolved

Synchronous — the re-identified file comes back in the response, not through a run id. Build the request with `ReidentifyFileRequest`: a `data_source` (`ReidentifySourceType.BASE64` or `PRESIGNED_URL`; re-identification does not accept a skyflow id as the source), the matching `value`, an optional `data_format`, and the same `RedactionLevel` rules as `reidentify_string`. `data_format` is limited to the formats that support `VAULT_TOKEN`: `txt`, `csv`, `json`, `jsonl`, `xml`, `docx`, `xlsx` and `pptx`.
Expand Down Expand Up @@ -820,6 +854,9 @@ if response.status == DetectRunStatus.SUCCESS:

Returns a `ReidentifyFileResponse` with `status`, `output_type`, `output` (a list of `FileOutput`) and `metrics`.

> [!TIP]
> See the full example in the samples directory: [reidentify_file.py](samples/detect_api/reidentify_file.py)

### Check Guardrails: `.check_guardrails(request)`

Synchronous. Screens text for toxicity and a list of denied topics — a natural pre-LLM check. Build the request with `CheckGuardrailsRequest`. `check_toxicity` and `deny_topics` are optional; the SDK sends each only when set and fills in no default. The response is passed through as the API returns it.
Expand All @@ -838,6 +875,9 @@ if response.validation == GuardrailsValidation.FAILED:

Returns a `CheckGuardrailsResponse` with `text`, `validation` (`GuardrailsValidation.PASSED` or `FAILED`), `toxic` and `denied_topic`.

> [!TIP]
> See the full example in the samples directory: [check_guardrails.py](samples/detect_api/check_guardrails.py)

All Detect methods raise `skyflow.error.SkyflowError` on validation failure, credential failure, or an API error — see [Error handling](#error-handling). Client-side validation fails before any network call.

## Connections
Expand Down
Loading
Loading