Add validate command and release 1.0.0

.github/workflows/ci.yml

@@ -24,3 +24,31 @@ jobs:
           pip install -e ".[dev]"
       - name: Run tests
         run: pytest
+
+  package:
+    runs-on: ubuntu-latest
+    needs: test
+    steps:
+      - uses: actions/checkout@v4
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - name: Build sdist and wheel
+        run: |
+          python -m pip install --upgrade pip build
+          python -m build
+      - name: Install wheel in a clean environment
+        run: |
+          python -m venv /tmp/wheel-env
+          /tmp/wheel-env/bin/pip install dist/*.whl
+      - name: Smoke-test the installed command
+        run: |
+          /tmp/wheel-env/bin/data-transform --version
+          echo '[{"id":1},{"id":2}]' | /tmp/wheel-env/bin/data-transform validate -r id -u id -f json -q
+          printf 'id\n1\n1\n' | /tmp/wheel-env/bin/data-transform validate -u id -f csv -q && exit 1 || test $? -eq 1
+      - name: Upload distributions
+        uses: actions/upload-artifact@v4
+        with:
+          name: dist
+          path: dist/

CHANGELOG.md

@@ -1,6 +1,19 @@
 # Changelog
 
-All notable changes to this project are listed here. Versions follow [Semantic Versioning](https://semver.org/); until 1.0.0, minor versions may change CLI behaviour.
+All notable changes to this project are listed here. Versions follow [Semantic Versioning](https://semver.org/). From 1.0.0, documented commands, options, exit codes, and output shapes only break in a new major version.
+
+## [1.0.0] - 2026-10-16
+
+### Added
+
+- `validate [file]` checks a list of records (or one object) and exits `0` when valid, `1` when not, and `2` when the check can't run.
+  - `-r, --require PATH`, `--type PATH=TYPE[|TYPE...]`, and `-u, --unique PATH`, all repeatable and using dotted paths.
+  - `-s, --schema FILE` reads `required`, `types`, and `unique` from a JSON or YAML file; command-line flags are added on top.
+  - `--coerce` lets numeric, `true` / `false`, and empty strings satisfy `integer`, `number`, `boolean`, and `null`, for CSV input.
+  - `--max-errors N` and `-q, --quiet`. `-t csv` / `-t ndjson` writes only the error rows.
+- `validate_rows` and `load_schema` in the library API.
+- CI now builds the sdist and wheel, installs the wheel in a clean virtual environment, and smoke-tests the `data-transform` command.
+- A Stability section in the README describing what 1.0.0 commits to.
 
 ## [0.8.0] - 2026-10-14
 

README.md

@@ -40,6 +40,7 @@ After install, the `data-transform` console script is on your `PATH`.
 | `count` | Count items, per KEY value, or per group |
 | `stats` | count / sum / min / max / avg of a numeric FIELD |
 | `join` | Join two arrays of objects on a top-level KEY (`--with` file) |
+| `validate` | Check records for required paths, types, and unique keys (exit code 0 / 1 / 2) |
 
 ### Options (most commands)
 
@@ -51,6 +52,7 @@ After install, the `data-transform` console script is on your `PATH`.
 - `--desc` — reverse sort order (`sort` only)
 - `--sep <text>` — separator between nested keys (`flatten` and `unflatten`, default `.`)
 - `-w, --with <file>`, `--how inner|left`, `--right-from <format>` — right-hand file, join type, and right file format (`join` only)
+- `-r, --require <path>`, `--type <path>=<type>`, `-u, --unique <path>`, `-s, --schema <file>`, `--coerce`, `--max-errors <n>`, `-q, --quiet` — checks and report options (`validate` only). See [Validate](#validate)
 
 Format is taken from `--from`, else the file extension (`.ndjson` / `.jsonl` → ndjson), else simple content heuristics.
 
@@ -311,6 +313,44 @@ cat users.json | data-transform join id -w teams.csv -f json \
 # → {"core":1,"web":1,"ops":1}
 ```
 
+### Validate
+
+`validate` checks data before it moves further down a pipeline. It works on a list of objects (each item is a record) or on a single object. Paths are dotted, like `select`, so `user.name` and `items.0.sku` work.
+
+- `-r, --require PATH` — the path must be present in every record. A `null` value counts as present; add a type check to rule it out.
+- `--type PATH=TYPE` — the value must be one of the listed types: `string`, `integer`, `number`, `boolean`, `object`, `array`, `null`. Join several with `|`, for example `email=string|null`. Records without the path are skipped, so pair it with `--require` when the path is mandatory. Booleans are never numbers, and `integer` also satisfies `number`.
+- `-u, --unique PATH` — the value must not repeat across records. Values compare like `group-by` labels, so `1` and `"1"` count as the same. Records without the path are skipped.
+- `-s, --schema FILE` — a JSON or YAML file with any of `required` (list), `types` (path → type, or a list of types), and `unique` (list). Flags on the command line are added on top.
+- `--coerce` — let strings that look like integers, numbers, `true` / `false`, or an empty string satisfy `integer`, `number`, `boolean`, or `null`. Use it for CSV, where every cell is a string.
+- `--max-errors N` — list at most `N` errors. `error_count` is always the full count.
+- `-q, --quiet` — print nothing and rely on the exit code.
+
+The report is `{"valid", "records", "error_count", "errors"}`, where each error has `record` (the list index, or `null` for an object root), `path`, and `message`. `-t yaml` writes the report as YAML; `-t csv` or `-t ndjson` writes just the errors, one row per error, which is handy as a CI artifact.
+
+Exit codes: `0` when the data is valid, `1` when it is not, and `2` when the check could not run (unreadable input, a bad schema, an unknown type, or no checks given).
+
+```bash
+echo '[{"id":1,"email":"a@x.io"},{"id":"2"},{"id":1,"email":null}]' \
+  | data-transform validate -r id -r email --type id=integer --type 'email=string|null' -u id -f json --minify
+# → {"valid":false,"records":3,"error_count":3,"errors":[
+#     {"record":1,"path":"email","message":"missing required value"},
+#     {"record":1,"path":"id","message":"expected integer, got string"},
+#     {"record":2,"path":"id","message":"duplicate value '1' (first seen in record 0)"}]}
+
+# Data contract kept next to the pipeline
+cat > users.schema.yaml <<'YAML'
+required: [id, email]
+types:
+  id: integer
+  email: [string, "null"]
+unique: [id]
+YAML
+data-transform validate -s users.schema.yaml --coerce users.csv -q || echo "users.csv failed checks"
+
+# Fail a CI step and keep an error table
+data-transform validate -s users.schema.yaml --coerce users.csv -t csv -o validation-errors.csv
+```
+
 ### Library use
 
 ```python
@@ -333,6 +373,7 @@ from data_transform import (
     count_rows,
     stats_rows,
     join_rows,
+    validate_rows,
 )
 
 yaml_text = convert('{"a":1}', "json", "yaml")
@@ -353,6 +394,7 @@ total = count_rows([{"id": 1}, {"id": 2}])
 per_team = count_rows([{"team": "a"}, {"team": "a"}], "team")
 summary = stats_rows([{"ms": 120}, {"ms": 80}], "ms")
 joined = join_rows([{"id": 1, "name": "Ada"}], [{"id": "1", "team": "core"}], "id")
+report = validate_rows([{"id": 1}, {"id": "x"}], required=["id"], types={"id": ["integer"]})
 ```
 
 ## NDJSON notes
@@ -363,6 +405,10 @@ NDJSON (also known as JSON Lines / `.jsonl`) is one JSON value per non-empty lin
 
 CSV support is intentionally small and robust (RFC 4180-ish) via the Python standard library `csv` module: quoted fields, escaped quotes (`""`), and newlines inside quotes. Object arrays use a header row; all cell values are strings when parsed from CSV.
 
+## Stability
+
+From 1.0.0 the command names, options, exit codes, and output shapes documented here follow [Semantic Versioning](https://semver.org/): they only change in a breaking way in a new major version. New commands and options may arrive in minor versions. The functions exported from `data_transform` follow the same rule; anything starting with `_` is internal.
+
 ## Changelog
 
 See [CHANGELOG.md](./CHANGELOG.md).

pyproject.toml

@@ -4,13 +4,13 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "data-transform-cli"
-version = "0.8.0"
+version = "1.0.0"
 description = "CLI for YAML ↔ JSON ↔ CSV ↔ NDJSON transforms for pipelines and shell use"
 readme = "README.md"
 license = { text = "MIT" }
 requires-python = ">=3.11"
 authors = [{ name = "rmkr-dev" }]
-keywords = ["cli", "yaml", "json", "csv", "ndjson", "transform", "pipeline"]
+keywords = ["cli", "yaml", "json", "csv", "ndjson", "transform", "pipeline", "validation"]
 dependencies = [
   "click>=8.1",
   "PyYAML>=6.0",

src/data_transform/__init__.py

@@ -19,6 +19,7 @@
     unflatten_object,
     unique_rows,
 )
+from .validate import load_schema, validate_rows
 
 __all__ = [
     "convert",
@@ -45,6 +46,8 @@
     "count_rows",
     "stats_rows",
     "join_rows",
+    "validate_rows",
+    "load_schema",
 ]
 
-__version__ = "0.8.0"
+__version__ = "1.0.0"

src/data_transform/cli.py

@@ -27,6 +27,7 @@
     unflatten_object,
     unique_rows,
 )
+from .validate import load_schema, parse_type_option, validate_rows
 
 FORMAT_CHOICES = ["json", "yaml", "csv", "ndjson"]
 
@@ -879,5 +880,127 @@ def join_cmd(
         sys.exit(1)
 
 
+@main.command("validate")
+@click.argument("file", required=False)
+@click.option(
+    "-r",
+    "--require",
+    "required",
+    multiple=True,
+    metavar="PATH",
+    help="dotted path every record must have (repeatable)",
+)
+@click.option(
+    "--type",
+    "type_specs",
+    multiple=True,
+    metavar="PATH=TYPE",
+    help="allowed type(s) for PATH, e.g. id=integer or email=string|null (repeatable)",
+)
+@click.option(
+    "-u",
+    "--unique",
+    multiple=True,
+    metavar="PATH",
+    help="dotted path whose values must not repeat across records (repeatable)",
+)
+@click.option(
+    "-s",
+    "--schema",
+    "schema_file",
+    default=None,
+    metavar="FILE",
+    help="JSON or YAML file with required / types / unique (merged with flags)",
+)
+@click.option(
+    "--coerce",
+    is_flag=True,
+    default=False,
+    help="let numeric / true / false / empty strings satisfy integer, number, "
+    "boolean, null (useful for CSV)",
+)
+@click.option(
+    "--max-errors",
+    type=click.IntRange(min=0),
+    default=None,
+    help="list at most N errors in the report (error_count stays exact)",
+)
+@click.option(
+    "-q", "--quiet", is_flag=True, default=False, help="print nothing; use the exit code"
+)
+@shaping_options
+def validate_cmd(
+    file: Optional[str],
+    required: tuple[str, ...],
+    type_specs: tuple[str, ...],
+    unique: tuple[str, ...],
+    schema_file: Optional[str],
+    coerce: bool,
+    max_errors: Optional[int],
+    quiet: bool,
+    output: Optional[str],
+    from_format: Optional[str],
+    to_format: Optional[str],
+    minify: bool,
+) -> None:
+    """Check records against required paths, types, and unique keys.
+
+    Prints a report with valid, records, error_count, and errors. With
+    --to csv or --to ndjson, prints just the errors, one row per error.
+    Types: string, integer, number, boolean, object, array, null.
+
+    Exit codes: 0 valid, 1 invalid, 2 the check could not run (bad input,
+    schema, or options).
+
+    Usage: data-transform validate [--require PATH] [--type PATH=TYPE]
+    [--unique PATH] [--schema FILE] [file]
+    """
+    try:
+        req: list[str] = []
+        types: dict[str, list[str]] = {}
+        uniq: list[str] = []
+        if schema_file:
+            schema_path = Path(schema_file)
+            if not schema_path.is_file():
+                raise ValueError(f"schema file not found: {schema_file}")
+            schema_text = schema_path.read_text(encoding="utf-8")
+            schema_src = resolve_from(schema_text, schema_file, None)
+            req, types, uniq = load_schema(parse(schema_text, schema_src))  # type: ignore[arg-type]
+        req += [p for p in required if p not in req]
+        for spec in type_specs:
+            path, names = parse_type_option(spec)
+            types[path] = names
+        uniq += [p for p in unique if p not in uniq]
+        if not (req or types or uniq):
+            raise ValueError(
+                "nothing to check; pass --require, --type, --unique, or --schema"
+            )
+
+        text, name = load_input(file)
+        src = resolve_from(text, name, from_format)
+        data = parse(text, src)  # type: ignore[arg-type]
+        report = validate_rows(
+            data,
+            required=req,
+            types=types,
+            unique=uniq,
+            coerce=coerce,
+            max_errors=max_errors,
+        )
+    except Exception as err:  # noqa: BLE001
+        click.echo(f"error: {err}", err=True)
+        sys.exit(2)
+
+    if not quiet:
+        try:
+            fmt = resolve_to(to_format, output)
+            result = report["errors"] if fmt in ("csv", "ndjson") else report
+            _emit(result, output, fmt, minify)
+        except Exception as err:  # noqa: BLE001
+            click.echo(f"error: {err}", err=True)
+            sys.exit(2)
+    sys.exit(0 if report["valid"] else 1)
+
+
 if __name__ == "__main__":
     main()