Authenticate with Authorization: Bearer for programmatic tokens

.dockerignore

@@ -4,3 +4,4 @@
 docker-compose.yml
 Dockerfile
 README.md
+.venv/

.flake8

@@ -1,2 +1,3 @@
 [flake8]
 max-line-length = 120
+extend-exclude = .venv

.gitignore

@@ -3,3 +3,4 @@ __pycache__/
 /.coverage
 /kbcstorage.egg-info
 /dist
+.venv/

Dockerfile

@@ -3,6 +3,5 @@ FROM python:3.11
 WORKDIR /code
 COPY . /code/
 RUN python -m pip install --upgrade pip
-RUN pip install --no-cache-dir flake8
-RUN pip install --no-cache-dir .
+RUN pip install --no-cache-dir ".[dev]"
 ENTRYPOINT ["python"]

README.md

@@ -18,9 +18,9 @@ pip install git+https://github.com/keboola/sapi-python-client.git
 
 ## Client Class Usage
 ```python
-from kbcstorage.client import Client
+from kbcstorage.client import Client, StorageApiToken
 
-client = Client('https://connection.keboola.com', 'your-token')
+client = Client('https://connection.keboola.com', StorageApiToken('your-token'))
 
 # get table data into local file
 client.tables.export_to_file(table_id='in.c-demo.some-table', path_name='/data/')
@@ -39,12 +39,50 @@ client.tables.detail('in.c-demo.some-table')
 
 ```
 
+## Authentication
+
+The `token` argument takes an authentication strategy, either `StorageApiToken` or `BearerToken`:
+
+```python
+from kbcstorage.client import Client, BearerToken, StorageApiToken
+
+# Storage API token, sent as `X-StorageApi-Token`
+client = Client('https://connection.keboola.com', StorageApiToken('your-token'))
+
+# Programmatic token, sent as `Authorization: Bearer` with the project id in `X-KBC-ProjectId`
+client = Client('https://connection.keboola.com', BearerToken('kbc_at_...', project_id=1234))
+```
+
+A bare string is also accepted and is equivalent to `StorageApiToken('your-token')`. It is fully
+supported and not deprecated:
+
+```python
+client = Client('https://connection.keboola.com', 'your-token')
+```
+
+Storage API tokens are bound to a project. Programmatic tokens (`kbc_at_*` session tokens,
+`kbc_pat_*` personal access tokens) are not, which is why `BearerToken` requires the project id.
+
+All three forms work for directly constructed endpoints too, and the token is used as given — the
+client never refreshes or decodes it, so an expired token surfaces as a `requests.HTTPError` with a
+401.
+
+### Breaking change in 0.11.0
+
+A bare string starting with `kbc_at_` or `kbc_pat_` now raises `ValueError` at construction. Before
+0.11.0 it was sent as `X-StorageApi-Token`, which the backend rejects with a 401 — the failure moved
+from request time to construction time and the message names the fix. Wrap such tokens in
+`BearerToken(token, project_id)`.
+
 ## Endpoint Classes Usage 
 ```python
+from kbcstorage.auth import StorageApiToken
 from kbcstorage.tables import Tables
 from kbcstorage.buckets import Buckets
 
-tables = Tables('https://connection.keboola.com', 'your-token')
+token = StorageApiToken('your-token')
+
+tables = Tables('https://connection.keboola.com', token)
 
 # get table data into local file
 tables.export_to_file(table_id='in.c-demo.some-table', path_name='/data/')
@@ -53,7 +91,7 @@ tables.export_to_file(table_id='in.c-demo.some-table', path_name='/data/')
 tables.create(name='some-table-2', bucket_id='in.c-demo', file_path='/data/some-table')
 
 # list buckets
-buckets = Buckets('https://connection.keboola.com', 'your-token')
+buckets = Buckets('https://connection.keboola.com', token)
 buckets.list()
 
 # list bucket tables
@@ -64,11 +102,74 @@ tables.detail('in.c-demo.some-table')
 
 ```
 
+## Development
+
+### Local environment
+
+The package requires Python 3.10 or newer (`workspaces.load_tables` uses `int | str` annotations); CI builds
+and tests on 3.11.
+
+Create a virtual environment and install the package into it in editable mode:
+
+```bash
+python3 -m venv .venv
+.venv/bin/pip install -e ".[dev]"
+```
+
+`pyproject.toml` declares the test dependencies alongside the runtime ones, and the `dev` extra adds
+the linter, so this single command covers everything. The `-e` flag keeps `kbcstorage` resolving to
+the working tree, so edits take effect without reinstalling.
+
+Prefixing commands with `.venv/bin/` always uses the environment. Activating it instead makes the
+plain `python` and `pip` commands resolve there for the rest of the shell session:
+
+```bash
+source .venv/bin/activate
+```
+
+On Windows the paths are `.venv\Scripts\python` and `.venv\Scripts\activate`.
+
+### Editor setup
+
+Editors and IDEs resolve imports through a configured interpreter. Until yours is pointed at the
+virtual environment, every third-party import is reported as unresolved. The interpreter path is:
+
+```
+.venv/bin/python
+```
+
+| Editor | Where to set it |
+| --- | --- |
+| VS Code | `Python: Select Interpreter` |
+| PyCharm | Settings → Project → Python Interpreter → Add → Existing environment |
+| Zed | `toolchain: select` |
+| Vim, Neovim, Emacs | the interpreter or `venv` option of your LSP client |
+
+Most editors scan for a directory named `.venv` or `venv` in the project root, so keeping the default
+name avoids manual configuration in the common case. A restart of the language server is often
+needed before existing errors clear.
+
+### Checks
+
+Lint. CI runs this before the tests, so a failure stops the build before anything is executed:
+
+```bash
+.venv/bin/python -m flake8
+```
+
+Unit tests, which mock all API responses and need no credentials:
+
+```bash
+.venv/bin/python -m unittest discover -s tests/mocks -t .
+```
+
+The functional tests run against a real project and are covered below.
+
 ## Tests
 Create `.env` file according to the `.env.template` file and run the tests with:
 
 ```bash
-$ docker compose run --rm -e KBC_TEST_TOKEN -e KBC_TEST_API_URL sapi-python-client -m unittest discover
+$ docker compose run --rm -e KBC_TEST_TOKEN -e KBC_TEST_API_URL ci -m unittest discover
 ```
 
 ## Contribution Guide

docker-compose.yml

@@ -1,4 +1,3 @@
-version: '3'
 services:
   ci: &ci
     build: .

kbcstorage/auth.py

@@ -0,0 +1,61 @@
+from typing import Dict, Union
+
+PROGRAMMATIC_PREFIXES = ('kbc_at_', 'kbc_pat_')
+
+
+class StorageApiToken:
+    """
+    Legacy Storage API token, sent as ``X-StorageApi-Token``.
+    """
+    def __init__(self, token: str) -> None:
+        if not token:
+            raise ValueError("Token is required.")
+        # never echo the token itself, the message can end up in logs
+        if token.startswith(PROGRAMMATIC_PREFIXES):
+            raise ValueError(
+                "A token with a 'kbc_at_'/'kbc_pat_' prefix is a programmatic "
+                "token and cannot be sent as X-StorageApi-Token. Use "
+                "kbcstorage.auth.BearerToken with the project id instead.")
+        self.token = token
+
+    def headers(self) -> Dict[str, str]:
+        return {'X-StorageApi-Token': self.token}
+
+
+class BearerToken:
+    """
+    Programmatic session or personal access token (``kbc_at_*``, ``kbc_pat_*``).
+    """
+    def __init__(self, token: str, project_id: Union[str, int]) -> None:
+        if not token:
+            raise ValueError("Token is required.")
+        if not project_id:
+            raise ValueError("Project ID is required for a bearer token.")
+        self.token = token
+        # requests rejects non-string header values, and callers naturally pass
+        # the numeric id straight from a token verify payload
+        self.project_id = str(project_id)
+
+    def headers(self) -> Dict[str, str]:
+        return {'Authorization': 'Bearer {}'.format(self.token),
+                'X-KBC-ProjectId': self.project_id}
+
+
+Auth = Union[StorageApiToken, BearerToken]
+
+
+def coerce(token: Union[str, Auth]) -> Auth:
+    """
+    Normalise the ``token`` argument of an endpoint into a strategy object.
+
+    Args:
+        token: A strategy object, or a bare token string. Endpoints build other
+            endpoints and pass their own strategy on, so this runs once per
+            endpoint and must stay free of side effects.
+
+    Returns:
+        The strategy object to authenticate with.
+    """
+    if isinstance(token, (StorageApiToken, BearerToken)):
+        return token
+    return StorageApiToken(token)

kbcstorage/base.py

@@ -11,6 +11,7 @@
 """
 import os
 
+from kbcstorage.auth import coerce
 from kbcstorage.retry_requests import MAX_RETRIES_DEFAULT, RetryRequests
 import requests
 
@@ -23,6 +24,9 @@ class Endpoint:
     Attributes:
         base_url (str): The base URL for this endpoint.
         token (str): A key for the Storage API.
+        auth (kbcstorage.auth.StorageApiToken|kbcstorage.auth.BearerToken): The
+            authentication strategy to pass on to endpoints created from this
+            one.
     """
     def __init__(self, root_url, path_component, token, max_requests_retries=MAX_RETRIES_DEFAULT):
         """
@@ -33,8 +37,9 @@ def __init__(self, root_url, path_component, token, max_requests_retries=MAX_RET
                 "https://connection.keboola.com/"
             path_component (str): The section of the path specific to the
                 endpoint. eg. "buckets"
-            token (str): A key for the Storage API. Can be found in the storage
-                console.
+            token (str|kbcstorage.auth.StorageApiToken|kbcstorage.auth.BearerToken):
+                A key for the Storage API, found in the storage console, or an
+                authentication strategy from `kbcstorage.auth`.
         """
         if not root_url:
             raise ValueError("Root URL is required.")
@@ -43,13 +48,24 @@ def __init__(self, root_url, path_component, token, max_requests_retries=MAX_RET
         self.root_url = root_url
         self.base_url = '{}/v2/storage/{}'.format(root_url.strip('/'),
                                                   path_component.strip('/'))
-        self.token = token
-        self._auth_header = {'X-StorageApi-Token': self.token,
+        self.auth = coerce(token)
+        self._auth_header = {**self.auth.headers(),
                              'X-KBC-RunId': os.environ.get('KBC_RUNID'),
                              'Accept-Encoding': 'gzip',
                              'User-Agent': 'Keboola Storage API Python Client'}
         self.requests = RetryRequests(max_requests_retries)
 
+    @property
+    def token(self):
+        """
+        The raw token string, whichever scheme is in use.
+
+        Under a bearer token this is the programmatic token, not a Storage API
+        token, so sending it as `X-StorageApi-Token` fails. Pass `auth` when
+        building further endpoints.
+        """
+        return self.auth.token
+
     def _get_raw(self, url, params=None, **kwargs):
         """
         Construct a requests GET call with args and kwargs and process the
@@ -68,8 +84,7 @@ def _get_raw(self, url, params=None, **kwargs):
         Raises:
             requests.HTTPError: If the API request fails.
         """
-        headers = kwargs.pop('headers', {})
-        headers.update(self._auth_header)
+        headers = {**kwargs.pop('headers', {}), **self._auth_header}
 
         r = self.requests.get(url, params=params, headers=headers, **kwargs)
         try:
@@ -114,8 +129,7 @@ def _post(self, *args, **kwargs):
         Raises:
             requests.HTTPError: If the API request fails.
         """
-        headers = kwargs.pop('headers', {})
-        headers.update(self._auth_header)
+        headers = {**kwargs.pop('headers', {}), **self._auth_header}
         r = self.requests.post(headers=headers, *args, **kwargs)
         try:
             r.raise_for_status()
@@ -140,8 +154,7 @@ def _put(self, *args, **kwargs):
         Raises:
             requests.HTTPError: If the API request fails.
         """
-        headers = kwargs.pop('headers', {})
-        headers.update(self._auth_header)
+        headers = {**kwargs.pop('headers', {}), **self._auth_header}
         r = self.requests.put(headers=headers, *args, **kwargs)
         try:
             r.raise_for_status()
@@ -166,8 +179,7 @@ def _delete(self, *args, **kwargs):
         Raises:
             requests.HTTPError: If the API request fails.
         """
-        headers = kwargs.pop('headers', {})
-        headers.update(self._auth_header)
+        headers = {**kwargs.pop('headers', {}), **self._auth_header}
         r = self.requests.delete(headers=headers, *args, **kwargs)
         try:
             r.raise_for_status()

kbcstorage/buckets.py

@@ -50,13 +50,11 @@ def list_tables(self, bucket_id, include=None):
         Raises:
             requests.HTTPError: If the API request fails.
         """
-        headers = {'X-StorageApi-Token': self.token}
-
         url = '{}/{}/tables'.format(self.base_url, bucket_id)
         params = {}
         if include is not None and isinstance(include, list):
             params['include'] = ','.join(include)
-        return self._get(url, headers=headers, params=params)
+        return self._get(url, params=params)
 
     def detail(self, bucket_id):
         """
@@ -121,7 +119,7 @@ def delete(self, bucket_id, force=False, asynchronous=True):
         params = {'force': force, 'async': asynchronous}
         if (asynchronous):
             job = self._delete(url, params=params)
-            jobs = Jobs(self.root_url, self.token)
+            jobs = Jobs(self.root_url, self.auth)
             job = jobs.block_until_completed(job['id'])
             if job['status'] == 'error':
                 raise RuntimeError(job['error']['message'])