Skip to content

feat: kms encryption plugin - #1331

Open
JuanLeee wants to merge 1 commit into
feat/kms-sql-analysisfrom
feat/kms-stack-4-plugin
Open

JuanLeee wants to merge 1 commit into
feat/kms-sql-analysisfrom
feat/kms-stack-4-plugin

Conversation

@JuanLeee

@JuanLeee JuanLeee commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Description

This adds the KMS Encryption Plugin (kms_encryption). It encrypts the values an application binds for configured columns before the driver sends them, and decrypts them when they are read back, so the application's SQL doesn't change. Each column has its own data key, and the database holds only a copy of it encrypted by an AWS Key Management Service (KMS) master key, so neither the plaintext nor a usable data key reaches the database. The plugin works with the synchronous wrapper, on PostgreSQL with psycopg and on MySQL with MySQL Connector/Python.

This is the last of four stacked PRs. It uses the plugin call context from #1330, the encryption library from #1300 and the SQL analysis from #1317. Its base is #1317's branch, so its diff is the plugin alone, and it stays a draft until #1330 merges.

  1. On execute and executemany, the plugin makes a copy of the arguments in which every bound parameter that the SQL analysis maps to an encrypted column, written or compared, is encrypted. It hands that copy to the plugins after it and to the driver through the call context (PluginService.current_call_context). The caller's SQL and parameter objects are never modified, and the default plugin stays in the chain. On MySQL the plugin adds _binary before each placeholder it encrypts, so the server takes the payload as binary data and raises no warning 1300.
  2. Encrypted result columns are decrypted on fetchone, fetchmany, fetchall, next(cursor), iteration and psycopg's chained reads (feat: plugin call context for cursor execute arguments #1330 routes the last three through the plugins), and each value comes back as the Python type it was written as. Each value's key id selects its key_storage row, so values written before a key rotation still decrypt. Inside psycopg's with conn.pipeline():, a result's column description arrives only with its rows, so the plugin reads the columns then, and rows of an encrypted result that arrive without one raise MetadataError.
  3. The plugin's weight is 390, so with auto_sort_wrapper_plugin_order on, it runs after aurora_connection_tracker and the read/write splitting plugins and before the failover plugins. The plugins that run before it receive the caller's values; the plugins after it and the driver receive the encrypted copies. The plugin raises its own errors before a failover plugin sees them, so applications get the plugin's error classes, and the MetadataErrors it reports for unreadable metadata don't start failover.
  4. Each connection that uses the plugin has its own metadata cache, data-key cache and KMS client, with up to two background threads that stop when the connection closes. The metadata load, each background refresh and the first read of a value written under an earlier key open a short-lived connection with the application connection's properties. Once the connection is closed, a statement, a fetch or a loop that began before the close raises MetadataError, without reading a key row or calling KMS.

Refused and passed through

  1. Before the driver is called, and before any KMS call for the value, the plugin refuses a literal or other non-parameter value written to an encrypted column (MetadataError), a value with no exact stored form such as a UUID, list or dict (EncryptionFailedError), and a value the analysis can't tie to the encrypted column it may belong to (MetadataError). It also refuses, with MetadataError, a statement with a placeholder inside a quoted literal, quoted identifier or comment when the analysis finds a table with encrypted columns in it, since the driver binds that placeholder too. The same goes for a %(name)s bound for an encrypted column and also used where the value must stay plain, because one parameter can't carry both the payload and the plaintext. A missing or unusable key row raises InvalidKeyMetadataError, and a master key that isn't in encryption_allowed_master_key_arns raises UnauthorizedMasterKeyError before KMS is called.
  2. A statement the plugin has to change is also refused, with EncryptionFailedError before the driver is called, when the plugin finds no call context for it or finds one that holds arguments other than the ones it analyzed. Every execute and executemany through the wrapper's cursor has its own context.
  3. A stored value that fails its HMAC check, or isn't binary, raises IntegrityCheckError unless encryption_return_unverified_data=True. Every error class derives from EncryptionPluginError in aws_advanced_python_wrapper.errors.
  4. A write whose target columns the analysis can't list (an INSERT without a column list, INSERT ... SELECT) runs unchanged, and so does SQL given as an object the plugin doesn't analyze. The plugin logs one warning per connection for each kind of statement it passes through this way, and the database-side trigger must reject any plaintext in them.
  5. The plugin's messages name tables, columns, parameter names and Python types, never a value, a key or SQL text, and the errors it raises itself carry no chained exception. Each value's data-key and HMAC-key copies are wiped after use, on a best-effort basis. The plugin builds its KMS client with KeyManager.create_kms_client, so feat: kms encryption crypto core and key/metadata layer #1300's botocore log filters are on from its first connection, and it imports cryptography and sqlglot only when it is created.

How to enable it

Install the extra, which adds cryptography and sqlglot, and the target driver as usual:

pip install "aws-advanced-python-wrapper[encryption]"

An administrator creates the two metadata tables (encryption_metadata and key_storage) once, sets up each column with KeyManagementUtility (a master key, the column's data key and its metadata rows) and adds a database-side trigger to every encrypted column. Then add kms_encryption to the connection's plugins and list the master keys the plugin may use:

import psycopg

from aws_advanced_python_wrapper import AwsWrapperConnection, release_resources

try:
    with AwsWrapperConnection.connect(
            psycopg.Connection.connect,
            "host=db.example.com dbname=mydb user=app password=...",
            plugins="initial_connection,aurora_connection_tracker,failover_v2,host_monitoring_v2,kms_encryption",
            encryption_kms_region="us-east-1",
            encryption_allowed_master_key_arns="arn:aws:kms:us-east-1:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab",
            autocommit=True,
    ) as conn, conn.cursor() as cur:
        cur.execute("INSERT INTO users (name, ssn) VALUES (%s, %s)", ("Jane Doe", "000-00-0000"))
        cur.execute("SELECT ssn FROM users WHERE name = %s", ("Jane Doe",))
        print(cur.fetchone())
finally:
    release_resources()

The guide gives the encryption_* parameters, the metadata tables' DDL and the per-column triggers, and the trigger SQL ships with the package as aws_advanced_python_wrapper/encryption/sql/trigger_pg.sql and trigger_mysql.sql. With SQLAlchemy, pass the plugin list as connect_args={"plugins": "..."} or ?wrapper_plugins=...; with Django (MySQL only), put it and the encryption_* parameters in the database's OPTIONS.

What this PR contains

One commit, 15 files, +5669/-3, with no dependency change; the encryption extra comes from #1300.

  1. The plugin module aws_advanced_python_wrapper/encryption/kms_encryption_plugin.py, its registration in PluginManager, its message texts, the audit parameter's help text, its unit tests, and an import-guard case that a connection asking for the plugin without the extra fails with the install hint.
  2. The guide, docs/using-the-python-wrapper/using-plugins/UsingTheKmsEncryptionPlugin.md: setup and key rotation with KeyManagementUtility, the parameters, what the plugin encrypts, refuses, passes through and doesn't see, the errors applications catch, hardening, notes for SQLAlchemy, Django and MySQL (including where log_bin_trust_function_creators is set on a self-managed server and on Aurora MySQL), the metadata tables' DDL, and installing the trigger SQL and each column's triggers. The plugin also gets rows in UsingThePythonWrapper.md and PluginChainCompatibility.md, an entry in docs/README.md, and a CHANGELOG entry under ## [Unreleased].
  3. tests/integration/test_kms_encryption_live.py, a live test module run by hand before a PR. No CI workflow runs it, and pytest ./tests/unit doesn't collect it.
  4. The trigger SQL, aws_advanced_python_wrapper/encryption/sql/trigger_pg.sql (version 2) and trigger_mysql.sql (version 1), which the wheel includes and the live module installs, and tests/unit/test_kms_trigger_sql.py, which checks each file's version stamp, its 65-byte minimum against EncryptionService.MIN_ENCRYPTED_LENGTH, and the metadata schema it reads.

Limitations

  1. Every encrypted column needs a database-side trigger. The plugin sees only the statements sent on connections that enable it, and it finds encrypted columns by reading the SQL, so some statements reach the database without the plugin seeing them: psycopg cursor.copy(), cursor.stream() and cursor.results(), result sets after nextset(), callproc(), SQL PREPARE and EXECUTE, LOAD DATA, stored routines, reads through a view, connections taken from target_connection, and the driver's cursor that cursor.target_cursor returns. The guide lists them all under "What the plugin does not see".
  2. Encryption is randomized, so a comparison with an encrypted column never matches, and a unique index or an upsert's conflict target on one never detects a duplicate. A value the plugin doesn't see that is compared with an encrypted column is sent as written, so the database's statement log holds it.
  3. A for loop over a cursor decrypts its rows for the statement that was current when the loop started. Finish the loop before running another statement on the same cursor.
  4. Error reporters that record local variables capture plaintext values from the wrapper's frames.
  5. Leave room for one more connection per application connection under any connection limit (with encryption_metadata_cache_enabled off, one for each thread using that connection at the same time).
  6. encryption_audit_logging_enabled is accepted, but no audit records are written; when it is True, each connection logs one warning saying so.
  7. On MySQL, connections that write encrypted columns should use utf8mb4 or a single-byte client character set, set with the charset connection argument rather than SET NAMES.
  8. There is no async support.

Testing

Items 2 to 4 ran before this branch was rebased onto #1330's current head and before the trigger SQL moved into the package, and haven't been re-run since.

  1. pytest ./tests/unit -Werror passes 4464 tests on Python 3.10 and 3.12 (524 of them new), and flake8, isort and mypy are clean on both. The plugin module has 100% statement and branch coverage. Each test that guards a security rule was checked by breaking the code it guards: of 245 such breaks of the plugin, the call context and the library code under it, every one made a test fail, apart from two that don't change behavior and one control. The tests for the hidden-placeholder and shared-parameter refusals and for pipeline-mode reads each fail with the change they cover reverted.
  2. The live module passes all 275 of its tests against PostgreSQL 16, MySQL 8 and LocalStack's KMS. It runs typical DB-API, SQLAlchemy and Django code through the plugin, decrypts every stored value without the plugin, and checks that neither server's statement log holds plaintext. It needs the three servers on 127.0.0.1 and the environment variables its docstring lists, drops and re-creates its tables and trigger routines, and fails, never skips, when something is missing: pytest tests/integration/test_kms_encryption_live.py.
  3. Three black-box suites written from the documentation, not part of this PR, pass 208, 314 and 340 tests against LocalStack with psycopg 3.3.6 and mysql-connector 9.7.0 (C extension and pure-Python build).
  4. Against AWS KMS in us-east-1, the suites pass 15, 316 and 8 tests (the first and third run only their real-KMS tests). An AES-128-GCM column gets a 16-byte data key, CloudTrail names the key row's ARN in each of the plugin's Decrypt events with no value in any event, the allow-list refuses a real key ARN that isn't on it, and no DEBUG record holds "Plaintext".
  5. On Aurora PostgreSQL 17.9 and Aurora MySQL 3.10.3 (Serverless v2, a writer and a reader each), from EC2 in the same region, every supported type, executemany, every fetch shape and loop, SQLAlchemy and Django on MySQL round-trip, and every stored value is an encrypted payload. The plugin's own connections use the application's TLS settings and log in as the same user under iam and aws_secrets_manager. A cluster failover with the guide's Aurora chain raises one FailoverSuccessError, after which writes and reads continue and every earlier value reads back, and with read_write_splitting or srw reads on the reader decrypt. The statement logs held none of the text and big-integer values written.
  6. Not tested: Python 3.13 and 3.14, RDS for PostgreSQL and RDS for MySQL, Aurora MySQL with binary logging on, KMS throttling, server errors and automatic key rotation, and the live module against Aurora. Items 3 to 5 weren't rerun after the changes for hidden placeholders, shared named parameters, pipeline-mode reads and naive datetime microseconds.

Cost

With the plugin enabled, a statement costs 3.5 to 9.2 µs more client CPU without an encrypted value and 29.0 to 35.9 µs more with one, plus 1.2 to 1.8 µs at the start of a loop (local servers and LocalStack, data key cached, measured before the closed-connection, hidden-placeholder and shared-parameter checks were added). Each new connection's first encrypted statement also waits for one KMS Decrypt on a new KMS client: about 35 ms from EC2 in the KMS region and about 300 ms from a laptop. Connections that don't enable the plugin pay only #1330's cost, plus at most 0.09 µs per call from this PR.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@JuanLeee

JuanLeee commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor Author

CI doesn't run on this PR while its base is #1317's branch, so here are the local results for d07bef8 (the same tree as fdf1034, with the commit message reduced to its subject line), from a clean git archive export with poetry install -E encryption on Python 3.12.15:

  • flake8 .: clean
  • isort --check-only .: clean
  • mypy .: no issues in 392 source files
  • pytest ./tests/unit -Werror: 3756 passed, and 3756 on Python 3.10.22 and 3.11.17 too
  • poetry check --lock: passes

@JuanLeee
JuanLeee force-pushed the feat/kms-sql-analysis branch from b82e57a to 1276122 Compare October 28, 2026 12:00
@JuanLeee
JuanLeee force-pushed the feat/kms-stack-4-plugin branch 2 times, most recently from d07bef8 to 58f1918 Compare October 29, 2026 12:02
@JuanLeee
JuanLeee force-pushed the feat/kms-sql-analysis branch from 1276122 to 4e4c9f9 Compare October 29, 2026 12:02
@JuanLeee
JuanLeee force-pushed the feat/kms-stack-4-plugin branch from 58f1918 to f5c843f Compare November 2, 2026 11:01
@JuanLeee
JuanLeee force-pushed the feat/kms-sql-analysis branch from 4e4c9f9 to 2cb1470 Compare November 2, 2026 11:01
@JuanLeee
JuanLeee force-pushed the feat/kms-sql-analysis branch from 2cb1470 to 448d68c Compare November 5, 2026 00:02
@JuanLeee
JuanLeee force-pushed the feat/kms-stack-4-plugin branch from f5c843f to 43ab309 Compare November 5, 2026 00:02
@JuanLeee
JuanLeee marked this pull request as ready for review November 5, 2026 21:41
-- replace public in public.hmac below with that schema.
CREATE EXTENSION IF NOT EXISTS pgcrypto WITH SCHEMA public;

-- Replace 'encrypt' below if encryption_metadata_schema is set to something else.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
-- Replace 'encrypt' below if encryption_metadata_schema is set to something else.
-- Replace SCHEMA_NAME below with encryption_metadata_schema (encrypt by default). Replace that
-- token only: a plain search for "encrypt" also matches enforce_encrypted_column,
-- encryption_metadata and kms_encryption.

Comment on lines +52 to +53
FROM encrypt.encryption_metadata em
JOIN encrypt.key_storage ks ON em.key_id = ks.id

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
FROM encrypt.encryption_metadata em
JOIN encrypt.key_storage ks ON em.key_id = ks.id
FROM SCHEMA_NAME.encryption_metadata em
JOIN SCHEMA_NAME.key_storage ks ON em.key_id = ks.id

Use SCHEMA_NAME to match the trigger_mysql.sql file


Every encrypted column needs a trigger that checks each value written to it. The trigger needs no help from the application: the HMAC key that signs each value is stored unencrypted in `key_storage`, so the database can check that a value carries a valid tag without holding the data key. The trigger SQL ships with the wrapper, in the `aws_advanced_python_wrapper/encryption/sql` folder of the package:

- [`trigger_pg.sql`](../../../aws_advanced_python_wrapper/encryption/sql/trigger_pg.sql) installs the `pgcrypto` extension if it isn't installed, and creates the `enforce_encrypted_column()` trigger function. The function reads the metadata tables as `encrypt.encryption_metadata` and `encrypt.key_storage`: replace `encrypt` in the file if `encryption_metadata_schema` is set to something else. If `pgcrypto` is already installed in a schema other than `public`, replace `public` in `public.hmac` with that schema.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- [`trigger_pg.sql`](../../../aws_advanced_python_wrapper/encryption/sql/trigger_pg.sql) installs the `pgcrypto` extension if it isn't installed, and creates the `enforce_encrypted_column()` trigger function. The function reads the metadata tables as `encrypt.encryption_metadata` and `encrypt.key_storage`: replace `encrypt` in the file if `encryption_metadata_schema` is set to something else. If `pgcrypto` is already installed in a schema other than `public`, replace `public` in `public.hmac` with that schema.
- [`trigger_pg.sql`](../../../aws_advanced_python_wrapper/encryption/sql/trigger_pg.sql) installs the `pgcrypto` extension if it isn't installed, and creates the `enforce_encrypted_column()` trigger function. The function reads the metadata tables as `SCHEMA_NAME.encryption_metadata` and `SCHEMA_NAME.key_storage`: replace `SCHEMA_NAME` in the file with your metadata schema (`encrypt` unless `encryption_metadata_schema` is set to something else). A file run with `SCHEMA_NAME` left in installs, then fails every write to an encrypted column with an error that the schema doesn't exist. If `pgcrypto` is already installed in a schema other than `public`, replace `public` in `public.hmac` with that schema.

Comment on lines +260 to +262
# PostgreSQL
with conn.cursor() as cur:
cur.execute((sql_dir / "trigger_pg.sql").read_text(encoding="utf-8"))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# PostgreSQL
with conn.cursor() as cur:
cur.execute((sql_dir / "trigger_pg.sql").read_text(encoding="utf-8"))
# PostgreSQL, with SCHEMA_NAME replaced by the metadata schema
with conn.cursor() as cur:
cur.execute((sql_dir / "trigger_pg.sql").read_text(encoding="utf-8").replace("SCHEMA_NAME", "encrypt"))

for statement in ("DROP TABLE IF EXISTS addresses, users CASCADE", "DROP SCHEMA IF EXISTS encrypt CASCADE",
"DROP FUNCTION IF EXISTS enforce_encrypted_column() CASCADE", _PG_METADATA_DDL):
cur.execute(statement)
cur.execute(_trigger_sql("trigger_pg.sql")) # no parameters, or psycopg parses the file's % signs

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
cur.execute(_trigger_sql("trigger_pg.sql")) # no parameters, or psycopg parses the file's % signs
cur.execute(_trigger_sql("trigger_pg.sql").replace("SCHEMA_NAME", "encrypt")) # no parameters, or psycopg parses the file's % signs

class KmsEncryptionPluginFactory(PluginFactory):
@staticmethod
def get_instance(plugin_service: PluginService, props: Properties) -> Plugin:
return KmsEncryptionPlugin(plugin_service, props)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With this structure we are potentially checking against nextset twice. Could we just simplify it to

        if method_name == _NEXTSET:
            return self._next_set(target, execute_func)

        if method_name in _EXECUTES:
            return self._execute(target, method_name, execute_func, args, kwargs)

Where _EXECUTES = frozenset({_EXECUTE, _EXECUTEMANY})

new_args, new_kwargs = self._encrypted_call(target, method_name, args, kwargs)
if new_args is not args or new_kwargs is not kwargs:
context = self._plugin_service.current_call_context
if context is None or not _holds(context, args, kwargs):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we implement _holds on PluginCallContext instead? So this line would become: if context is None or not context.holds(args, kwargs):

Comment on lines +220 to +226
if self._config.metadata_cache_enabled: # the snapshot answers the lookups; a scope would only add its cost
new_args, new_kwargs = self._encrypted_call(target, method_name, args, kwargs)
else:
with self._source.statement(): # the statement's lookups share one connection, closed before the driver call
new_args, new_kwargs = self._encrypted_call(target, method_name, args, kwargs)
if new_args is not args or new_kwargs is not kwargs:
context = self._plugin_service.current_call_context

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if self._config.metadata_cache_enabled: # the snapshot answers the lookups; a scope would only add its cost
new_args, new_kwargs = self._encrypted_call(target, method_name, args, kwargs)
else:
with self._source.statement(): # the statement's lookups share one connection, closed before the driver call
new_args, new_kwargs = self._encrypted_call(target, method_name, args, kwargs)
if new_args is not args or new_kwargs is not kwargs:
context = self._plugin_service.current_call_context
scope = nullcontext() if self._config.metadata_cache_enabled else self._source.statement()
with scope:
new_args, new_kwargs = self._encrypted_call(target, method_name, args, kwargs)

What do you think?

Comment on lines +238 to +243
many = method_name == _EXECUTEMANY
if many and params is not None and not isinstance(params, (list, tuple)):
params = list(params) # an iterator can be read only once
args, kwargs = _replace(args, kwargs, params_at, params)
if many and params is not None and len(params) == 0:
return args, kwargs # an empty batch has no value to encrypt, and the driver sends nothing

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
many = method_name == _EXECUTEMANY
if many and params is not None and not isinstance(params, (list, tuple)):
params = list(params) # an iterator can be read only once
args, kwargs = _replace(args, kwargs, params_at, params)
if many and params is not None and len(params) == 0:
return args, kwargs # an empty batch has no value to encrypt, and the driver sends nothing
if method_name == _EXECUTEMANY and params is not None:
if not isinstance(params, (list, tuple)):
params = list(params) # an iterator can be read only once
args, kwargs = _replace(args, kwargs, params_at, params)
if not params:
return args, kwargs # an empty batch has no value to encrypt, and the driver sends nothing

Comment on lines +244 to +257
text: Any = sql
codec: Optional[str] = None
if isinstance(sql, bytes):
codec = self._codec(target)
text = _decoded(sql, codec)
# mysql-connector keeps reporting the character set it negotiated (python_charset) when SQL changes the
# session's, and every client character set MySQL accepts, except swe7, reads ASCII bytes as ASCII, so only
# all-ASCII ``bytes`` SQL is analyzed. swe7 has no Python codec, so a connection reporting it raises above; a
# session switched to swe7 in SQL is a documented limit.
if not sql.isascii() and SqlDialect.for_driver_dialect(self._driver_dialect) is MYSQL:
text = None
elif type(sql).__module__ == "psycopg.sql":
text = sql.as_string(target) # a psycopg Composable: analyzed as text, sent as the object
if not isinstance(text, str): # other SQL objects, and bytes the checks above reject, run unanalyzed

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think if we delegate this whole section to driver dialect?

Suggested change
text: Any = sql
codec: Optional[str] = None
if isinstance(sql, bytes):
codec = self._codec(target)
text = _decoded(sql, codec)
# mysql-connector keeps reporting the character set it negotiated (python_charset) when SQL changes the
# session's, and every client character set MySQL accepts, except swe7, reads ASCII bytes as ASCII, so only
# all-ASCII ``bytes`` SQL is analyzed. swe7 has no Python codec, so a connection reporting it raises above; a
# session switched to swe7 in SQL is a documented limit.
if not sql.isascii() and SqlDialect.for_driver_dialect(self._driver_dialect) is MYSQL:
text = None
elif type(sql).__module__ == "psycopg.sql":
text = sql.as_string(target) # a psycopg Composable: analyzed as text, sent as the object
if not isinstance(text, str): # other SQL objects, and bytes the checks above reject, run unanalyzed
text, codec = self._driver_dialect.analyzable_sql(sql, target)
if text is None: # other SQL objects, and bytes the driver dialect rejects, run unanalyzed # other SQL objects, and bytes the checks above reject, run unanalyzed

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then the driver dialects could have something like this (example written for the PG driver dialect):

    def analyzable_sql(self, sql: Any, target: Any) -> Tuple[Optional[str], Optional[str]]:
        if isinstance(sql, psycopg_sql.Composable):
            return sql.as_string(target), None  # analyzed as text, sent as the object
        if isinstance(sql, bytes):
            # psycopg raises NotSupportedError for an encoding Python has no codec for.
            conn = self.get_connection_from_obj(target)
            codec = None if conn is None else conn.info.encoding
            return self._decode_sql(sql, codec), codec
        return super().analyzable_sql(sql, target)

_ONE_ROW = frozenset({_FETCHONE, _NEXT}) # the reads that return one row
_SQL_NAMES = ("query", "operation")
_PARAM_NAMES = ("params", "params_seq", "seq_params")
_PREPARED = ("MySQLCursorPrepared", "CMySQLCursorPrepared") # mysql-connector 9.7.0; the *Dict classes subclass them

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reminder to check all the callers for this and have callsites delegate call to target driver dialect instead

parts: List[str] = []
previous = 0
for start in starts:
parts += (sql[previous:start], "_binary ")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this MySQL specific?

positions[i] = config
factory = getattr(target, "row_factory", None) # psycopg only; checked before any row is built, except through _decoder_after_fetch
if positions and factory is not None:
from psycopg import rows

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reminder: move to driver dialect

"""Whether a warning for ``key`` is still to be logged on this connection; records it. With ``bounded``, only
while fewer than 256 keys are recorded."""
with self._state_lock:
if key in self._warned or (bounded and len(self._warned) >= 256):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we store 256 as a constant

only = state.analysis.tables[0].name if len(state.analysis.tables) == 1 else None
configs = state.configs # a column whose table has none isn't encrypted
positions: Dict[int, ColumnEncryptionConfig] = {}
for i, name in enumerate(names):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
for i, name in enumerate(names):
for i, _ in enumerate(names):

return self._unverified(value, config, "KmsEncryptionPlugin.NotAPayload")
if self._closed: # a loop that began before the close decodes lazily and never reaches _decoder again
raise MetadataError(Messages.get("KmsEncryptionPlugin.Closed"))
assert self._keys is not None

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we check for none and raise a MetadataError instead?

raise self._unusable(config)
else: # a value written before the column's key was rotated, or a value that isn't a payload
if not EncryptionAlgorithm.is_supported(config.algorithm): # before the lookup: no earlier key row makes it usable
raise self._unusable(config)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_unusable performs the same check again, should we just raise directly here instead of using the method call?

            return InvalidAlgorithmError(
                Messages.get_formatted("KmsEncryptionPlugin.UnsupportedAlgorithm", config.column_identifier,
                                       repr(config.algorithm), ", ".join(EncryptionAlgorithm.ALL)),
                algorithm=config.algorithm)

@karenc-bq karenc-bq left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also let some extra comments on #1300 (review)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants