Repository navigation
Conversation
|
CI doesn't run on this PR while its base is #1317's branch, so here are the local results for
|
b82e57a to
1276122
Compare
d07bef8 to
58f1918
Compare
1276122 to
4e4c9f9
Compare
58f1918 to
f5c843f
Compare
4e4c9f9 to
2cb1470
Compare
2cb1470 to
448d68c
Compare
f5c843f to
43ab309
Compare
| -- replace public in public.hmac below with that schema. | ||
| CREATE EXTENSION IF NOT EXISTS pgcrypto WITH SCHEMA public; | ||
|
|
||
| -- Replace 'encrypt' below if encryption_metadata_schema is set to something else. |
There was a problem hiding this comment.
| -- Replace 'encrypt' below if encryption_metadata_schema is set to something else. | |
| -- Replace SCHEMA_NAME below with encryption_metadata_schema (encrypt by default). Replace that | |
| -- token only: a plain search for "encrypt" also matches enforce_encrypted_column, | |
| -- encryption_metadata and kms_encryption. |
| FROM encrypt.encryption_metadata em | ||
| JOIN encrypt.key_storage ks ON em.key_id = ks.id |
There was a problem hiding this comment.
| FROM encrypt.encryption_metadata em | |
| JOIN encrypt.key_storage ks ON em.key_id = ks.id | |
| FROM SCHEMA_NAME.encryption_metadata em | |
| JOIN SCHEMA_NAME.key_storage ks ON em.key_id = ks.id |
Use SCHEMA_NAME to match the trigger_mysql.sql file
|
|
||
| Every encrypted column needs a trigger that checks each value written to it. The trigger needs no help from the application: the HMAC key that signs each value is stored unencrypted in `key_storage`, so the database can check that a value carries a valid tag without holding the data key. The trigger SQL ships with the wrapper, in the `aws_advanced_python_wrapper/encryption/sql` folder of the package: | ||
|
|
||
| - [`trigger_pg.sql`](../../../aws_advanced_python_wrapper/encryption/sql/trigger_pg.sql) installs the `pgcrypto` extension if it isn't installed, and creates the `enforce_encrypted_column()` trigger function. The function reads the metadata tables as `encrypt.encryption_metadata` and `encrypt.key_storage`: replace `encrypt` in the file if `encryption_metadata_schema` is set to something else. If `pgcrypto` is already installed in a schema other than `public`, replace `public` in `public.hmac` with that schema. |
There was a problem hiding this comment.
| - [`trigger_pg.sql`](../../../aws_advanced_python_wrapper/encryption/sql/trigger_pg.sql) installs the `pgcrypto` extension if it isn't installed, and creates the `enforce_encrypted_column()` trigger function. The function reads the metadata tables as `encrypt.encryption_metadata` and `encrypt.key_storage`: replace `encrypt` in the file if `encryption_metadata_schema` is set to something else. If `pgcrypto` is already installed in a schema other than `public`, replace `public` in `public.hmac` with that schema. | |
| - [`trigger_pg.sql`](../../../aws_advanced_python_wrapper/encryption/sql/trigger_pg.sql) installs the `pgcrypto` extension if it isn't installed, and creates the `enforce_encrypted_column()` trigger function. The function reads the metadata tables as `SCHEMA_NAME.encryption_metadata` and `SCHEMA_NAME.key_storage`: replace `SCHEMA_NAME` in the file with your metadata schema (`encrypt` unless `encryption_metadata_schema` is set to something else). A file run with `SCHEMA_NAME` left in installs, then fails every write to an encrypted column with an error that the schema doesn't exist. If `pgcrypto` is already installed in a schema other than `public`, replace `public` in `public.hmac` with that schema. |
| # PostgreSQL | ||
| with conn.cursor() as cur: | ||
| cur.execute((sql_dir / "trigger_pg.sql").read_text(encoding="utf-8")) |
There was a problem hiding this comment.
| # PostgreSQL | |
| with conn.cursor() as cur: | |
| cur.execute((sql_dir / "trigger_pg.sql").read_text(encoding="utf-8")) | |
| # PostgreSQL, with SCHEMA_NAME replaced by the metadata schema | |
| with conn.cursor() as cur: | |
| cur.execute((sql_dir / "trigger_pg.sql").read_text(encoding="utf-8").replace("SCHEMA_NAME", "encrypt")) |
| for statement in ("DROP TABLE IF EXISTS addresses, users CASCADE", "DROP SCHEMA IF EXISTS encrypt CASCADE", | ||
| "DROP FUNCTION IF EXISTS enforce_encrypted_column() CASCADE", _PG_METADATA_DDL): | ||
| cur.execute(statement) | ||
| cur.execute(_trigger_sql("trigger_pg.sql")) # no parameters, or psycopg parses the file's % signs |
There was a problem hiding this comment.
| cur.execute(_trigger_sql("trigger_pg.sql")) # no parameters, or psycopg parses the file's % signs | |
| cur.execute(_trigger_sql("trigger_pg.sql").replace("SCHEMA_NAME", "encrypt")) # no parameters, or psycopg parses the file's % signs |
| class KmsEncryptionPluginFactory(PluginFactory): | ||
| @staticmethod | ||
| def get_instance(plugin_service: PluginService, props: Properties) -> Plugin: | ||
| return KmsEncryptionPlugin(plugin_service, props) |
There was a problem hiding this comment.
With this structure we are potentially checking against nextset twice. Could we just simplify it to
if method_name == _NEXTSET:
return self._next_set(target, execute_func)
if method_name in _EXECUTES:
return self._execute(target, method_name, execute_func, args, kwargs)Where _EXECUTES = frozenset({_EXECUTE, _EXECUTEMANY})
| new_args, new_kwargs = self._encrypted_call(target, method_name, args, kwargs) | ||
| if new_args is not args or new_kwargs is not kwargs: | ||
| context = self._plugin_service.current_call_context | ||
| if context is None or not _holds(context, args, kwargs): |
There was a problem hiding this comment.
Can we implement _holds on PluginCallContext instead? So this line would become: if context is None or not context.holds(args, kwargs):
| if self._config.metadata_cache_enabled: # the snapshot answers the lookups; a scope would only add its cost | ||
| new_args, new_kwargs = self._encrypted_call(target, method_name, args, kwargs) | ||
| else: | ||
| with self._source.statement(): # the statement's lookups share one connection, closed before the driver call | ||
| new_args, new_kwargs = self._encrypted_call(target, method_name, args, kwargs) | ||
| if new_args is not args or new_kwargs is not kwargs: | ||
| context = self._plugin_service.current_call_context |
There was a problem hiding this comment.
| if self._config.metadata_cache_enabled: # the snapshot answers the lookups; a scope would only add its cost | |
| new_args, new_kwargs = self._encrypted_call(target, method_name, args, kwargs) | |
| else: | |
| with self._source.statement(): # the statement's lookups share one connection, closed before the driver call | |
| new_args, new_kwargs = self._encrypted_call(target, method_name, args, kwargs) | |
| if new_args is not args or new_kwargs is not kwargs: | |
| context = self._plugin_service.current_call_context | |
| scope = nullcontext() if self._config.metadata_cache_enabled else self._source.statement() | |
| with scope: | |
| new_args, new_kwargs = self._encrypted_call(target, method_name, args, kwargs) |
What do you think?
| many = method_name == _EXECUTEMANY | ||
| if many and params is not None and not isinstance(params, (list, tuple)): | ||
| params = list(params) # an iterator can be read only once | ||
| args, kwargs = _replace(args, kwargs, params_at, params) | ||
| if many and params is not None and len(params) == 0: | ||
| return args, kwargs # an empty batch has no value to encrypt, and the driver sends nothing |
There was a problem hiding this comment.
| many = method_name == _EXECUTEMANY | |
| if many and params is not None and not isinstance(params, (list, tuple)): | |
| params = list(params) # an iterator can be read only once | |
| args, kwargs = _replace(args, kwargs, params_at, params) | |
| if many and params is not None and len(params) == 0: | |
| return args, kwargs # an empty batch has no value to encrypt, and the driver sends nothing | |
| if method_name == _EXECUTEMANY and params is not None: | |
| if not isinstance(params, (list, tuple)): | |
| params = list(params) # an iterator can be read only once | |
| args, kwargs = _replace(args, kwargs, params_at, params) | |
| if not params: | |
| return args, kwargs # an empty batch has no value to encrypt, and the driver sends nothing |
| text: Any = sql | ||
| codec: Optional[str] = None | ||
| if isinstance(sql, bytes): | ||
| codec = self._codec(target) | ||
| text = _decoded(sql, codec) | ||
| # mysql-connector keeps reporting the character set it negotiated (python_charset) when SQL changes the | ||
| # session's, and every client character set MySQL accepts, except swe7, reads ASCII bytes as ASCII, so only | ||
| # all-ASCII ``bytes`` SQL is analyzed. swe7 has no Python codec, so a connection reporting it raises above; a | ||
| # session switched to swe7 in SQL is a documented limit. | ||
| if not sql.isascii() and SqlDialect.for_driver_dialect(self._driver_dialect) is MYSQL: | ||
| text = None | ||
| elif type(sql).__module__ == "psycopg.sql": | ||
| text = sql.as_string(target) # a psycopg Composable: analyzed as text, sent as the object | ||
| if not isinstance(text, str): # other SQL objects, and bytes the checks above reject, run unanalyzed |
There was a problem hiding this comment.
What do you think if we delegate this whole section to driver dialect?
| text: Any = sql | |
| codec: Optional[str] = None | |
| if isinstance(sql, bytes): | |
| codec = self._codec(target) | |
| text = _decoded(sql, codec) | |
| # mysql-connector keeps reporting the character set it negotiated (python_charset) when SQL changes the | |
| # session's, and every client character set MySQL accepts, except swe7, reads ASCII bytes as ASCII, so only | |
| # all-ASCII ``bytes`` SQL is analyzed. swe7 has no Python codec, so a connection reporting it raises above; a | |
| # session switched to swe7 in SQL is a documented limit. | |
| if not sql.isascii() and SqlDialect.for_driver_dialect(self._driver_dialect) is MYSQL: | |
| text = None | |
| elif type(sql).__module__ == "psycopg.sql": | |
| text = sql.as_string(target) # a psycopg Composable: analyzed as text, sent as the object | |
| if not isinstance(text, str): # other SQL objects, and bytes the checks above reject, run unanalyzed | |
| text, codec = self._driver_dialect.analyzable_sql(sql, target) | |
| if text is None: # other SQL objects, and bytes the driver dialect rejects, run unanalyzed # other SQL objects, and bytes the checks above reject, run unanalyzed |
There was a problem hiding this comment.
Then the driver dialects could have something like this (example written for the PG driver dialect):
def analyzable_sql(self, sql: Any, target: Any) -> Tuple[Optional[str], Optional[str]]:
if isinstance(sql, psycopg_sql.Composable):
return sql.as_string(target), None # analyzed as text, sent as the object
if isinstance(sql, bytes):
# psycopg raises NotSupportedError for an encoding Python has no codec for.
conn = self.get_connection_from_obj(target)
codec = None if conn is None else conn.info.encoding
return self._decode_sql(sql, codec), codec
return super().analyzable_sql(sql, target)
| _ONE_ROW = frozenset({_FETCHONE, _NEXT}) # the reads that return one row | ||
| _SQL_NAMES = ("query", "operation") | ||
| _PARAM_NAMES = ("params", "params_seq", "seq_params") | ||
| _PREPARED = ("MySQLCursorPrepared", "CMySQLCursorPrepared") # mysql-connector 9.7.0; the *Dict classes subclass them |
There was a problem hiding this comment.
Reminder to check all the callers for this and have callsites delegate call to target driver dialect instead
| parts: List[str] = [] | ||
| previous = 0 | ||
| for start in starts: | ||
| parts += (sql[previous:start], "_binary ") |
| positions[i] = config | ||
| factory = getattr(target, "row_factory", None) # psycopg only; checked before any row is built, except through _decoder_after_fetch | ||
| if positions and factory is not None: | ||
| from psycopg import rows |
There was a problem hiding this comment.
reminder: move to driver dialect
| """Whether a warning for ``key`` is still to be logged on this connection; records it. With ``bounded``, only | ||
| while fewer than 256 keys are recorded.""" | ||
| with self._state_lock: | ||
| if key in self._warned or (bounded and len(self._warned) >= 256): |
There was a problem hiding this comment.
Can we store 256 as a constant
| only = state.analysis.tables[0].name if len(state.analysis.tables) == 1 else None | ||
| configs = state.configs # a column whose table has none isn't encrypted | ||
| positions: Dict[int, ColumnEncryptionConfig] = {} | ||
| for i, name in enumerate(names): |
There was a problem hiding this comment.
| for i, name in enumerate(names): | |
| for i, _ in enumerate(names): |
| return self._unverified(value, config, "KmsEncryptionPlugin.NotAPayload") | ||
| if self._closed: # a loop that began before the close decodes lazily and never reaches _decoder again | ||
| raise MetadataError(Messages.get("KmsEncryptionPlugin.Closed")) | ||
| assert self._keys is not None |
There was a problem hiding this comment.
Can we check for none and raise a MetadataError instead?
| raise self._unusable(config) | ||
| else: # a value written before the column's key was rotated, or a value that isn't a payload | ||
| if not EncryptionAlgorithm.is_supported(config.algorithm): # before the lookup: no earlier key row makes it usable | ||
| raise self._unusable(config) |
There was a problem hiding this comment.
_unusable performs the same check again, should we just raise directly here instead of using the method call?
return InvalidAlgorithmError(
Messages.get_formatted("KmsEncryptionPlugin.UnsupportedAlgorithm", config.column_identifier,
repr(config.algorithm), ", ".join(EncryptionAlgorithm.ALL)),
algorithm=config.algorithm)
karenc-bq
left a comment
There was a problem hiding this comment.
Also let some extra comments on #1300 (review)
Description
This adds the KMS Encryption Plugin (
kms_encryption). It encrypts the values an application binds for configured columns before the driver sends them, and decrypts them when they are read back, so the application's SQL doesn't change. Each column has its own data key, and the database holds only a copy of it encrypted by an AWS Key Management Service (KMS) master key, so neither the plaintext nor a usable data key reaches the database. The plugin works with the synchronous wrapper, on PostgreSQL with psycopg and on MySQL with MySQL Connector/Python.This is the last of four stacked PRs. It uses the plugin call context from #1330, the encryption library from #1300 and the SQL analysis from #1317. Its base is #1317's branch, so its diff is the plugin alone, and it stays a draft until #1330 merges.
executeandexecutemany, the plugin makes a copy of the arguments in which every bound parameter that the SQL analysis maps to an encrypted column, written or compared, is encrypted. It hands that copy to the plugins after it and to the driver through the call context (PluginService.current_call_context). The caller's SQL and parameter objects are never modified, and the default plugin stays in the chain. On MySQL the plugin adds_binarybefore each placeholder it encrypts, so the server takes the payload as binary data and raises no warning 1300.fetchone,fetchmany,fetchall,next(cursor), iteration and psycopg's chained reads (feat: plugin call context for cursor execute arguments #1330 routes the last three through the plugins), and each value comes back as the Python type it was written as. Each value's key id selects itskey_storagerow, so values written before a key rotation still decrypt. Inside psycopg'swith conn.pipeline():, a result's column description arrives only with its rows, so the plugin reads the columns then, and rows of an encrypted result that arrive without one raiseMetadataError.auto_sort_wrapper_plugin_orderon, it runs afteraurora_connection_trackerand the read/write splitting plugins and before the failover plugins. The plugins that run before it receive the caller's values; the plugins after it and the driver receive the encrypted copies. The plugin raises its own errors before a failover plugin sees them, so applications get the plugin's error classes, and theMetadataErrors it reports for unreadable metadata don't start failover.MetadataError, without reading a key row or calling KMS.Refused and passed through
MetadataError), a value with no exact stored form such as a UUID, list or dict (EncryptionFailedError), and a value the analysis can't tie to the encrypted column it may belong to (MetadataError). It also refuses, withMetadataError, a statement with a placeholder inside a quoted literal, quoted identifier or comment when the analysis finds a table with encrypted columns in it, since the driver binds that placeholder too. The same goes for a%(name)sbound for an encrypted column and also used where the value must stay plain, because one parameter can't carry both the payload and the plaintext. A missing or unusable key row raisesInvalidKeyMetadataError, and a master key that isn't inencryption_allowed_master_key_arnsraisesUnauthorizedMasterKeyErrorbefore KMS is called.EncryptionFailedErrorbefore the driver is called, when the plugin finds no call context for it or finds one that holds arguments other than the ones it analyzed. Everyexecuteandexecutemanythrough the wrapper's cursor has its own context.IntegrityCheckErrorunlessencryption_return_unverified_data=True. Every error class derives fromEncryptionPluginErrorinaws_advanced_python_wrapper.errors.INSERTwithout a column list,INSERT ... SELECT) runs unchanged, and so does SQL given as an object the plugin doesn't analyze. The plugin logs one warning per connection for each kind of statement it passes through this way, and the database-side trigger must reject any plaintext in them.KeyManager.create_kms_client, so feat: kms encryption crypto core and key/metadata layer #1300's botocore log filters are on from its first connection, and it importscryptographyandsqlglotonly when it is created.How to enable it
Install the extra, which adds
cryptographyandsqlglot, and the target driver as usual:pip install "aws-advanced-python-wrapper[encryption]"An administrator creates the two metadata tables (
encryption_metadataandkey_storage) once, sets up each column withKeyManagementUtility(a master key, the column's data key and its metadata rows) and adds a database-side trigger to every encrypted column. Then addkms_encryptionto the connection'spluginsand list the master keys the plugin may use:The guide gives the
encryption_*parameters, the metadata tables' DDL and the per-column triggers, and the trigger SQL ships with the package asaws_advanced_python_wrapper/encryption/sql/trigger_pg.sqlandtrigger_mysql.sql. With SQLAlchemy, pass the plugin list asconnect_args={"plugins": "..."}or?wrapper_plugins=...; with Django (MySQL only), put it and theencryption_*parameters in the database'sOPTIONS.What this PR contains
One commit, 15 files, +5669/-3, with no dependency change; the
encryptionextra comes from #1300.aws_advanced_python_wrapper/encryption/kms_encryption_plugin.py, its registration inPluginManager, its message texts, the audit parameter's help text, its unit tests, and an import-guard case that a connection asking for the plugin without the extra fails with the install hint.docs/using-the-python-wrapper/using-plugins/UsingTheKmsEncryptionPlugin.md: setup and key rotation withKeyManagementUtility, the parameters, what the plugin encrypts, refuses, passes through and doesn't see, the errors applications catch, hardening, notes for SQLAlchemy, Django and MySQL (including wherelog_bin_trust_function_creatorsis set on a self-managed server and on Aurora MySQL), the metadata tables' DDL, and installing the trigger SQL and each column's triggers. The plugin also gets rows inUsingThePythonWrapper.mdandPluginChainCompatibility.md, an entry indocs/README.md, and a CHANGELOG entry under## [Unreleased].tests/integration/test_kms_encryption_live.py, a live test module run by hand before a PR. No CI workflow runs it, andpytest ./tests/unitdoesn't collect it.aws_advanced_python_wrapper/encryption/sql/trigger_pg.sql(version 2) andtrigger_mysql.sql(version 1), which the wheel includes and the live module installs, andtests/unit/test_kms_trigger_sql.py, which checks each file's version stamp, its 65-byte minimum againstEncryptionService.MIN_ENCRYPTED_LENGTH, and the metadata schema it reads.Limitations
cursor.copy(),cursor.stream()andcursor.results(), result sets afternextset(),callproc(), SQLPREPAREandEXECUTE,LOAD DATA, stored routines, reads through a view, connections taken fromtarget_connection, and the driver's cursor thatcursor.target_cursorreturns. The guide lists them all under "What the plugin does not see".forloop over a cursor decrypts its rows for the statement that was current when the loop started. Finish the loop before running another statement on the same cursor.encryption_metadata_cache_enabledoff, one for each thread using that connection at the same time).encryption_audit_logging_enabledis accepted, but no audit records are written; when it isTrue, each connection logs one warning saying so.utf8mb4or a single-byte client character set, set with thecharsetconnection argument rather thanSET NAMES.Testing
Items 2 to 4 ran before this branch was rebased onto #1330's current head and before the trigger SQL moved into the package, and haven't been re-run since.
pytest ./tests/unit -Werrorpasses 4464 tests on Python 3.10 and 3.12 (524 of them new), and flake8, isort and mypy are clean on both. The plugin module has 100% statement and branch coverage. Each test that guards a security rule was checked by breaking the code it guards: of 245 such breaks of the plugin, the call context and the library code under it, every one made a test fail, apart from two that don't change behavior and one control. The tests for the hidden-placeholder and shared-parameter refusals and for pipeline-mode reads each fail with the change they cover reverted.pytest tests/integration/test_kms_encryption_live.py.Decryptevents with no value in any event, the allow-list refuses a real key ARN that isn't on it, and no DEBUG record holds"Plaintext".executemany, every fetch shape and loop, SQLAlchemy and Django on MySQL round-trip, and every stored value is an encrypted payload. The plugin's own connections use the application's TLS settings and log in as the same user underiamandaws_secrets_manager. A cluster failover with the guide's Aurora chain raises oneFailoverSuccessError, after which writes and reads continue and every earlier value reads back, and withread_write_splittingorsrwreads on the reader decrypt. The statement logs held none of the text and big-integer values written.datetimemicroseconds.Cost
With the plugin enabled, a statement costs 3.5 to 9.2 µs more client CPU without an encrypted value and 29.0 to 35.9 µs more with one, plus 1.2 to 1.8 µs at the start of a loop (local servers and LocalStack, data key cached, measured before the closed-connection, hidden-placeholder and shared-parameter checks were added). Each new connection's first encrypted statement also waits for one KMS
Decrypton a new KMS client: about 35 ms from EC2 in the KMS region and about 300 ms from a laptop. Connections that don't enable the plugin pay only #1330's cost, plus at most 0.09 µs per call from this PR.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.