Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions Doc/library/dis.rst
Original file line number Diff line number Diff line change
Expand Up @@ -2003,6 +2003,16 @@ but are replaced by real opcodes or removed before bytecode is generated.
.. versionchanged:: 3.13
This opcode is now a pseudo-instruction.

.. opcode:: STORE_CLOSURE (i)

Stores the TOS into the cell slot ``i`` of the "fast locals" storage.
Used to restore a cell saved by ``LOAD_CLOSURE`` when isolating an
inlined comprehension that reuses an enclosing free variable.

Note that ``STORE_CLOSURE`` is replaced with ``STORE_FAST`` in the assembler.

.. versionadded:: next


.. _opcode_collections:

Expand Down
6 changes: 6 additions & 0 deletions Include/internal/pycore_interpframe.h
Original file line number Diff line number Diff line change
Expand Up @@ -376,6 +376,12 @@ _PyFrame_Traverse(_PyInterpreterFrame *frame, visitproc visit, void *arg);
bool
_PyFrame_HasHiddenLocals(_PyInterpreterFrame *frame);

/* True when localsplus[oparg] is a free cell that currently differs from
* the function's func_closure cell — i.e. an inlined comprehension has
* temporarily replaced it. */
PyAPI_FUNC(bool)
_PyFrame_IsInlinedCompTempFree(_PyInterpreterFrame *frame, int oparg);

PyObject *
_PyFrame_GetLocals(_PyInterpreterFrame *frame);

Expand Down
24 changes: 16 additions & 8 deletions Include/internal/pycore_opcode_metadata.h

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 2 additions & 1 deletion Include/opcode_ids.h

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

49 changes: 38 additions & 11 deletions InternalDocs/inlined_comprehensions.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,15 +86,35 @@ The walk stops at a class: nested scopes do not see class locals.
Class-closure names that would otherwise be free through a class become
`GLOBAL_IMPLICIT`.

If the inlined name is `LOCAL` or `CELL` but the nearest non-inlined
enclosing table has it as `FREE` (or `DEF_FREE_CLASS`), resolve it as
`FREE` so the comprehension reuses that localsplus slot.
`compiler_cellvars()` also skips adding those child cells, which would
otherwise create a second same-named entry. Class-closure names
(`__class__` and friends) are never reused: zero-arg `super()` and
class-cell bookkeeping need the real free cell.

### Isolating iteration variables

`codegen_push_inlined_comprehension_locals()` in
[`Python/codegen.c`](../Python/codegen.c) isolates names bound in the
comprehension:
[`Python/codegen.c`](../Python/codegen.c) isolates each name bound in
the comprehension on one of two paths:

Reuse an enclosing free (`_PyCompile_GetRefType()` is `FREE`):

* `LOAD_CLOSURE` saves the enclosing cell without clearing the slot
(so free-slot introspection never sees a NULL).
* `MAKE_CELL` on a `CO_FAST_FREE` slot always installs a fresh empty
cell, replacing the saved one. The comprehension then uses `DEREF`.
* Restore uses `STORE_CLOSURE` (a cell/free-index pseudo that becomes
`STORE_FAST` after `fix_cell_offsets`).

* `LOAD_FAST_AND_CLEAR` saves the enclosing value (possibly `NULL`) and
clears the slot.
* `MAKE_CELL` runs if the name is a cell for this comprehension.
Own fast-local slot (everything else):

* `LOAD_FAST_AND_CLEAR` saves the enclosing value (possibly `NULL`)
and clears the slot.
* `MAKE_CELL` runs only if the name is a cell for this comprehension.
* Restore uses `STORE_FAST_MAYBE_NULL`.
* In module and class units the name is added to `u_fasthidden` so
assemble can set `CO_FAST_HIDDEN`.

Expand All @@ -105,10 +125,19 @@ or `finally` sees the original values.
Runtime
-------

An inlined comprehension cell can share a localsplus name with an
enclosing free variable (for example `[lambda: x for x in x]` inside a
nested function). `FrameLocalsProxy` keys, values, items, and `len`
keep the first slot of each name so they agree with `getitem`.
An inlined comprehension local that collides with an enclosing free
(for example `[x for x in x]` or `[lambda: x for x in x]` inside a
nested function) reuses the free slot. Isolation saves that cell and
installs a temporary one so `STORE_DEREF` does not change the value
seen by existing closures; lambdas that capture the iteration variable
share the temporary cell. After the comprehension, the original cell
is restored.

While the temporary cell is installed, `_PyFrame_IsInlinedCompTempFree()`
is true (frame cell differs from `func_closure`). That drives
class/module `locals()` to use `FrameLocalsProxy` even without a
`CO_FAST_HIDDEN` slot, and makes an empty temporary cell raise
`UnboundLocalError` rather than `NameError`.

Source
------
Expand All @@ -128,5 +157,3 @@ Source
`InlinedComprehensionBlock`
* [`Include/internal/pycore_compile.h`](../Include/internal/pycore_compile.h):
`_PyCompile_InlinedComprehensionState`
* [`Objects/frameobject.c`](../Objects/frameobject.c):
`FrameLocalsProxy` duplicate-name handling
3 changes: 2 additions & 1 deletion Lib/_opcode_metadata.py

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading