Improve performance of dict merges by avoiding incref/decref pairs

[eendebakpt]

Feature or enhancement

The per-item loop in dict_dict_merge() takes an extra reference to key and value before dispatching on override. On the override == 1 path insertdict() already receives its own references through Py_NewRef() and
nothing uses key or value afterwards, so the outer pair is wasted refcount operations per item (atomic ones in the free-threaded build). The pair is still needed on the other path, where _PyDict_Contains_KnownHash() can run __eq__ and *dupkey hands the reference to the caller.

override == 1 covers dict.update(d), d1 \| d2, {**a, **b}, any merge into an empty dict, and dict(d) / d.copy() when the key-table clone fast path does not apply (sparse, split-table or subclass source).

Benchmarks

case	main	PR	change
ctl: a + b	41.1 ns	41.0 ns	not significant
ctl: dct_100[key]	48.3 ns	48.6 ns	not significant
ctl: dict(dct_100) (clone path)	369 ns	371 ns	not significant
dct.update(dct_5)	150 ns	146 ns	not significant
dct.update(dct_100)	1.63 us	1.56 us	1.05x faster
dct.update(dct_1000)	17.1 us	16.5 us	1.04x faster
dct.update(dct_100) (int keys)	1.18 us	1.09 us	1.08x faster
dct_100 | other_100	2.84 us	2.74 us	1.04x faster
{**dct_5, **other_5}	230 ns	224 ns	1.03x faster
sparse_1000.copy() (500 deleted)	10.4 us	9.93 us	1.05x faster
dict(vars(obj)) (4 attributes)	170 ns	168 ns	not significant

Free-threaded build shows a slightly larger gain.

Benchmark script

"""dict_dict_merge() cases. ctl rows do not run the per-item merge loop:
dense copies take the key-table clone fast path."""
import os
import pyperf

runner = pyperf.Runner()
def S(n): return "{'key%%d' %% i: i for i in range(%d)}" % n

CASES = [
    ("ctl: a + b", "a + b", "a = 1000; b = 2000"),
    ("ctl: dct_100[key]", "d[k]", "d = " + S(100) + "; k = 'key50'"),
    ("ctl: dict(dct_5)  (clone path)", "dict(d)", "d = dict.fromkeys('abcde', 1)"),
    ("ctl: dict(dct_100)  (clone path)", "dict(d)", "d = " + S(100)),

    ("dct.update(dct_5)", "c.update(b)", "b = dict.fromkeys('abcde', 1); c = dict(b)"),
    ("dct.update(dct_100)", "c.update(b)", "b = " + S(100) + "; c = dict(b)"),
    ("dct.update(dct_1000)", "c.update(b)", "b = " + S(1000) + "; c = dict(b)"),
    ("dct.update(dct_100)  (int keys)", "c.update(b)", "b = {i: i for i in range(1000, 1100)}; c = dict(b)"),
    ("dct_100 | other_100", "a | b", "a = " + S(100) + "; b = {'other%d' % i: i for i in range(100)}"),
    ("{**dct_5, **other_5}", "{**a, **b}", "a = dict.fromkeys('abcde', 1); b = dict.fromkeys('fghij', 1)"),
    ("sparse_1000.copy()  (500 deleted)", "d.copy()",
     "d = " + S(1000) + "\nfor i in range(0, 1000, 2): del d['key%d' % i]"),
    ("dict(vars(obj))  (4 attributes)", "dict(v)",
     "class C:\n    def __init__(self): self.a = 1; self.b = 2; self.c = 3; self.d = 4\nv = vars(C())"),
]
for name, stmt, setup in CASES:
    runner.timeit(name, stmt, setup=setup)

Generated with help from Claude Code

Has this already been discussed elsewhere?

This is a minor feature, which does not need previous discussion elsewhere

Links to previous discussion of this feature:

No response

Linked PRs

• gh-158863: Drop redundant incref/decref pair in dict_dict_merge() #158864