Repository navigation
Split up Modules/gcmodule.c #113688
Description
Feature or enhancement
The free-threaded builds require substantial changes to the garbage collection implementation, but we don't want to make those changes to the default build. In other parts of CPython, we have used "local" #ifdef Py_GIL_DISABLED guards around free-threaded specific bits, but I think doing this throughout Modules/gcmodule.c will make the GC harder to maintain than having separate files for the free-threaded and default builds. Additionally, I think @markshannon already suggested splitting the core GC parts from the Python "gc" module.
There's definitely an increased maintenance cost for having two GC implementations in separate files, but I think this maintenance will be less than having a single file with intermixed implementations and lots of #ifdef Py_GIL_DISABLED guards.
Here's a proposed split:
Modules/gcmodule.c - gc Python package interface
Python/gc.c - core GC functionality common to both build configurations (GIL & free-threaded)
Python/gc_gil.c - GIL (non-free-threaded) specific implementations
Python/gc_free_threaded.c - free-threaded specific implementations
At first, I'd expect most code to be in either Modules/gcmodule.c or Python/gc.c. As work on the free-threaded build continues, I would expect Python/gc.c to shrink as code is moved into Python/gc_gil.c and Python/gc_free_threaded.c.
@nascheme @markshannon @pablogsal @DinoV @vstinner
Activity
nascheme commentedon Jan 3, 2024
A single file with both versions of the GC sounds quite confusing so I favor splitting it. Your scheme looks okay to me. Possibly we need an internal header file that's shared between the .c files. The original gcmodule.c was written with the assumption that small functions could be inlined by the compiler as it wishes. With split files, an LTO build probably results in the same inlining. Otherwise, we might need some inline functions in the .h file. As with any optimization, it's probably best to do that only as required for payoff.
pablogsal commentedon Jan 3, 2024
I’m fine with the splitting as well as I really don’t think we should have both versions in the same file so let’s go ahead unless someone has some strong reservations.
Refarding the performance concer a, we do have benchmarks that are checking the performance of GC runs in the pyperformance suite so we can always monitor if changes we made are significant. I am mentioning this so we are not too conservative with the changes and being worried that they are going to affect performance a lot because at the end of the day is very difficult to actually predict how these kind of alterations are going to affect performance.
So my proposal is that we do optimise for maintainability and then check performance and adapt from there.
gpshead commentedon Jan 3, 2024
Agreed, I think this split makes sense. What y'all have already said above.
corona10 commentedon Jan 4, 2024
I agree, too. Other runtime parts are still possible to maintain implementation details except caring whether GIL exists or not. (or with a small cost)
But GC is wholly different. GC should care where objects are located. For example, some objects existed in the interpreter area only, but now we should move them into the thread area. Those kinds of things require separation, and this suggestion will improve maintainability.
And if we introduce more GC implementation details(STW, deferred refcount) in a single file, I can not imagine what will happen :)
pablogsal commentedon Jan 4, 2024
Technically, the current one doesn't care, and that was an advantage. The new one does, which is a different advantage ;)
colesbury commentedon Jan 4, 2024
Great - it seems like there's consensus on splitting the files.
@nascheme, I was thinking that maybe
Python/gc.ccould#includeone ofPython/gc_gil.corPython/gc_free_threaded.c. Something like:We already do this in a few places, with clinic generated files, in
Modules/socketmodule.c, obmalloc with mimalloc files, and a few other modules.It's a bit icky, but it avoids both the concerns around inlining and complexity with configuring the build system to include one or the other file since that's handled at the macro pre-processor level.
nascheme commentedon Jan 4, 2024
I'd prefer not to do that include of .c files. It might make debugging harder, for example. I'd include both files in the build always and then put
ifdefsinside of them to disable code if needed. That avoids needing extra logic when generating the makefile or whatever.gpshead commentedon Jan 4, 2024
This is something we should be able to adjust the how of based on practical experiences if the initial approach winds up causing any issues. Debuggers know which file any given line of code came from, I'm not worried at all about that for #includes.
colesbury commentedon Jan 4, 2024
@nascheme - that sounds fine. I'll follow that approach.
pythongh-113688: Split up gcmodule.c
pythongh-113688: Split up gcmodule.c
Merge branch 'pythongh-113688' into pythongh-112529-gc
gh-113688: Split up gcmodule.c (gh-113715)
pythongh-113688: fix dtrace build on Solaris
kulikjak commentedon Jan 8, 2024
This change broke
--with-dtracebuild on Solaris (or other systems where object files need to be modified with dtrace before linking) with error like this:#113814 fixes the issue.
gh-113688: fix dtrace build on Solaris (#113814)
pythongh-113688: Split up gcmodule.c (pythongh-113715)
pythongh-113688: fix dtrace build on Solaris (python#113814)
pythongh-113688: Split up gcmodule.c (pythongh-113715)
pythongh-113688: fix dtrace build on Solaris (python#113814)
pythongh-113688: Split up gcmodule.c (pythongh-113715)
pythongh-113688: fix dtrace build on Solaris (python#113814)