Repository navigation
Refresh Screen Provided By curses.wrapper Causes Seg Fault (macOS, xcode 15 Apple supplied ncurses 6.0 breakage) #109617
Description
Activity
I've been encountering this on
maintoo after updating (thetest_cursestests in our suite seem to trigger it). One really annoying side-effect is that it breaks your terminal after the crash:% ./python.exe -m test --multiprocess 1 --use curses --verbose test_curses0:00:00 load avg: 1.27 [1/1/1] test_curses process crashed (Exit code -11) test_has_extended_color_support (test.test_curses.MiscTests.test_has_extended_color_support) ... ok test_ncurses_version (test.test_curses.MiscTests.test_ncurses_version) ... ncurses_version = curses.ncurses_version(major=6, minor=0, patch=20150808) ok test_update_lines_cols (test.test_curses.MiscTests.test_update_lines_cols) ... ok test_alt (test.test_curses.TestAscii.test_alt) ... ok test_ascii (test.test_curses.TestAscii.test_ascii) ... ok test_controlnames (test.test_curses.TestAscii.test_controlnames) ... ok test_ctrl (test.test_curses.TestAscii.test_ctrl) ... ok test_ctypes (test.test_curses.TestAscii.test_ctypes) ... ok test_unctrl (test.test_curses.TestAscii.test_unctrl) ... ok TERM=xterm-256color test_attributes (test.test_curses.TestCurses.test_attributes) ... ok test_background (test.test_curses.TestCurses.test_background) ... ok test_beep (test.test_curses.TestCurses.test_beep) ... ok test_borders_and_lines (test.test_curses.TestCurses.test_borders_and_lines) ... ok test_chgat (test.test_curses.TestCurses.test_chgat) ... ok test_clear (test.test_curses.TestCurses.test_clear) ... ok test_color_attrs (test.test_curses.TestCurses.test_color_attrs) ... ok test_color_content (test.test_curses.TestCurses.test_color_content) ... ok test_create_windows (test.test_curses.TestCurses.test_create_windows) ... == Tests result: FAILURE == 1 test failed: test_curses Total duration: 198 ms Total tests: run=0 Total test files: run=1/1 failed=1 Result: FAILURE %I'm not a
cursesexpert, but I'm not sure that this is something we can fix on our end, so this should probably be closed. It might be worth disabling the crashy tests if we can detect the brokencursesversion, though?CC @Yhg1s as a
cursesexpert.The affected tests appear to be
test_create_windows,test_move_cursor,test_output_character,test_refresh, andtest_refresh_control.FWIW this problem should not arise when using a python from a macOS python.org installer or MacPorts as they supply their own version of ncurses and do not use the system version.
Reacted by Gregory P. SmithFWIW this problem should not arise when using a python from a macOS python.org installer or MacPorts as they supply their own version of ncurses and do not use the system version.
I can confirm this. I was only able to reproduce it for @timway when using the system version of python.
Downgrading from Xcode 15.0 (
curses.ncurses_version(major=6, minor=0, patch=20150808)) to 14.3 (curses.ncurses_version(major=5, minor=7, patch=20081102)) fixes it for me.- changed the title
[-]Refresh Screen Provided By `curses.wrapper` Causes Seg Fault (MacOS)[/-][+]Refresh Screen Provided By `curses.wrapper` Causes Seg Fault (macOS, xcode 15 Apple supplied Python build)[/+]on Sep 20, 2023 - changed the title
[-]Refresh Screen Provided By `curses.wrapper` Causes Seg Fault (macOS, xcode 15 Apple supplied Python build)[/-][+]Refresh Screen Provided By `curses.wrapper` Causes Seg Fault (macOS, xcode 15 Apple supplied ncurses 6.0 breakage)[/+]on Sep 20, 2023 It sounds like they shipped a potentially broken version of ncurses embedded within xcode 15?
We've been building and linking with ncurses >= 6 in the rest of the world since 2015 without this problem...
It is really strange behavior on Apple's part to "upgrade" to providing 6.0 from 2015 in their 2023 toolchain when ncurses 6.4 came out last year.
We're at the point where it'd be fine for CPython to simply drop the ability to build and link against 6.0 in 3.13+ given that no supported Linux distro platform will be using it anymore by the time we release 3.13. (RHEL 8 shipped with ncurses 6.1 making that a reasonable minimum) -- I do not expect or encourage anyone to go through and do that to the
_cursesmodule code.Recommendation: do what we do with python.org builds, never build against apple's ncurses.
If a workaround for this isn't obvious to someone with a macos debugger, it is reasonable to have a configure check blocklist the xcode 15 supplied ncurses and refuse to use it. Spending our time dealing with brazenly outdated libraries embedded in someone elses toolchain that we don't ship our own builds with isn't worthwhile (it'd never end).
Thank you all for looking into this - as next steps I've found the Feedback Assistant and the general feedback form on Apple's website. I'll work to post to those methods and provide any feedback here.
I could also look at writing a small C example and see if I can confirm if the problem is in their ncurses implementation and fully removed from Python or not.
Reacted by Gregory P. SmithIf you want to require a specific version or recommend libedit or ncurses on macOS, you can update the doc: https://docs.python.org/dev/using/configure.html
Issues like this should also be reported to Apple as this is a crash with their copy of Python.
I've filed FB13196764 about this.
Reacted by Gregory P. Smith, Victor Stinner, Tim Way and Erlend E. Aasland16 remaining items
if (HAVE_CURSES_IS_PAD_RUNTIME) { return is_pad(w); } else { // ... what goes here? ... }Don't define
is_padmacro if the function is not available.Modules/_cursesmodule.ccan already be built without is_pad() function. Example in _curses_window_echochar_impl():#ifdef py_is_pad if (py_is_pad(self->win)) { return PyCursesCheckERR(pechochar(self->win, ch_ | (attr_t)attr), "echochar"); } else #endif return PyCursesCheckERR(wechochar(self->win, ch_ | (attr_t)attr), "echochar");
Or well, just always return 0?
Modules/_cursesmodule.ccan already be built without is_pad() functionEh, kinda, but not really in a way that solves the problem. It falls back in two different ways, depending on checks done at configure time:
- if is_pad is a function, define py_is_pad to use that;
- otherwise: if is_pad is not available1, but window flags are inspectable, it defines py_is_pad to check flags. This is what happens on older Apple SDKs (which work!) and on any system with ncurses < 5.8.
- otherwise: don't define
py_is_padat all, and ignore the distinction between windows and pads. This is meant to support some other curses, and is never the case with ncurses.
What you are suggesting is 3, but ncurses doesn't like that, e.g. curs_pad(3x):
It is not legal to call wrefresh with a pad as an argument; the routines prefresh or pnoutrefresh should be called instead.In the example you pasted, this would end up calling
wechocharon a pad instead ofpechochar, which might not work as expected, or impact performance. Note that this might break in a way that is not covered by test_curses. I don't see tests on pads stuff.
On a personal note, I'm getting confused by this conversation. I suggested a fix and tested it. It took some work to figure out the different cases, and I documented that work in this issue. It's not really complicated, only a bit messy with all the historical cruft. But the comments don't address the proposed solution, and seem to argue that some other unspecified solution should be preferable. I have a bucketful of impostor syndrome to handle, and I'd prefer not to feed it more.
Footnotes
-
yeah, in the current setup, with ncurses, either
is_padis a function (detected by configure, which would set HAVE_CURSES_IS_PAD) or it's not used at all. There is no path that uses theis_padmacro from ncurses.h, which is legal and documented. ↩
I understand that with Xcode 15.0, the is_pad() function is not available, but
win->_flags & _ISPADcan be tested if theNCURSES_OPAQUEmacro is set to 0. For me, it's surprising to have to tune NCURSES_OPAQUE macro and to inspect a structure member starting with an underscore.But apparently, this issue affects many macOS issues, it's annoying, and I should stop nitpicking. So just use NCURSES_OPAQUE=0 on macOS, and it should "just works", no?
Does someone want to propose a PR? Nobody proposed a PR so far, no?
I understand that with Xcode 15.0, the is_pad() function is not available, but
win->_flags & _ISPADcan be tested if theNCURSES_OPAQUEmacro is set to 0. For me, it's surprising to have to tune NCURSES_OPAQUE macro and to inspect a structure member starting with an underscore.Nope. "Inspecting a structure member starting with an underscore" is what cursesmodule.c does now and has been doing for the last couple decades. If you built on Mac before Xcode 15, you used that.
There is a convenient and documented
is_padmacro, defined in Ncurses public headers for a long while now. Since Apple now includes this version, we can use it.Does someone want to propose a PR? Nobody proposed a PR so far, no?
I asked for early feedback on a diff a couple times above. I will open a PR in a little bit. Maybe it will reduce the confusion.
Suggestion: let's rename the issue to something that better describes the problem.
Is there a workaround? I want to get back to work on my project.
Is there a workaround? I want to get back to work on my project.
I'm no longer able to replicate the issue with Sonoma 14.2.1 and Command Line Tools for Xcode 15.1.
Alternatively, versions from Python.org shouldn't have the issue.
Edit: It looks like 14.2 got some patches for ncurses behavior (https://support.apple.com/en-us/HT214036). While they don't specifically mention this as an issue it does appear to resolve it for me at least.
Yes, downloading the latest directly from python.org made the problem go away. It would be nice if there were something I could do in code, other than telling customers "you need to install a new version of Python on your system"
- added a commit that references this issue
on Jun 14, 2024 FYI, as of the Python 3.14.0 alpha 5 pre-release and the upcoming 3.13.3 and 3.12.10 releases, python.org macOS installers now build and link with ncurses 6.5, the current stable release, and no longer with either ncurses 5.9 or with the Apple-supplied and -patched macOS system ncurses.
Since I think the original issue reported here has been addressed by PR #111258 and possibly by Apple - at least there don't seem to be recent reports of people being bitten by this - let's close this issue as fixed.
Crash report
What happened?
MacOS began pushing out updates to XCode Command Line Tools to install 15.0 recently. Upon updating I began having issues with
curses. This happens with the Python provided by Apple. I'm not aware of the best way to communicate this issue to Apple, hopefully someone here knows who to ping or is watching.Save the below as
curses-segfault.py:Run the script in
zshin MacOS Terminal via:An easy way to see if you got the update is via the terminal by running:
CPython versions tested on:
3.9
Operating systems tested on:
macOS
Output from running 'python -VV' on the command line:
python3 -VV Python 3.9.6 (default, Aug 11 2023, 19:44:49) [Clang 15.0.0 (clang-1500.0.40.1)]
Linked PRs