How to link with a non-system libffi?

@ned-deily

BPO	14527
Nosy	@ned-deily, @pmp-p, @rupertnash, @munocat, @thesamesam
PRs	bpo-14527: fix (some) _ctypes liffi build problems #20451

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2012-04-08.02:25:34.547>
labels = ['build', '3.9', '3.10', '3.11', 'ctypes', 'type-feature']
title = 'How to link with a non-system libffi?'
updated_at = <Date 2021-10-29.16:13:24.306>
user = 'https://bugs.python.org/pda'

bugs.python.org fields:

activity = <Date 2021-10-29.16:13:24.306>
actor = 'thesamesam'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Build', 'ctypes']
creation = <Date 2012-04-08.02:25:34.547>
creator = 'pda'
dependencies = []
files = []
hgrepos = []
issue_num = 14527
keywords = ['patch']
message_count = 13.0
messages = ['157776', '157777', '157795', '157932', '158004', '368399', '368413', '370088', '370580', '376885', '381577', '381601', '389390']
nosy_count = 10.0
nosy_names = ['ned.deily', 'pmpp', 'rosslagerwall', 'pda', 'rnash', 'Joshua Merchant', 'munocat', 'zhao.wang.unsw', 'raphael.kl', 'thesamesam']
pr_nums = ['20451']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue14527'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

I trying to build python using an external libffi package I have installed -- is there some trick in directing --with-system-ffi to the path where it's located. I don't see clues in config.log or anywhere to help.

If it is in a non-standard location, try setting the environment variables:

LDFLAGS linker flags, e.g. -L<lib dir> if you have libraries in a nonstandard directory <lib dir>
CPPFLAGS (Objective) C/C++ preprocessor flags, e.g. -I<include dir> if you have headers in a nonstandard directory <include dir>

On Sun, Apr 08, 2012 at 05:58:29AM +0000, Ross Lagerwall wrote:

Ross Lagerwall <rosslagerwall@gmail.com> added the comment:

If it is in a non-standard location, try setting the environment variables:

LDFLAGS linker flags, e.g. -L<lib dir> if you have libraries in a nonstandard directory <lib dir>
CPPFLAGS (Objective) C/C++ preprocessor flags, e.g. -I<include dir> if you have headers in a nonstandard directory <include dir>

I'm pretty sure I've already been doing that, but I'll verify again soon
when I get the chance.

Yes, I think my libffi setup is okay, but python apparently doesn't (according to the deeper-down log files I didn't initially know about). The following is a suspicious-looking snippet from build/temp.hp-ux-B.11.31-ia64-2.7/libffi/config.log... I have to question the usefulness of that linker error message. My immediate thought is that maybe conftstm.o is a 32-bit object file, but I don't see anything earlier in the log to indicate it was even created.

configure:6159: gcc463 -o conftest -I. -IInclude -I./Include -D_TERMIOS_INCLUDED -I/usr/local/lp64/include -mlp64 -L/usr/local/src/Python-2.7.2 -L/usr/local/lp64/lib conftest.c conftstm.o >&5
ld: Mismatched Data ABI. Expected EF_IA_64_ABI64 but found None in file conftstm.o
Fatal error.

While this is no solution by any means, I think it'd be better for the scenario to be a fatal configure error. After all, if I say --with-system-ffi, it means I really, really want want to use my own libffi.

As of current Python 3 releases, like 3.8, the python build no longer vendors a copy of libffi for most Unix systems and for those systems the --with-system-ffi configure option is ignored, i.e. configure and setup.py will always try to find an external libffi. On most systems, if you use are OK with using the system-supplied libffi, building and execution should usually "just work". But if you want to use another version of libffi, for example, one that you build from source, it can be very non-intuitive of how to successfully do that *especially* if there is already a system libffi installed. Depending on the system, you may need to set environment build variables like LDFLAGS, CFLAGS or PKG_CONFIG_PATH, and LD_LIBRARY_PATH for the Python build and/or you might need to override some of the libffi build variables to install in the proper locations for your system. It seems like, as has been suggested in various places, setup.py should at least try to use pkg-config to find libffi library locations as it already does to find libffi include files. But that might not be enough to get a working setup, i.e. you still might need to set LD_LIBRARY_PATH or the equivalent to find libffi. At the very least, there should be some documentation on how to build with a non-system libffi.

See also earlier discussions in bpo-31710, bpo-34823, bpo-40488, and bpo-40535, closed as duplicated of this issue.

I have just struggled with building CPython with the _ctypes module. Fundamentally, the problem appears to be that configure uses pkgconfig to find the libffi include directory, while setup.py's detect_ctypes only uses the global list of library directories.

I have made an attempt at fixing this by having configure produce the directory containing libffi (LIBFFI_LIBDIR) and setup.py use this. However I've hardly any experience with autotools, so I would be very happy to be corrected if this is no use at all.

The PR is #20451

I think the actual PR20451 with its proper use of pkgconfig could close https://bugs.python.org/issue31710 ( fixing at once android and wasm/wasi compilation )

Due to some major changes in setup.py around libffi, I've had to re-implement my patch - please see Github. I'd appreciate any comments so we can get this merged. Thanks!

I downloaded Python-3.10.4.tar.xz and applied https://github.com/python/cpython/pull/20451.diff . I want to compile Python to run in a chrooted environment. I call

../configure --with-lto --enable-shared --without-static-libpython --prefix=/root/chroot-n/py2 --with-system-ffi --disable-ipv6 --enable-optimizations CFLAGS='--sysroot=/root/chroot-n -Ichr/include'

With this patch _ctypes is linked correctly.

After compiling libffi from source with the following commands:

rm /usr/lib64/libffi.*
./configure --prefix=/usr --disable-static --libdir=/usr/lib --disable-multi-os-directory --with-gcc-arch=native

I was able to build _ctypes.

The mentioned command installs the binaries at /usr/lib/ instead of /usr/lib64/

The setup changed again; setup.py and --with-system-ffi is no more. The environment variables to set are listed in ./configure --help:

  LIBFFI_CFLAGS
              C compiler flags for LIBFFI, overriding pkg-config
  LIBFFI_LIBS linker flags for LIBFFI, overriding pkg-config

For example: LIBFFI_CFLAGS="-I $HOME/include/" LIBFFI_LIBS="-L $HOME/lib64/ -lffi"

I'll close the issue. Please open a new one if these environment variables don't work.

Navigation Menu

Repository navigation

How to link with a non-system libffi? #58732

4 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

How to link with a non-system libffi? #58732

Description

Activity

pda commented on Apr 8, 2012

rosslagerwall commented on Apr 8, 2012

pda commented on Apr 8, 2012

pda commented on Apr 10, 2012

pda commented on Apr 11, 2012

ned-deily commented on May 8, 2020

ned-deily commented on May 8, 2020

rupertnash commented on May 27, 2020

4 remaining items

pmp-p commented on Nov 22, 2020

rupertnash commented on Mar 23, 2021

dilyanpalauzov commented on Apr 15, 2022

MhmRhm commented on Dec 30, 2023

encukou commented on Mar 25, 2024

encukou commented on Apr 1, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions