Installation on NERSC #24

Activity

1. mreineck commented on Jun 17, 2019

   

   mreineck

   on Jun 17, 2019

   Collaborator

   More actions

   Can you try adding -fPIC to CCFLAGS_NO_C and CLFLAGS in the config.auto file and recompiling?
2. abaleato commented on Jun 17, 2019

   

   abaleato

   on Jun 17, 2019

   Last edited by abaleato

   Author

   More actions

   Thank you very much for your fast response.

   I just tried your recommendation, and although the error message is different, we're still not quite there.

   To be clear, what I did was: 1) delete my libsharp directory, clone it again. 2) follow the instructions in the readme, making sure to add -fPIC to CCFLAGS_NO_C and CLFLAGS in config/config.auto before running make. 3) define LIBSHARP=/global/homes/a/ab2368/Software/libsharp/auto/, then export LIBSHARP. 4) Finally, cd python and then python setup.py install, after which I obtain the following error message:

   (delensing_modfsht) ab2368@cori09:~/Software/libsharp/python> python setup.py install Compiling libsharp/libsharp.pyx because it changed. Compiling libsharp/libsharp_mpi.pyx because it changed. [1/2] Cythonizing libsharp/libsharp.pyx /global/homes/a/ab2368/.conda/envs/delensing_modfsht/lib/python2.7/site-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /global/u1/a/ab2368/Software/libsharp/python/libsharp/libsharp.pxd tree = Parsing.p_module(s, pxd, full_module_name) [2/2] Cythonizing libsharp/libsharp_mpi.pyx /global/homes/a/ab2368/.conda/envs/delensing_modfsht/lib/python2.7/site-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /global/u1/a/ab2368/Software/libsharp/python/libsharp/libsharp_mpi.pyx tree = Parsing.p_module(s, pxd, full_module_name) running install running bdist_egg running egg_info creating libsharp.egg-info writing requirements to libsharp.egg-info/requires.txt writing libsharp.egg-info/PKG-INFO writing top-level names to libsharp.egg-info/top_level.txt writing dependency_links to libsharp.egg-info/dependency_links.txt writing manifest file 'libsharp.egg-info/SOURCES.txt' reading manifest file 'libsharp.egg-info/SOURCES.txt' writing manifest file 'libsharp.egg-info/SOURCES.txt' installing library code to build/bdist.linux-x86_64/egg running install_lib running build_py creating build creating build/lib.linux-x86_64-2.7 creating build/lib.linux-x86_64-2.7/libsharp copying libsharp/__init__.py -> build/lib.linux-x86_64-2.7/libsharp creating build/lib.linux-x86_64-2.7/libsharp/tests copying libsharp/tests/test_smoothing_noise_pol_mpi.py -> build/lib.linux-x86_64-2.7/libsharp/tests copying libsharp/tests/test_legendre_table.py -> build/lib.linux-x86_64-2.7/libsharp/tests copying libsharp/tests/test_sht.py -> build/lib.linux-x86_64-2.7/libsharp/tests copying libsharp/tests/__init__.py -> build/lib.linux-x86_64-2.7/libsharp/tests copying libsharp/tests/test_legendre.py -> build/lib.linux-x86_64-2.7/libsharp/tests running build_ext building 'libsharp.libsharp' extension creating build/temp.linux-x86_64-2.7 creating build/temp.linux-x86_64-2.7/libsharp gcc -pthread -B /global/homes/a/ab2368/.conda/envs/delensing_modfsht/compiler_compat -Wl,--sysroot=/ -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/global/homes/a/ab2368/Software/libsharp/auto/include -I/global/homes/a/ab2368/.conda/envs/delensing_modfsht/lib/python2.7/site-packages/numpy/core/include -I/global/homes/a/ab2368/.conda/envs/delensing_modfsht/include/python2.7 -c libsharp/libsharp.c -o build/temp.linux-x86_64-2.7/libsharp/libsharp.o In file included from /global/homes/a/ab2368/.conda/envs/delensing_modfsht/lib/python2.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1824:0, from /global/homes/a/ab2368/.conda/envs/delensing_modfsht/lib/python2.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:12, from /global/homes/a/ab2368/.conda/envs/delensing_modfsht/lib/python2.7/site-packages/numpy/core/include/numpy/arrayobject.h:4, from libsharp/libsharp.c:630: /global/homes/a/ab2368/.conda/envs/delensing_modfsht/lib/python2.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp] #warning "Using deprecated NumPy API, disable it with " \ ^~~~~~~ gcc -pthread -shared -B /global/homes/a/ab2368/.conda/envs/delensing_modfsht/compiler_compat -L/global/homes/a/ab2368/.conda/envs/delensing_modfsht/lib -Wl,-rpath=/global/homes/a/ab2368/.conda/envs/delensing_modfsht/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-2.7/libsharp/libsharp.o -L/global/homes/a/ab2368/Software/libsharp/auto/lib -L/global/homes/a/ab2368/.conda/envs/delensing_modfsht/lib -lsharp -lfftpack -lc_utils -lpython2.7 -o build/lib.linux-x86_64-2.7/libsharp/libsharp.so -fopenmp building 'libsharp.libsharp_mpi' extension gcc -pthread -B /global/homes/a/ab2368/.conda/envs/delensing_modfsht/compiler_compat -Wl,--sysroot=/ -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/global/homes/a/ab2368/Software/libsharp/auto/include -I/global/homes/a/ab2368/.conda/envs/delensing_modfsht/lib/python2.7/site-packages/numpy/core/include -I/global/homes/a/ab2368/.conda/envs/delensing_modfsht/include/python2.7 -c libsharp/libsharp_mpi.c -o build/temp.linux-x86_64-2.7/libsharp/libsharp_mpi.o libsharp/libsharp_mpi.c:619:10: fatal error: mpi.h: No such file or directory #include "mpi.h" ^~~~~~~ compilation terminated. error: command 'gcc' failed with exit status 1

   even recompiling with ./configure --disable-mpi --enable-pic does not do it... I shall keep trying.

   Thanks again for your help!
3. mreineck commented on Jun 18, 2019

   

   mreineck

   on Jun 18, 2019

   Collaborator

   More actions

   Sorry, I fear I'm not familiar enough with the Python<->C ecosystem to provide any useful input. @dagss, any insight?
4. abaleato commented on Jun 18, 2019

   

   abaleato

   on Jun 18, 2019

   Author

   More actions

   Thank you very much @mreineck !
5. Gabriel-Ducrocq commented on Dec 30, 2019

   

   Gabriel-Ducrocq

   on Dec 30, 2019

   More actions

   Hi everyone,

   I have the exact same problem as mentioned by @abaleato: when I run python setup.py install to install the python wrapper, I get the following error:

   Compiling libsharp/libsharp.pyx because it changed. Compiling libsharp/libsharp_mpi.pyx because it changed. [1/2] Cythonizing libsharp/libsharp.pyx /home/gducrocq/anaconda3/lib/python3.7/site-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /home/gducrocq/libsharp/python/libsharp/libsharp.pxd tree = Parsing.p_module(s, pxd, full_module_name) [2/2] Cythonizing libsharp/libsharp_mpi.pyx /home/gducrocq/anaconda3/lib/python3.7/site-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /home/gducrocq/libsharp/python/libsharp/libsharp_mpi.pyx tree = Parsing.p_module(s, pxd, full_module_name) running install running bdist_egg running egg_info writing libsharp.egg-info/PKG-INFO writing dependency_links to libsharp.egg-info/dependency_links.txt writing requirements to libsharp.egg-info/requires.txt writing top-level names to libsharp.egg-info/top_level.txt reading manifest file 'libsharp.egg-info/SOURCES.txt' writing manifest file 'libsharp.egg-info/SOURCES.txt' installing library code to build/bdist.linux-x86_64/egg running install_lib running build_py running build_ext building 'libsharp.libsharp' extension gcc -pthread -B /home/gducrocq/anaconda3/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Ilibsharp -I/home/gducrocq/libsharp/auto/include -I/home/gducrocq/anaconda3/lib/python3.7/site-packages/numpy/core/include -I/home/gducrocq/anaconda3/include/python3.7m -c libsharp/libsharp.c -o build/temp.linux-x86_64-3.7/libsharp/libsharp.o In file included from /home/gducrocq/anaconda3/lib/python3.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1824:0, from /home/gducrocq/anaconda3/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:12, from /home/gducrocq/anaconda3/lib/python3.7/site-packages/numpy/core/include/numpy/arrayobject.h:4, from libsharp/libsharp.c:631: /home/gducrocq/anaconda3/lib/python3.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp] #warning "Using deprecated NumPy API, disable it with " \ ^ gcc -pthread -shared -B /home/gducrocq/anaconda3/compiler_compat -L/home/gducrocq/anaconda3/lib -Wl,-rpath=/home/gducrocq/anaconda3/lib-Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.7/libsharp/libsharp.o -L/home/gducrocq/libsharp/auto/lib -lsharp -lfftpack-lc_utils -o build/lib.linux-x86_64-3.7/libsharp/libsharp.cpython-37m-x86_64-linux-gnu.so -fopenmp building 'libsharp.libsharp_mpi' extension gcc -pthread -B /home/gducrocq/anaconda3/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/gducrocq/libsharp/auto/include -I/home/gducrocq/anaconda3/lib/python3.7/site-packages/numpy/core/include -I/home/gducrocq/anaconda3/include/python3.7m -c libsharp/libsharp_mpi.c -o build/temp.linux-x86_64-3.7/libsharp/libsharp_mpi.o libsharp/libsharp_mpi.c:619:17: fatal error: mpi.h: No such file or directory compilation terminated. error: command 'gcc' failed with exit status 1

   Did you find a way to handle that error ?

   Thank you,
   Gabriel.
6. Gabriel-Ducrocq commented on Dec 31, 2019

   

   Gabriel-Ducrocq

   on Dec 31, 2019

   More actions

   I finally found a temporary way around this:

   After adding -fPIC to CCFLAGS_NO_C and CLFLAGS in the config.auto file and recompiling, I removed the libsharp_mpi extension from the setup.py file.

   Then I ran python setup.py install and everything went fine. But when I tried:
   import libsharp
   I got the following error:
   Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/gducrocq/libsharp/python/libsharp/__init__.py", line 1, in <module> from .libsharp import * ModuleNotFoundError: No module named 'libsharp.libsharp'

   It seems that's because the folder of the module in the site-packages of my environment was named libsharp-0.1-py3.7-linux-x86_64.egg. Renaming this folder as libsharp made the trick, and now I can import it and use it.

   I guess this comes at the cost of efficiency but at least I can test and debugg my code.
   Hope this helps :)