coreneuron_modtests::{direct_hoc,datareturn_py} tests sometimes fail on GPU

Describe the issue
The following CI tests sometimes fail, for example in this NMODL+GPU job on 4c6e6aa:

The following tests FAILED:
	 10 - coreneuron_modtests::direct_hoc (Failed)
	 13 - coreneuron_modtests::datareturn_py (Failed)

In this case the coreneuron_modtests::direct_hoc failure is in the i_mem check, which was re-enabled by neuronsimulator/nrn#1313 after #574 was merged.

The direct_py test also fails, but is not registered as failing by CTest. The non-failure should be addressed by neuronsimulator/nrn#1358.

The datareturn_py test in that job fails with:

13:  Start time (t) = 0
13: 
13:  Memory (MBs) :  After mk_spikevec_buffer : Max 341.0586, Min 341.0586, Avg 341.0586 
13: (null) lives at 0x2813880 size 36 partially present
13: Present table dump for device[1]: NVIDIA Tesla GPU 0, compute capability 7.0, threadid=1
13: host:0xad67d0 device:0x2acaffe00d00 size:56 presentcount:0+1 line:-1 name:_ZN10coreneuron14Exp2Syn_globalE
13: host:0xad6860 device:0x2acaffe00e00 size:48 presentcount:0+1 line:-1 name:_ZN10coreneuron13ExpSyn_globalE
13: host:0xad68e0 device:0x2acaffe00f00 size:24 presentcount:0+1 line:-1 name:_ZN10coreneuron20ForNetConTest_globalE
13: host:0xad6950 device:0x2acaffe01000 size:64 presentcount:0+1 line:-1 name:_ZN10coreneuron9hh_globalE
13: host:0xad6a50 device:0x2acaffe01100 size:24 presentcount:0+1 line:-1 name:_ZN10coreneuron14NaTrans_globalE
13: host:0xad6b30 device:0x2acaffe01200 size:16 presentcount:0+1 line:-1 name:_ZN10coreneuron10pas_globalE
13: host:0xad6c00 device:0x2acaffe01300 size:16 presentcount:0+1 line:-1 name:_ZN10coreneuron13Sample_globalE
13: host:0xad6c60 device:0x2acaffe01400 size:24 presentcount:0+1 line:-1 name:_ZN10coreneuron13IClamp_globalE
13: host:0xad6cd0 device:0x2acaffe01500 size:24 presentcount:0+1 line:-1 name:_ZN10coreneuron14SEClamp_globalE
13: host:0xad6d40 device:0x2acaffe01600 size:24 presentcount:0+1 line:-1 name:_ZN10coreneuron16UnitsTest_globalE
13: host:0xad8720 device:0x2acaffe01800 size:4 presentcount:0+1 line:-1 name:_ZN10coreneuron11secondorderE
13: host:0xad8738 device:0x2acaffe01700 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron7celsiusE
13: host:0xad8740 device:0x2acaffe01900 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron2piE
13: host:0x1d80240 device:0x2acaffb04000 size:232 presentcount:0+1 line:-1 name:(null)
13: host:0x1d803c0 device:0x2acaffb0c400 size:72 presentcount:0+1 line:-1 name:(null)
13: host:0x1d80480 device:0x2acaffb0bc00 size:72 presentcount:0+1 line:-1 name:(null)
13: host:0x2811a80 device:0x2acaffb09a00 size:72 presentcount:0+1 line:-1 name:(null)
13: host:0x28138a0 device:0x2acaffb0b200 size:16 presentcount:0+1 line:-1 name:(null)
13: host:0x2813c50 device:0x2acaffb0a200 size:16 presentcount:0+1 line:-1 name:(null)
13: host:0x2813e80 device:0x2acaffb04200 size:40 presentcount:0+1 line:-1 name:(null)
13: host:0x281df00 device:0x2acaffb04600 size:40 presentcount:0+1 line:-1 name:(null)
13: host:0x281e540 device:0x2acaffb0b000 size:40 presentcount:0+1 line:-1 name:(null)
13: host:0x2820188 device:0x2acaffafa600 size:912 presentcount:0+1 line:-1 name:(null)
13: host:0x28241c0 device:0x2acaffb09c00 size:204 presentcount:0+1 line:-1 name:(null)
13: host:0x28242c0 device:0x2acaffb0a400 size:192 presentcount:0+1 line:-1 name:(null)
13: host:0x28243c0 device:0x2acaffb0ce00 size:204 presentcount:0+1 line:-1 name:(null)
13: host:0x28244c0 device:0x2acaffb0d000 size:224 presentcount:0+1 line:-1 name:(null)
13: host:0x2824600 device:0x2acaffb0be00 size:204 presentcount:0+1 line:-1 name:(null)
13: host:0x2824700 device:0x2acaffb04400 size:224 presentcount:0+1 line:-1 name:(null)
13: host:0x3216030 device:0x2acaffb0dc00 size:136 presentcount:0+1 line:-1 name:(null)
13: host:0x32160c0 device:0x2acaffafa000 size:24 presentcount:0+1 line:-1 name:(null)
13: host:0x324b6d0 device:0x2acaffafa400 size:24 presentcount:0+1 line:-1 name:(null)
13: host:0x32e1480 device:0x2acaffb0a600 size:40 presentcount:0+1 line:-1 name:(null)
13: host:0x32e31c0 device:0x2acaffb04800 size:72 presentcount:0+1 line:-1 name:(null)
13: host:0x32e3e80 device:0x2acaffb0a000 size:72 presentcount:0+1 line:-1 name:(null)
13: host:0x32e9480 device:0x2acaffb09e00 size:40 presentcount:0+1 line:-1 name:(null)
13: host:0x3433940 device:0x2acaffafa200 size:276 presentcount:0+1 line:-1 name:(null)
13: host:0x344b340 device:0x2acaffafaa00 size:24640 presentcount:0+1 line:-1 name:(null)
13: allocated block device:0x2acaffafa000 size:512 thread:1
13: allocated block device:0x2acaffafa200 size:512 thread:1
13: allocated block device:0x2acaffafa400 size:512 thread:1
13: allocated block device:0x2acaffafa600 size:1024 thread:1
13: allocated block device:0x2acaffafaa00 size:38400 thread:1
13: allocated block device:0x2acaffb04000 size:512 thread:1
13: allocated block device:0x2acaffb04200 size:512 thread:1
13: allocated block device:0x2acaffb04400 size:512 thread:1
13: allocated block device:0x2acaffb04600 size:512 thread:1
13: allocated block device:0x2acaffb04800 size:512 thread:1
13: allocated block device:0x2acaffb09a00 size:512 thread:1
13: allocated block device:0x2acaffb09c00 size:512 thread:1
13: allocated block device:0x2acaffb09e00 size:512 thread:1
13: allocated block device:0x2acaffb0a000 size:512 thread:1
13: allocated block device:0x2acaffb0a200 size:512 thread:1
13: allocated block device:0x2acaffb0a400 size:512 thread:1
13: allocated block device:0x2acaffb0a600 size:512 thread:1
13: allocated block device:0x2acaffb0b000 size:512 thread:1
13: allocated block device:0x2acaffb0b200 size:512 thread:1
13: allocated block device:0x2acaffb0bc00 size:512 thread:1
13: allocated block device:0x2acaffb0be00 size:512 thread:1
13: allocated block device:0x2acaffb0c400 size:512 thread:1
13: allocated block device:0x2acaffb0ce00 size:512 thread:1
13: allocated block device:0x2acaffb0d000 size:512 thread:1
13: allocated block device:0x2acaffb0dc00 size:512 thread:1
13: deleted block   device:0x2acaffb0ae00 size:512 threadid=1 
13: deleted block   device:0x2acaffb05600 size:512 threadid=1 
13: deleted block   device:0x2acaffb04a00 size:512 threadid=1 
13: deleted block   device:0x2acaffb04c00 size:512 threadid=1 
13: deleted block   device:0x2acaffb04e00 size:512 threadid=1 
13: deleted block   device:0x2acaffb06e00 size:512 threadid=1 
13: deleted block   device:0x2acaffb07000 size:512 threadid=1 
13: deleted block   device:0x2acaffb07200 size:512 threadid=1 
13: deleted block   device:0x2acaffb07400 size:512 threadid=1 
13: deleted block   device:0x2acaffb05800 size:512 threadid=1 
13: deleted block   device:0x2acaffb05a00 size:512 threadid=1 
13: deleted block   device:0x2acaffb05c00 size:512 threadid=1 
13: deleted block   device:0x2acaffb05e00 size:512 threadid=1 
13: deleted block   device:0x2acaffb05000 size:512 threadid=1 
13: deleted block   device:0x2acaffb05200 size:512 threadid=1 
13: deleted block   device:0x2acaffb05400 size:512 threadid=1 
13: deleted block   device:0x2acaffb06800 size:512 threadid=1 
13: deleted block   device:0x2acaffb06a00 size:512 threadid=1 
13: deleted block   device:0x2acaffb06c00 size:512 threadid=1 
13: deleted block   device:0x2acaffb09400 size:512 threadid=1 
13: deleted block   device:0x2acaffb09600 size:512 threadid=1 
13: deleted block   device:0x2acaffb09800 size:512 threadid=1 
13: deleted block   device:0x2acaffb07600 size:512 threadid=1 
13: deleted block   device:0x2acaffb07800 size:512 threadid=1 
13: deleted block   device:0x2acaffb07a00 size:512 threadid=1 
13: deleted block   device:0x2acaffb06000 size:512 threadid=1 
13: deleted block   device:0x2acaffb06200 size:512 threadid=1 
13: deleted block   device:0x2acaffb06400 size:512 threadid=1 
13: deleted block   device:0x2acaffb06600 size:512 threadid=1 
13: deleted block   device:0x2acaffb0a800 size:512 threadid=1 
13: deleted block   device:0x2acaffb0aa00 size:512 threadid=1 
13: deleted block   device:0x2acaffb0ac00 size:512 threadid=1 
13: deleted block   device:0x2acaffb08600 size:512 threadid=1 
13: deleted block   device:0x2acaffb08800 size:512 threadid=1 
13: deleted block   device:0x2acaffb08a00 size:512 threadid=1 
13: deleted block   device:0x2acaffb08c00 size:512 threadid=1 
13: deleted block   device:0x2acaffb08e00 size:512 threadid=1 
13: deleted block   device:0x2acaffb09000 size:512 threadid=1 
13: deleted block   device:0x2acaffb09200 size:512 threadid=1 
13: deleted block   device:0x2acaffb0d200 size:512 threadid=1 
13: deleted block   device:0x2acaffb0d400 size:512 threadid=1 
13: deleted block   device:0x2acaffb0d800 size:512 threadid=1 
13: deleted block   device:0x2acaffb0d600 size:512 threadid=1 
13: deleted block   device:0x2acaffb0da00 size:512 threadid=1 
13: deleted block   device:0x2acaffb0b400 size:1024 threadid=1 
13: deleted block   device:0x2acaffb0b800 size:1024 threadid=1 
13: deleted block   device:0x2acaffb0c000 size:1024 threadid=1 
13: deleted block   device:0x2acaffb0c600 size:2048 threadid=1 
13: deleted block   device:0x2acaffb07c00 size:2560 threadid=1 
13: FATAL ERROR: variable in data clause is partially present on the device: name=(unknown)
13:  file:(OpenACC API) acc_copyin line:-1
13:

To Reproduce
We should use this ticket to collect more information. For example:

does it occur with MOD2C as well as NMODL?
...

Expected behavior
The test should pass reliably.

System (please complete the following information)

OS: BB5
Compiler: NVHPC 21.2
Version: master
Backend: GPU

In https://bbpgitlab.epfl.ch/hpc/coreneuron/-/jobs/33972 the datareturn_py test failed again, this time with MOD2C not NMODL:

13: (null) lives at 0x281a740 size 40 partially present
13: Present table dump for device[1]: NVIDIA Tesla GPU 0, compute capability 7.0, threadid=1
13: host:0xa07870 device:0x2acaffe01700 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron13mol_UnitsTestE
13: host:0xa07878 device:0x2acaffe01800 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron11E_UnitsTestE
13: host:0xa07880 device:0x2acaffe01900 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron17FARADAY_UnitsTestE
13: host:0xa07888 device:0x2acaffe01a00 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron11h_UnitsTestE
13: host:0xa07890 device:0x2acaffe01b00 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron12hb_UnitsTestE
13: host:0xa07898 device:0x2acaffe01c00 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron11R_UnitsTestE
13: host:0xa078a0 device:0x2acaffe01d00 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron19boltzmann_UnitsTestE
13: host:0xa078a8 device:0x2acaffe01e00 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron13avo_UnitsTestE
13: host:0xae3a68 device:0x2acaffe00b00 size:8 presentcount:0+1 line:-1 name:_st__gpfs_bbp_cscs_ch_ssd_gitlab_map_jobs_bbpcihpcproj12_P10480_33968_spack_build_spack_stage_neuron_develop_qdta367h6ly225azfnup34o2tpbkhbcb_spack_build_qdta367_test_nrnivmodl_8e220c327f2b8882adcf04884baa4209f37d0bbcef5677f046766f546d969ffd_x86_64_corenrn_mod2c_exp2syn_cpp__ZN54_INTERNAL_32_x86_64_corenrn_mod2c_exp2syn_cpp_25c7f1d810coreneuron2A0E
13: host:0xae3a70 device:0x2acaffe00c00 size:8 presentcount:0+1 line:-1 name:_st__gpfs_bbp_cscs_ch_ssd_gitlab_map_jobs_bbpcihpcproj12_P10480_33968_spack_build_spack_stage_neuron_develop_qdta367h6ly225azfnup34o2tpbkhbcb_spack_build_qdta367_test_nrnivmodl_8e220c327f2b8882adcf04884baa4209f37d0bbcef5677f046766f546d969ffd_x86_64_corenrn_mod2c_exp2syn_cpp__ZN54_INTERNAL_32_x86_64_corenrn_mod2c_exp2syn_cpp_25c7f1d810coreneuron2B0E
13: host:0xae3aa8 device:0x2acaffe00900 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron15_slist1_Exp2SynE
13: host:0xae3ab0 device:0x2acaffe00a00 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron15_dlist1_Exp2SynE
13: host:0xae3ae8 device:0x2acaffe00f00 size:8 presentcount:0+1 line:-1 name:_st__gpfs_bbp_cscs_ch_ssd_gitlab_map_jobs_bbpcihpcproj12_P10480_33968_spack_build_spack_stage_neuron_develop_qdta367h6ly225azfnup34o2tpbkhbcb_spack_build_qdta367_test_nrnivmodl_8e220c327f2b8882adcf04884baa4209f37d0bbcef5677f046766f546d969ffd_x86_64_corenrn_mod2c_expsyn_cpp__ZN53_INTERNAL_31_x86_64_corenrn_mod2c_expsyn_cpp_0127273610coreneuron2g0E
13: host:0xae3b18 device:0x2acaffe00d00 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron14_slist1_ExpSynE
13: host:0xae3b20 device:0x2acaffe00e00 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron14_dlist1_ExpSynE
13: host:0xae3b50 device:0x2acaffe01000 size:4 presentcount:0+1 line:-1 name:_ZN10coreneuron23_mechtype_ForNetConTestE
13: host:0xae3ba8 device:0x2acaffe01100 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron12_celsius__hhE
13: host:0xae3bb0 device:0x2acaffe01400 size:8 presentcount:0+1 line:-1 name:_st__gpfs_bbp_cscs_ch_ssd_gitlab_map_jobs_bbpcihpcproj12_P10480_33968_spack_build_spack_stage_neuron_develop_qdta367h6ly225azfnup34o2tpbkhbcb_spack_build_qdta367_test_nrnivmodl_8e220c327f2b8882adcf04884baa4209f37d0bbcef5677f046766f546d969ffd_x86_64_corenrn_mod2c_hh_cpp__ZN49_INTERNAL_27_x86_64_corenrn_mod2c_hh_cpp_4898369610coreneuron2h0E
13: host:0xae3bb8 device:0x2acaffe01500 size:8 presentcount:0+1 line:-1 name:_st__gpfs_bbp_cscs_ch_ssd_gitlab_map_jobs_bbpcihpcproj12_P10480_33968_spack_build_spack_stage_neuron_develop_qdta367h6ly225azfnup34o2tpbkhbcb_spack_build_qdta367_test_nrnivmodl_8e220c327f2b8882adcf04884baa4209f37d0bbcef5677f046766f546d969ffd_x86_64_corenrn_mod2c_hh_cpp__ZN49_INTERNAL_27_x86_64_corenrn_mod2c_hh_cpp_4898369610coreneuron2m0E
13: host:0xae3bc0 device:0x2acaffe01600 size:8 presentcount:0+1 line:-1 name:_st__gpfs_bbp_cscs_ch_ssd_gitlab_map_jobs_bbpcihpcproj12_P10480_33968_spack_build_spack_stage_neuron_develop_qdta367h6ly225azfnup34o2tpbkhbcb_spack_build_qdta367_test_nrnivmodl_8e220c327f2b8882adcf04884baa4209f37d0bbcef5677f046766f546d969ffd_x86_64_corenrn_mod2c_hh_cpp__ZN49_INTERNAL_27_x86_64_corenrn_mod2c_hh_cpp_4898369610coreneuron2n0E
13: host:0xae3bf8 device:0x2acaffe01200 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron10_slist1_hhE
13: host:0xae3c00 device:0x2acaffe01300 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron10_dlist1_hhE
13: host:0xae5880 device:0x2acaffe02000 size:4 presentcount:0+1 line:-1 name:_ZN10coreneuron11secondorderE
13: host:0xae5898 device:0x2acaffe01f00 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron7celsiusE
13: host:0xae58a0 device:0x2acaffe02100 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron2piE
13: host:0x2818a40 device:0x2acaffb0a000 size:40 presentcount:0+1 line:-1 name:(null)
13: host:0x2818c40 device:0x2acaffb09200 size:72 presentcount:0+1 line:-1 name:(null)
13: host:0x2819080 device:0x2acaffb09c00 size:40 presentcount:0+1 line:-1 name:(null)
13: host:0x2819d40 device:0x2acaffb09800 size:72 presentcount:0+1 line:-1 name:(null)
13: host:0x281a400 device:0x2acaffb09a00 size:204 presentcount:0+1 line:-1 name:(null)
13: host:0x281a500 device:0x2acaffb06200 size:128 presentcount:0+1 line:-1 name:(null)
13: host:0x281a760 device:0x2acaffb0a200 size:16 presentcount:0+1 line:-1 name:(null)
13: host:0x281d600 device:0x2acaffb06400 size:40 presentcount:0+1 line:-1 name:(null)
13: host:0x281ebc0 device:0x2acaffafae00 size:276 presentcount:0+1 line:-1 name:(null)
13: host:0x28250c0 device:0x2acaffb05400 size:72 presentcount:0+1 line:-1 name:(null)
13: host:0x2825400 device:0x2acaffb08200 size:40 presentcount:0+1 line:-1 name:(null)
13: host:0x2827900 device:0x2acaffb0c000 size:224 presentcount:0+1 line:-1 name:(null)
13: host:0x2827a80 device:0x2acaffb0ae00 size:204 presentcount:0+1 line:-1 name:(null)
13: host:0x2827b80 device:0x2acaffb06800 size:204 presentcount:0+1 line:-1 name:(null)
13: host:0x2829c80 device:0x2acaffb04c00 size:232 presentcount:0+1 line:-1 name:(null)
13: host:0x282e1c0 device:0x2acaffb06600 size:72 presentcount:0+1 line:-1 name:(null)
13: host:0x320d5c0 device:0x2acaffafa400 size:4 presentcount:0+1 line:606 name:_ZN10coreneuron14_slist1_ExpSynE
13: host:0x320d5e0 device:0x2acaffafa600 size:4 presentcount:0+1 line:606 name:_ZN10coreneuron14_dlist1_ExpSynE
13: host:0x320e040 device:0x2acaffafa800 size:8 presentcount:0+1 line:639 name:_ZN10coreneuron15_slist1_Exp2SynE
13: host:0x320e060 device:0x2acaffafaa00 size:8 presentcount:0+1 line:639 name:_ZN10coreneuron15_dlist1_Exp2SynE
13: host:0x3227f90 device:0x2acaffb0cc00 size:136 presentcount:0+1 line:-1 name:(null)
13: host:0x3228020 device:0x2acaffafac00 size:24 presentcount:0+1 line:-1 name:(null)
13: host:0x325e1e0 device:0x2acaffafb000 size:24 presentcount:0+1 line:-1 name:(null)
13: host:0x32abbd0 device:0x2acaffafa000 size:12 presentcount:0+1 line:618 name:_ZN10coreneuron10_slist1_hhE
13: host:0x32abbf0 device:0x2acaffafa200 size:12 presentcount:0+1 line:618 name:_ZN10coreneuron10_dlist1_hhE
13: host:0x32f00c0 device:0x2acaffafb600 size:24640 presentcount:0+1 line:-1 name:(null)
13: host:0x32f6600 device:0x2acaffb09600 size:192 presentcount:0+1 line:-1 name:(null)
13: host:0x32f6740 device:0x2acaffb0be00 size:204 presentcount:0+1 line:-1 name:(null)
13: host:0x32f6ac0 device:0x2acaffb0b400 size:72 presentcount:0+1 line:-1 name:(null)
13: host:0x32fc200 device:0x2acaffb04e00 size:40 presentcount:0+1 line:-1 name:(null)
13: host:0x32fe080 device:0x2acaffb09e00 size:36 presentcount:0+1 line:-1 name:(null)
13: host:0x32fee40 device:0x2acaffb05200 size:40 presentcount:0+1 line:-1 name:(null)
13: host:0x3302340 device:0x2acaffb09400 size:16 presentcount:0+1 line:-1 name:(null)
13: host:0x343c8e8 device:0x2acaffafb200 size:912 presentcount:0+1 line:-1 name:(null)
13: host:0x343cc80 device:0x2acaffb0ac00 size:72 presentcount:0+1 line:-1 name:(null)
13: host:0x3458280 device:0x2acaffb05000 size:224 presentcount:0+1 line:-1 name:(null)
13: host:0x34583c0 device:0x2acaffb0b600 size:1344 presentcount:0+1 line:-1 name:(null)
13: host:0x3459100 device:0x2acaffb05600 size:64 presentcount:0+1 line:-1 name:(null)
13: host:0x34591c0 device:0x2acaffb07e00 size:40 presentcount:0+1 line:-1 name:(null)
13: host:0x3459280 device:0x2acaffb08000 size:36 presentcount:0+1 line:-1 name:(null)
13: host:0x3459300 device:0x2acaffb05a00 size:36 presentcount:0+1 line:-1 name:(null)
13: host:0x34593c0 device:0x2acaffb07a00 size:72 presentcount:0+1 line:-1 name:(null)
13: host:0x3459480 device:0x2acaffb07c00 size:72 presentcount:0+1 line:-1 name:(null)
13: host:0x345a4c0 device:0x2acaffb05800 size:36 presentcount:0+1 line:-1 name:(null)
13: allocated block device:0x2acaffafa000 size:512 thread:1
13: allocated block device:0x2acaffafa200 size:512 thread:1
13: allocated block device:0x2acaffafa400 size:512 thread:1
13: allocated block device:0x2acaffafa600 size:512 thread:1
13: allocated block device:0x2acaffafa800 size:512 thread:1
13: allocated block device:0x2acaffafaa00 size:512 thread:1
13: allocated block device:0x2acaffafac00 size:512 thread:1
13: allocated block device:0x2acaffafae00 size:512 thread:1
13: allocated block device:0x2acaffafb000 size:512 thread:1
13: allocated block device:0x2acaffafb200 size:1024 thread:1
13: allocated block device:0x2acaffafb600 size:38400 thread:1
13: allocated block device:0x2acaffb04c00 size:512 thread:1
13: allocated block device:0x2acaffb04e00 size:512 thread:1
13: allocated block device:0x2acaffb05000 size:512 thread:1
13: allocated block device:0x2acaffb05200 size:512 thread:1
13: allocated block device:0x2acaffb05400 size:512 thread:1
13: allocated block device:0x2acaffb05600 size:512 thread:1
13: allocated block device:0x2acaffb05800 size:512 thread:1
13: allocated block device:0x2acaffb05a00 size:512 thread:1
13: allocated block device:0x2acaffb06200 size:512 thread:1
13: allocated block device:0x2acaffb06400 size:512 thread:1
13: allocated block device:0x2acaffb06600 size:512 thread:1
13: allocated block device:0x2acaffb06800 size:512 thread:1
13: allocated block device:0x2acaffb07a00 size:512 thread:1
13: allocated block device:0x2acaffb07c00 size:512 thread:1
13: allocated block device:0x2acaffb07e00 size:512 thread:1
13: allocated block device:0x2acaffb08000 size:512 thread:1
13: allocated block device:0x2acaffb08200 size:512 thread:1
13: allocated block device:0x2acaffb09200 size:512 thread:1
13: allocated block device:0x2acaffb09400 size:512 thread:1
13: allocated block device:0x2acaffb09600 size:512 thread:1
13: allocated block device:0x2acaffb09800 size:512 thread:1
13: allocated block device:0x2acaffb09a00 size:512 thread:1
13: allocated block device:0x2acaffb09c00 size:512 thread:1
13: allocated block device:0x2acaffb09e00 size:512 thread:1
13: allocated block device:0x2acaffb0a000 size:512 thread:1
13: allocated block device:0x2acaffb0a200 size:512 thread:1
13: allocated block device:0x2acaffb0ac00 size:512 thread:1
13: allocated block device:0x2acaffb0ae00 size:512 thread:1
13: allocated block device:0x2acaffb0b400 size:512 thread:1
13: allocated block device:0x2acaffb0b600 size:2048 thread:1
13: allocated block device:0x2acaffb0be00 size:512 thread:1
13: allocated block device:0x2acaffb0c000 size:512 thread:1
13: allocated block device:0x2acaffb0cc00 size:512 thread:1
13: deleted block   device:0x2acaffb06a00 size:512 threadid=1 
13: deleted block   device:0x2acaffb05c00 size:512 threadid=1 
13: deleted block   device:0x2acaffb05e00 size:512 threadid=1 
13: deleted block   device:0x2acaffb06000 size:512 threadid=1 
13: deleted block   device:0x2acaffb07400 size:512 threadid=1 
13: deleted block   device:0x2acaffb07600 size:512 threadid=1 
13: deleted block   device:0x2acaffb07800 size:512 threadid=1 
13: deleted block   device:0x2acaffb06e00 size:512 threadid=1 
13: deleted block   device:0x2acaffb07000 size:512 threadid=1 
13: deleted block   device:0x2acaffb07200 size:512 threadid=1 
13: deleted block   device:0x2acaffb08400 size:512 threadid=1 
13: deleted block   device:0x2acaffb08600 size:512 threadid=1 
13: deleted block   device:0x2acaffb06c00 size:512 threadid=1 
13: deleted block   device:0x2acaffb0c200 size:512 threadid=1 
13: deleted block   device:0x2acaffb0c400 size:512 threadid=1 
13: deleted block   device:0x2acaffb0c800 size:512 threadid=1 
13: deleted block   device:0x2acaffb0c600 size:512 threadid=1 
13: deleted block   device:0x2acaffb0ca00 size:512 threadid=1 
13: deleted block   device:0x2acaffb0a400 size:1024 threadid=1 
13: deleted block   device:0x2acaffb0a800 size:1024 threadid=1 
13: deleted block   device:0x2acaffb0b000 size:1024 threadid=1 
13: deleted block   device:0x2acaffb08800 size:2560 threadid=1 
13: FATAL ERROR: variable in data clause is partially present on the device: name=(unknown)
13:  file:(OpenACC API) acc_copyin line:-1

In https://bbpgitlab.epfl.ch/hpc/coreneuron/-/jobs/35914 testing #590 the datareturn_py test failed again:

13: (null) lives at 0x281a740 size 36 partially present
13: Present table dump for device[1]: NVIDIA Tesla GPU 0, compute capability 7.0, threadid=1
13: host:0xa07870 device:0x2acaffe01700 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron13mol_UnitsTestE
13: host:0xa07878 device:0x2acaffe01800 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron11E_UnitsTestE
13: host:0xa07880 device:0x2acaffe01900 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron17FARADAY_UnitsTestE
13: host:0xa07888 device:0x2acaffe01a00 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron11h_UnitsTestE
13: host:0xa07890 device:0x2acaffe01b00 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron12hb_UnitsTestE
13: host:0xa07898 device:0x2acaffe01c00 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron11R_UnitsTestE
13: host:0xa078a0 device:0x2acaffe01d00 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron19boltzmann_UnitsTestE
13: host:0xa078a8 device:0x2acaffe01e00 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron13avo_UnitsTestE
13: host:0xae3a88 device:0x2acaffe00b00 size:8 presentcount:0+1 line:-1 name:_st__gpfs_bbp_cscs_ch_ssd_gitlab_map_jobs_bbpcihpcproj12_P11087_35910_spack_build_spack_stage_neuron_develop_qdta367h6ly225azfnup34o2tpbkhbcb_spack_build_qdta367_test_nrnivmodl_8e220c327f2b8882adcf04884baa4209f37d0bbcef5677f046766f546d969ffd_x86_64_corenrn_mod2c_exp2syn_cpp__ZN54_INTERNAL_32_x86_64_corenrn_mod2c_exp2syn_cpp_25c7f1d810coreneuron2A0E
13: host:0xae3a90 device:0x2acaffe00c00 size:8 presentcount:0+1 line:-1 name:_st__gpfs_bbp_cscs_ch_ssd_gitlab_map_jobs_bbpcihpcproj12_P11087_35910_spack_build_spack_stage_neuron_develop_qdta367h6ly225azfnup34o2tpbkhbcb_spack_build_qdta367_test_nrnivmodl_8e220c327f2b8882adcf04884baa4209f37d0bbcef5677f046766f546d969ffd_x86_64_corenrn_mod2c_exp2syn_cpp__ZN54_INTERNAL_32_x86_64_corenrn_mod2c_exp2syn_cpp_25c7f1d810coreneuron2B0E
13: host:0xae3ac8 device:0x2acaffe00900 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron15_slist1_Exp2SynE
13: host:0xae3ad0 device:0x2acaffe00a00 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron15_dlist1_Exp2SynE
13: host:0xae3b08 device:0x2acaffe00f00 size:8 presentcount:0+1 line:-1 name:_st__gpfs_bbp_cscs_ch_ssd_gitlab_map_jobs_bbpcihpcproj12_P11087_35910_spack_build_spack_stage_neuron_develop_qdta367h6ly225azfnup34o2tpbkhbcb_spack_build_qdta367_test_nrnivmodl_8e220c327f2b8882adcf04884baa4209f37d0bbcef5677f046766f546d969ffd_x86_64_corenrn_mod2c_expsyn_cpp__ZN53_INTERNAL_31_x86_64_corenrn_mod2c_expsyn_cpp_0127273610coreneuron2g0E
13: host:0xae3b38 device:0x2acaffe00d00 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron14_slist1_ExpSynE
13: host:0xae3b40 device:0x2acaffe00e00 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron14_dlist1_ExpSynE
13: host:0xae3b70 device:0x2acaffe01000 size:4 presentcount:0+1 line:-1 name:_ZN10coreneuron23_mechtype_ForNetConTestE
13: host:0xae3bc8 device:0x2acaffe01100 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron12_celsius__hhE
13: host:0xae3bd0 device:0x2acaffe01400 size:8 presentcount:0+1 line:-1 name:_st__gpfs_bbp_cscs_ch_ssd_gitlab_map_jobs_bbpcihpcproj12_P11087_35910_spack_build_spack_stage_neuron_develop_qdta367h6ly225azfnup34o2tpbkhbcb_spack_build_qdta367_test_nrnivmodl_8e220c327f2b8882adcf04884baa4209f37d0bbcef5677f046766f546d969ffd_x86_64_corenrn_mod2c_hh_cpp__ZN49_INTERNAL_27_x86_64_corenrn_mod2c_hh_cpp_4898369610coreneuron2h0E
13: host:0xae3bd8 device:0x2acaffe01500 size:8 presentcount:0+1 line:-1 name:_st__gpfs_bbp_cscs_ch_ssd_gitlab_map_jobs_bbpcihpcproj12_P11087_35910_spack_build_spack_stage_neuron_develop_qdta367h6ly225azfnup34o2tpbkhbcb_spack_build_qdta367_test_nrnivmodl_8e220c327f2b8882adcf04884baa4209f37d0bbcef5677f046766f546d969ffd_x86_64_corenrn_mod2c_hh_cpp__ZN49_INTERNAL_27_x86_64_corenrn_mod2c_hh_cpp_4898369610coreneuron2m0E
13: host:0xae3be0 device:0x2acaffe01600 size:8 presentcount:0+1 line:-1 name:_st__gpfs_bbp_cscs_ch_ssd_gitlab_map_jobs_bbpcihpcproj12_P11087_35910_spack_build_spack_stage_neuron_develop_qdta367h6ly225azfnup34o2tpbkhbcb_spack_build_qdta367_test_nrnivmodl_8e220c327f2b8882adcf04884baa4209f37d0bbcef5677f046766f546d969ffd_x86_64_corenrn_mod2c_hh_cpp__ZN49_INTERNAL_27_x86_64_corenrn_mod2c_hh_cpp_4898369610coreneuron2n0E
13: host:0xae3c18 device:0x2acaffe01200 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron10_slist1_hhE
13: host:0xae3c20 device:0x2acaffe01300 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron10_dlist1_hhE
13: host:0xae58a0 device:0x2acaffe02000 size:4 presentcount:0+1 line:-1 name:_ZN10coreneuron11secondorderE
13: host:0xae58b8 device:0x2acaffe01f00 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron7celsiusE
13: host:0xae58c0 device:0x2acaffe02100 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron2piE
13: host:0x2818c40 device:0x2acaffb09200 size:72 presentcount:0+1 line:-1 name:(null)
13: host:0x2819d80 device:0x2acaffb0a000 size:40 presentcount:0+1 line:-1 name:(null)
13: host:0x281a760 device:0x2acaffb0a200 size:16 presentcount:0+1 line:-1 name:(null)
13: host:0x281b100 device:0x2acaffb04e00 size:40 presentcount:0+1 line:-1 name:(null)
13: host:0x2825400 device:0x2acaffb08200 size:40 presentcount:0+1 line:-1 name:(null)
13: host:0x2827900 device:0x2acaffb09a00 size:204 presentcount:0+1 line:-1 name:(null)
13: host:0x2827a00 device:0x2acaffb09600 size:192 presentcount:0+1 line:-1 name:(null)
13: host:0x2827b00 device:0x2acaffb0be00 size:204 presentcount:0+1 line:-1 name:(null)
13: host:0x282e1c0 device:0x2acaffafae00 size:276 presentcount:0+1 line:-1 name:(null)
13: host:0x320d660 device:0x2acaffafa400 size:4 presentcount:0+1 line:606 name:_ZN10coreneuron14_slist1_ExpSynE
13: host:0x320d680 device:0x2acaffafa600 size:4 presentcount:0+1 line:606 name:_ZN10coreneuron14_dlist1_ExpSynE
13: host:0x320e0e0 device:0x2acaffafa800 size:8 presentcount:0+1 line:639 name:_ZN10coreneuron15_slist1_Exp2SynE
13: host:0x320e100 device:0x2acaffafaa00 size:8 presentcount:0+1 line:639 name:_ZN10coreneuron15_dlist1_Exp2SynE
13: host:0x3228030 device:0x2acaffb0cc00 size:136 presentcount:0+1 line:-1 name:(null)
13: host:0x32280c0 device:0x2acaffafac00 size:24 presentcount:0+1 line:-1 name:(null)
13: host:0x325e280 device:0x2acaffafb000 size:24 presentcount:0+1 line:-1 name:(null)
13: host:0x32abc70 device:0x2acaffafa000 size:12 presentcount:0+1 line:618 name:_ZN10coreneuron10_slist1_hhE
13: host:0x32abc90 device:0x2acaffafa200 size:12 presentcount:0+1 line:618 name:_ZN10coreneuron10_dlist1_hhE
13: host:0x32f0180 device:0x2acaffafb600 size:24640 presentcount:0+1 line:-1 name:(null)
13: host:0x32f6b80 device:0x2acaffb0b400 size:72 presentcount:0+1 line:-1 name:(null)
13: host:0x32f71c0 device:0x2acaffb05200 size:40 presentcount:0+1 line:-1 name:(null)
13: host:0x32f7f80 device:0x2acaffb09c00 size:40 presentcount:0+1 line:-1 name:(null)
13: host:0x32f98c0 device:0x2acaffb09800 size:72 presentcount:0+1 line:-1 name:(null)
13: host:0x32fcf40 device:0x2acaffb05400 size:72 presentcount:0+1 line:-1 name:(null)
13: host:0x3302400 device:0x2acaffb09400 size:16 presentcount:0+1 line:-1 name:(null)
13: host:0x343cd88 device:0x2acaffafb200 size:912 presentcount:0+1 line:-1 name:(null)
13: host:0x34407c0 device:0x2acaffb04c00 size:232 presentcount:0+1 line:-1 name:(null)
13: host:0x3440900 device:0x2acaffb0ac00 size:72 presentcount:0+1 line:-1 name:(null)
13: host:0x3458300 device:0x2acaffb0c000 size:224 presentcount:0+1 line:-1 name:(null)
13: host:0x3458440 device:0x2acaffb0ae00 size:204 presentcount:0+1 line:-1 name:(null)
13: host:0x3458540 device:0x2acaffb05000 size:224 presentcount:0+1 line:-1 name:(null)
13: allocated block device:0x2acaffafa000 size:512 thread:1
13: allocated block device:0x2acaffafa200 size:512 thread:1
13: allocated block device:0x2acaffafa400 size:512 thread:1
13: allocated block device:0x2acaffafa600 size:512 thread:1
13: allocated block device:0x2acaffafa800 size:512 thread:1
13: allocated block device:0x2acaffafaa00 size:512 thread:1
13: allocated block device:0x2acaffafac00 size:512 thread:1
13: allocated block device:0x2acaffafae00 size:512 thread:1
13: allocated block device:0x2acaffafb000 size:512 thread:1
13: allocated block device:0x2acaffafb200 size:1024 thread:1
13: allocated block device:0x2acaffafb600 size:38400 thread:1
13: allocated block device:0x2acaffb04c00 size:512 thread:1
13: allocated block device:0x2acaffb04e00 size:512 thread:1
13: allocated block device:0x2acaffb05000 size:512 thread:1
13: allocated block device:0x2acaffb05200 size:512 thread:1
13: allocated block device:0x2acaffb05400 size:512 thread:1
13: allocated block device:0x2acaffb08200 size:512 thread:1
13: allocated block device:0x2acaffb09200 size:512 thread:1
13: allocated block device:0x2acaffb09400 size:512 thread:1
13: allocated block device:0x2acaffb09600 size:512 thread:1
13: allocated block device:0x2acaffb09800 size:512 thread:1
13: allocated block device:0x2acaffb09a00 size:512 thread:1
13: allocated block device:0x2acaffb09c00 size:512 thread:1
13: allocated block device:0x2acaffb0a000 size:512 thread:1
13: allocated block device:0x2acaffb0a200 size:512 thread:1
13: allocated block device:0x2acaffb0ac00 size:512 thread:1
13: allocated block device:0x2acaffb0ae00 size:512 thread:1
13: allocated block device:0x2acaffb0b400 size:512 thread:1
13: allocated block device:0x2acaffb0be00 size:512 thread:1
13: allocated block device:0x2acaffb0c000 size:512 thread:1
13: allocated block device:0x2acaffb0cc00 size:512 thread:1
13: deleted block   device:0x2acaffb09e00 size:512 threadid=1
13: deleted block   device:0x2acaffb06200 size:512 threadid=1
13: deleted block   device:0x2acaffb05600 size:512 threadid=1
13: deleted block   device:0x2acaffb05800 size:512 threadid=1
13: deleted block   device:0x2acaffb05a00 size:512 threadid=1
13: deleted block   device:0x2acaffb07a00 size:512 threadid=1
13: deleted block   device:0x2acaffb07c00 size:512 threadid=1
13: deleted block   device:0x2acaffb07e00 size:512 threadid=1
13: deleted block   device:0x2acaffb08000 size:512 threadid=1
13: deleted block   device:0x2acaffb06400 size:512 threadid=1
13: deleted block   device:0x2acaffb06600 size:512 threadid=1
13: deleted block   device:0x2acaffb06800 size:512 threadid=1
13: deleted block   device:0x2acaffb06a00 size:512 threadid=1
13: deleted block   device:0x2acaffb05c00 size:512 threadid=1
13: deleted block   device:0x2acaffb05e00 size:512 threadid=1
13: deleted block   device:0x2acaffb06000 size:512 threadid=1
13: deleted block   device:0x2acaffb07400 size:512 threadid=1
13: deleted block   device:0x2acaffb07600 size:512 threadid=1
13: deleted block   device:0x2acaffb07800 size:512 threadid=1
13: deleted block   device:0x2acaffb06e00 size:512 threadid=1
13: deleted block   device:0x2acaffb07000 size:512 threadid=1
13: deleted block   device:0x2acaffb07200 size:512 threadid=1
13: deleted block   device:0x2acaffb08400 size:512 threadid=1
13: deleted block   device:0x2acaffb08600 size:512 threadid=1
13: deleted block   device:0x2acaffb06c00 size:512 threadid=1
13: deleted block   device:0x2acaffb0c200 size:512 threadid=1
13: deleted block   device:0x2acaffb0c400 size:512 threadid=1
13: deleted block   device:0x2acaffb0c800 size:512 threadid=1
13: deleted block   device:0x2acaffb0c600 size:512 threadid=1
13: deleted block   device:0x2acaffb0ca00 size:512 threadid=1
13: deleted block   device:0x2acaffb0a400 size:1024 threadid=1
13: deleted block   device:0x2acaffb0a800 size:1024 threadid=1
13: deleted block   device:0x2acaffb0b000 size:1024 threadid=1
13: deleted block   device:0x2acaffb0b600 size:2048 threadid=1
13: deleted block   device:0x2acaffb08800 size:2560 threadid=1
13: FATAL ERROR: variable in data clause is partially present on the device: name=(unknown)
13:  file:(OpenACC API) acc_copyin line:-1

Navigation Menu

Repository navigation

coreneuron_modtests::{direct_hoc,datareturn_py} tests sometimes fail on GPU #586

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

coreneuron_modtests::{direct_hoc,datareturn_py} tests sometimes fail on GPU #586

Description

Activity

olupton commented on Jul 6, 2021

olupton commented on Jul 19, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions