[pramodk]

* Analysis of the bug:
  - Blue Brain model (channel-benchmark) was showing diferent spikes
    after few timesteps. Spikes were shifted and initially appeared
    to be floating point discrepencies.
  - Closely looking into NET_RECEIVE method revealed that the mechanism
    data accessed in net_receieve function was wrong. Somehow accessing
    variables like NMDA_ratio was showing wrong values. The nrn_state()
    had correct values.
  - The net_receieve function was accessing ml->data inside OpenACC loop.
    Accessing ml->data was providing different pointer inside OpenACC loop.
* Fix for the bug:
  - some mechanisms can have dparam semantics for diam data and it is
    added into nt->data (after matrix elements).
  - When calculating offert for mechanism data, this extra semantics
    data was not considered. Adjusting the offset fixes the issue.
  - Additionally, we do not need to rely on offset but we can use
    acc_deviceptr to find corresponding device pointer directly.
    This is more reliable than manually calculating offset.

See initial fix in a7a95c9 and then simplification in 0ac3ab9.

How to Run?

Run channel benchmark on CPU vs GPU and compare spikes

export SIM_TIME=100
export NUM_CELLS=100
export PRCELL_GID=1

srun -n 1 dplace ./cpu/x86_64/special -mpi -c arg_tstop=$SIM_TIME -c arg_dump_coreneuron_model=1 -c arg_target_count=$NUM_CELLS -c arg_prcell_gid=$PRCELL_GID $HOC_LIBRARY_PATH/init.hoc 2>&1 | tee neuron.log
srun -n 1 dplace ./cpu/x86_64/special-core -e $SIM_TIME --mpi -d coredat --prcellgid $PRCELL_GID 2>&1 | tee coreneuron.log
mv out.dat out.dat.cpu
srun -n 1 dplace ./gpu/x86_64/special-core -e $SIM_TIME --mpi -d coredat --prcellgid $PRCELL_GID --gpu --cell-permute=1 2>&1 | tee coreneuron.gpu.log
mv out.dat out.dat.gpu
diff out.dat out.dat.cpu out.dat out.dat.gpu

Test System
BB5 + GPU

Use certain branches for the SimulationStack CI

CI_BRANCHES:NEURON_BRANCH=master,

[bbpbuildbot]

Logfiles from GitLab pipeline #11087 (:no_entry:) have been uploaded here!

Status and direct links:

• ✅ spack_setup
• ✅ build:nmodl:intel
• ✅ build:nmodl:gpu
• ✅ build:coreneuron+nmodl:intel
• ✅ build:coreneuron:intel
• ✅ build:coreneuron+nmodl:gpu
• ✅ build:coreneuron:gpu
• ✅ test:coreneuron+nmodl:intel
• ✅ test:coreneuron:intel
• ✅ test:coreneuron+nmodl:gpu
• ✅ test:coreneuron:gpu
• ✅ build:neuron+nmodl:intel
• ✅ build:neuron:intel
• ✅ build:neuron+nmodl:gpu
• ✅ build:neuron:gpu
• ✅ test:neuron+nmodl:intel
• ✅ test:neuron:intel
• ✅ test:neuron+nmodl:gpu
• ⛔ test:neuron:gpu

[olupton]

LGTM. The CI failure looks like #586 to me, so I think we can ignore it for now.

(I won't be sad when this nrn_acc_manager code is replaced!)

[iomaganaris]

LGTM
Nasty bug and good catch!
+1 to Olli's comment