Skip to content

[BUG] multiple-pipeline test fails with DSP EXCEPTION with multicore nocodec topology #8492

Closed

Description

@fredoh9

Describe the bug
multiple-pipeline test failed with DSP Exception. dmesg has ipc error with MOD_SET_DX.
ADLP_RVP_NOCODEC and TGLU_RVP_NOCODEC has same/similar issue.

[  616.335232] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx      : 0x47000000|0x0: MOD_SET_DX [data size: 8]
[  616.335728] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx reply: 0x67000006|0x0: MOD_SET_DX
[  616.335730] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: FW reported error: 6 - Unknown error while processing the request
[  616.335869] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc error for msg 0x47000000|0x0
[  616.335875] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error: failed to disable target core: 2 for widget pipeline.6
[  616.335878] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: Failed to free connected widgets
[  616.335882] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ASoC: error at snd_soc_pcm_component_hw_free on 0000:00:1f.3: -22

mtrace shows FATAL EXCEPTION

[   51.966658] <inf> ipc: ipc_cmd: rx	: 0x47000000|0x0
[   51.966756] <err> ipc: ipc4_module_process_dx: failed to disable core 2
[   51.966766] <err> ipc: ipc_cmd: ipc4: MODULE_MSG failed with err 6
ASSERTION FAIL [k_sched_current_thread_query()->base.sched_locked != 0U] @ /srv/home/jenkins/workspace/sof_config_build@2/zephyr/kernel/sched.c:1117
	
[   51.967073] <err> os: xtensa_excint1_c:  ** FATAL EXCEPTION
ASSERTION FAIL [z_spin_lock_valid(l)] @ /srv/home/jenkins/workspace/sof_config_build@2/zephyr/include/zephyr/spinlock.h:136
[   51.967153] <err> os: xtensa_excint1_c:  **    (INTLEVEL:0 EXCM: 0 UM:1 RING:0 WOE:1 OWB:4 CALLINC:2)
[   51.967373] <err> os: xtensa_excint1_c:  **    (INTLEVEL:0 EXCM: 0 UM:1 RING:0 WOE:1 OWB:10 CALLINC:2)
[   51.967585] <err> os: xtensa_excint1_c:  **    (INTLEVEL:0 EXCM: 0 UM:1 RING:0 WOE:1 OWB:0 CALLINC:2)
[   51.967828] <err> os: xtensa_excint1_c:  **    (INTLEVEL:0 EXCM: 0 UM:1 RING:0 WOE:1 OWB:6 CALLINC:2)
[   51.967985] <err> os: xtensa_excint1_c:  **    (INTLEVEL:0 EXCM: 0 UM:1 RING:0 WOE:1 OWB:12 CALLINC:2)
[   51.968188] <err> os: xtensa_excint1_c:  **    (INTLEVEL:0 EXCM: 0 UM:1 RING:0 WOE:1 OWB:2 CALLINC:2)
[   51.968425] <err> os: xtensa_excint1_c:  **    (INTLEVEL:0 EXCM: 0 UM:1 RING:0 WOE:1 OWB:8 CALLINC:2)
	Invalid spinlock 0x9e0b8518
	Invalid spinlock 0x9e0b8518
	Invalid spinlock 0x9e0b8518
	Invalid spinlock 0x9e0b8518
ASSERTION FAIL [z_spin_lock_valid(l)] @ /srv/home/jenkins/workspace/sof_config_build@2/zephyr/include/zephyr/spinlock.h:136
ASSERTION FAIL [z_spin_lock_valid(l)] @ /srv/home/jenkins/workspace/sof_config_build@2/zephyr/include/zephyr/spinlock.h:136
ASSERTION FAIL [z_spin_lock_valid(l)] @ /srv/home/jenkins/workspace/sof_config_build@2/zephyr/include/zephyr/spinlock.h:136

To Reproduce
TPLG=/lib/firmware/intel/development/sof-adl-nocodec.tplg MODEL=ADLP_RVP_NOCODEC-ipc4 SOF_TEST_INTERVAL=5 ~/sof-test/test-case/multiple-pipeline.sh -f a -c 20 -l 50

Reproduction Rate
Need some iteration but 100%

Environment
First broken daily build,

  • SOF Build Result: 1379
  • SOF Branch: main
  • SOF Commit: 9eae757
  • Zephyr Commit: 26002b060708

The TPLG has multicore support.
sof-adl-nocodec-multicore

Last good daily build,

  • SOF Build Result: 1366
  • SOF Branch: main
  • SOF Commit: b7d7fb0
  • Zephyr Commit: 26002b060708

The TPLG doesn't have multicore support.
sof-adl-nocodec

NOTE: If I just replace the tplg from good build, this is not reproducible.
This PR #8240 makes multicore part of default nocodec topology and causes the problem.

Screenshots or console output

dmesg-adl-fw-error6.txt
mtrace-adl-dsp-exception.txt

Activity

  1. added
    bugSomething isn't working as expected
    P1Blocker bugs or important features
    I2SApplies to I2S bus for codec connection
    TGLApplies to Tiger Lake
    multicoreIssues observed when not only core#0 is used.
    ADLApplies to Alder Lake platform
    multicore-3coresIssues observed when 3 cores are used.
    on Nov 17, 2023
  2. marc-hb commented on Nov 18, 2023

    @marc-hb
    Collaborator

    Another multiple-pipeline failure, dunno whether it's related but it's a very recent regression too: https://sof-ci.01.org/sofpr/PR8494/build14998/devicetest/index.html?model=ADLP_RVP_NOCODEC_IPC4ZPH&testcase=multiple-pipeline-all (#8494)

  3. fredoh9 commented on Nov 18, 2023

    @fredoh9
    ContributorAuthor

    multiple-pipeline-playback-50 test has similar but a bit different logs.

    dmesg

    [ 4227.456412] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx      : 0x47000000|0x0: MOD_SET_DX [data size: 8]
    [ 4227.456934] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx reply: 0x67000006|0x0: MOD_SET_DX
    [ 4227.456938] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: FW reported error: 6 - Unknown error while processing the request
    [ 4227.457151] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc error for msg 0x47000000|0x0
    [ 4227.457156] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error: failed to disable target core: 2 for widget pipeline.6
    [ 4227.457158] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: Failed to free connected widgets
    [ 4227.457163] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ASoC: error at snd_soc_pcm_component_hw_free on 0000:00:1f.3: -22
    [ 4227.457187] kernel: snd_sof:sof_pcm_hw_free: sof-audio-pci-intel-tgl 0000:00:1f.3: pcm: free stream 1 dir 0
    [ 4227.457195] kernel: snd_sof_intel_hda_common:hda_dsp_stream_trigger: sof-audio-pci-intel-tgl 0000:00:1f.3: FW Poll Status: reg[0x1c0]=0x20440000 successful
    [ 4227.457204] kernel: snd_sof:sof_ipc4_trigger_pipelines: sof-audio-pci-intel-tgl 0000:00:1f.3: trigger cmd: 0 state: 2
    [ 4227.457208] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx      : 0x13000002|0x1: GLB_SET_PIPELINE_STATE [data size: 12]
    [ 4227.958422] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc timed out for 0x13000002|0x1
    

    mtrace has ASSERTION FAIL

    [   62.122463] <wrn> copier: copier_comp_trigger: comp:2 0x20004 dai is not ready
    [   62.122585] <inf> src: src_reset: comp:2 0x8 src_reset()
    [   62.123711] <inf> ipc: ipc_cmd: rx	: 0x46010002|0x10003
    [   62.124348] <inf> ipc: ipc_cmd: rx	: 0x12020000|0x0
    [   62.124416] <inf> dma: dma_put: dma_put(), dma = 0x9e09f250, sref = 1
    [   62.124636] <inf> src: src_free: comp:2 0x8 src_free()
    [   62.125286] <inf> ipc: ipc_cmd: rx	: 0x12030000|0x0
    [   62.125596] <inf> dma: dma_put: dma_put(), dma = 0x9e09f1b0, sref = 0
    [   62.125613] <inf> dai_intel_ssp: dai_ssp_pm_runtime_dis_ssp_power: dai_ssp_pm_runtime_dis_ssp_power index 2
    [   62.125755] <err> dai_intel_ssp: dai_ssp_po����������������Çÿÿÿ����	:�¾8—�¾”Î�¾Ø—�¾Ø—�¾¸—�¾����ay poll Ø—�¾¸—�¾��������������������”Î�¾��������è–�¾%U�¾x—�¾”Î�¾Ø—�¾”Î�¾Ø—�¾¸—�¾����Ø—�¾¸—�¾����p_pm¸—�¾����ÇÿÿÿP%
    žX�¾¨—�¾”Î�¾����pm_runtime_dis_ssp_power warning: timeout
    [   62.125778] <inf> dai_intel_ssp: dai_ssp_pm_runtime_dis_ssp_power: dai_ssp_pm_runtime_dis_ssp_power I2SLCTL
    [   62.126203] <inf> ipc: ipc_cmd: rx	: 0x47000000|0x0
    [   62.126355] <err> ipc: ipc4_module_process_dx: failed to disable core 2
    [   62.126370] <err> ipc: ipc_cmd: ipc4: MODULE_MSG failed with err 6
    ASSERTION FAIL [k_sched_current_thread_query()->base.sched_locked != 0U] @ /srv/home/jenkins/workspace/sof_config_build@2/zephyr/kernel/sched.c:1117
    [   62.126713] <err> os: z_xtensa_dump_stack:  **  A0 0xbe0154f8  SP 0xbe0b20f0  A2 0x4  A3 0xbe0b2100
    [   62.126963] <err> os: z_xtensa_dump_stack:  **  A0 0xbe0154f8  SP 0xbe0b3ba0  A2 0x4  A3 0xbe0b3bb0
    [   62.127201] <err> os: z_xtensa_dump_stack:  **  A0 0xbe0154f8  SP 0xbe0b3938  A2 0x4  A3 0xbe0b3948
    [   62.127411] <err> os: z_xtensa_dump_stack:  **  A0 0xbe0154f8  SP 0xbe0b36d0  A2 0x4  A3 0xbe0b36e0
    [   62.127623] <err> os: z_xtensa_dump_stack:  **  A0 0xbe0154f8  SP 0xbe0b3468  A2 0x4  A3 0xbe0b3478
    [   62.127833] <err> os: z_xtensa_dump_stack:  **  A0 0xbe0154f8  SP 0xbe0b3200  A2 0x4  A3 0xbe0b3210
    [   62.128086] <err> os: z_xtensa_dump_stack:  **  A0 0xbe0154f8  SP 0xbe0b2f98  A2 0x4  A3 0xbe0b2fa8
    	Invalid spinlock 0x9e0b8518
    	Invalid spinlock 0x9e0b8518
    	Invalid spinlock 0x9e0b8518
    	Invalid spinlock 0x9e0b8518
    

    Intel Internal Test link:
    planresultdetail/34750?model=ADLP_RVP_NOCODEC-ipc4&testcase=multiple-pipeline-playback-50

  4. added this to the v2.9 milestone on Nov 23, 2023
  5. fredoh9 commented on Nov 27, 2023

    @fredoh9
    ContributorAuthor

    working fine with the Zephyr fix and west update. Closing now

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    ADLApplies to Alder Lake platformI2SApplies to I2S bus for codec connectionP1Blocker bugs or important featuresTGLApplies to Tiger LakebugSomething isn't working as expectedmulticoreIssues observed when not only core#0 is used.multicore-3coresIssues observed when 3 cores are used.

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions