Repository navigation
Invalid DMA trace stream tag causing FW oops #443
Description
Summary:
DSP panic happened after system booted.
[ 8.142381] sof-audio sof-audio: error : DSP panic!
[ 8.142451] sof-audio sof-audio: panic: dsp_oops_offset 1329152 offset 1329152
[ 8.142484] sof-audio sof-audio: error: runtime exception
[ 8.142544] sof-audio sof-audio: error: trace point 90010000
[ 8.142605] sof-audio sof-audio: error: panic happen at :0
[ 8.142665] sof-audio sof-audio: error: DSP Firmware Oops
[ 8.142724] sof-audio sof-audio: error: Exception Cause: IllegalInstructionCause, Illegal instruction
[ 8.142821] sof-audio sof-audio: EXCCAUSE 0x00000000 EXCVADDR 0x00000000 PS 0x00070520 SAR 0x0000001d
[ 8.142926] sof-audio sof-audio: EPC1 0xff2c099b EPC2 0x00000000 EPC3 0xff2c0edf EPC4 0xff2c0edf
[ 8.143030] sof-audio sof-audio: EPC5 0x00000000 EPC6 0x00000000 EPC7 0x00000000 DEPC 0x00000000
[ 8.143134] sof-audio sof-audio: EPS2 0x00000000 EPS3 0x00060720 EPS4 0x00060720 EPS5 0x00000000
[ 8.143238] sof-audio sof-audio: EPS6 0x00000000 EPS7 0x00000000 INTENABL 0x00078f40 INTERRU 0x000000a2
[ 8.143342] sof-audio sof-audio: stack dump from 0xff327bc0
[ 8.143403] sof-audio sof-audio: 0xff327bc0: 0x02480002 0x00060130 0x0000001d 0xff2c0f69
[ 8.143488] sof-audio sof-audio: 0xff327bc4: 0x00000000 0xff309180 0xff306a8c 0x02480000
[ 8.143574] sof-audio sof-audio: 0xff327bc8: 0x00000000 0x00000000 0xff2d0832 0xff2d0848
[ 8.143660] sof-audio sof-audio: 0xff327bcc: 0x00000000 0x00000000 0x00000000 0x00000000
[ 8.143745] sof-audio sof-audio: 0xff327bd0: 0xff319200 0xff319284 0xff327d40 0xff319284
[ 8.143831] sof-audio sof-audio: 0xff327bd4: 0xbf2c1f91 0xff327c60 0xff327c80 0x00000010
[ 8.143916] sof-audio sof-audio: 0xff327bd8: 0xbf2c408d 0xff327c70 0x00000002 0x00000001
[ 8.144002] sof-audio sof-audio: 0xff327bdc: 0xbf2cb89c 0xff327c80 0x20003e48 0x00000318
[ 8.417462] intel_soc_dts_thermal: request_threaded_irq ret -22
Test env:
sof master: a97cd4f
sof tool: 918e401
kernel sof-dev: 1f6c9cf
tplg: test-ssp2-mclk-0-I2S-volume-s16le-s16le-48k-19200k-codec.tplg
Log:
dmesg-panic.log
Activity
ZhendanYang commentedon Oct 8, 2018
Summary:
DSP panic happened after system booted.
[ 5.736743] sof-audio sof-audio: error : DSP panic!
[ 5.736752] sof-audio sof-audio: panic: dsp_oops_offset 788480 offset 788480
[ 5.736756] sof-audio sof-audio: status: fw entered - code 00000005
[ 5.736778] sof-audio sof-audio: error: runtime exception
[ 5.736780] sof-audio sof-audio: error: trace point 00004000
[ 5.736782] sof-audio sof-audio: error: panic happen at :0
[ 5.736785] sof-audio sof-audio: error: DSP Firmware Oops
[ 5.736788] sof-audio sof-audio: error: Exception Cause: LoadProhibitedCause, A load referenced a page mapped with an attribute that does not permit loads
[ 5.736791] sof-audio sof-audio: EXCCAUSE 0x0000001c EXCVADDR 0xfffffff8 PS 0x00060020 SAR 0x0000001d
[ 5.736794] sof-audio sof-audio: EPC1 0xbe015a93 EPC2 0xbe0118f6 EPC3 0xbe013a48 EPC4 0x00000000
[ 5.736796] sof-audio sof-audio: EPC5 0x00000000 EPC6 0x00000000 EPC7 0x00000000 DEPC 0x00000000
[ 5.736799] sof-audio sof-audio: EPS2 0x00060720 EPS3 0x00060722 EPS4 0x00000000 EPS5 0x00000000
[ 5.736802] sof-audio sof-audio: EPS6 0x00000000 EPS7 0x00000000 INTENABL 0x000101d0 INTERRU 0x00000222
[ 5.736804] sof-audio sof-audio: stack dump from 0xbe07fd20
be015a93: 7c1852 l16ui a5, a8, 248
case COMP_TYPE_COMPONENT:
if (icd->cd->comp.id == id)
Test env:
sof-master: aeb18ee
sof tool: 2d7a933
kernel sof-dev: 93853cb
tplg: sof-apl-nocodec.tplg
Log:
rmbox and rmbox -t are both empty.
dmesg-apl.log
lgirdwood commentedon Oct 8, 2018
@ZhendanYang adding a little more context shows it's related to trace init on the first one.
lgirdwood commentedon Oct 8, 2018
and on APL it is a topology loading issue
So this issue is really 2 issues, one for trace init on BYT and the other for topology on APL.
ZhendanYang commentedon Oct 9, 2018
Add BYT DSP panic info:
942 static inline uint32_t arch_interrupt_global_disable(void)
943 {
944 uint32_t flags;
945
946 asm volatile("rsil %0, 5"
947 ff2c099b: 006590 rsil a9, 5
948 {
949 struct sof_ipc_dsp_oops_xtensa *x =
950 (struct sof_ipc_dsp_oops_xtensa ) mailbox_get_exception_base();
951
952 / Exception Vector number - 0x0 */
953 __ asm__ __ volatile__ ("rsr %0, EXCCAUSE" : "=a" (x->exccause) : : "memory");
954 ff2c099e: 03e8a0 rsr.exccause a10
955 ff2c09a1: fff681 l32r a8, ff2c097c <_irq_task+0x8c>
956 ff2c09a4: 0068a2 s32i a10, a8, 0
ZhendanYang commentedon Oct 15, 2018
Another Exception Cause on UP2:
[ 6.132460] sof-audio sof-audio: ipc tx: 0x80010000
[ 6.132735] sof-audio sof-audio: error : DSP panic!
[ 6.132742] sof-audio sof-audio: panic: dsp_oops_offset 788480 offset 788480
[ 6.132745] sof-audio sof-audio: status: fw entered - code 00000005
[ 6.132766] sof-audio sof-audio: error: runtime exception
[ 6.132769] sof-audio sof-audio: error: trace point 00004000
[ 6.132771] sof-audio sof-audio: error: panic happen at :0
[ 6.132774] sof-audio sof-audio: error: DSP Firmware Oops
[ 6.132776] sof-audio sof-audio: error: Exception Cause: LoadStorePIFDataErrorCause, Synchronous PIF data error during LoadStore access
[ 6.132779] sof-audio sof-audio: EXCCAUSE 0x0000000d EXCVADDR 0x0000039c PS 0x00060020 SAR 0x00000001
[ 6.132782] sof-audio sof-audio: EPC1 0xbe0166d7 EPC2 0xbe014f4e EPC3 0xbe013d48 EPC4 0x00000000
[ 6.132785] sof-audio sof-audio: EPC5 0x00000000 EPC6 0x00000000 EPC7 0x00000000 DEPC 0x00000000
[ 6.132787] sof-audio sof-audio: EPS2 0x00060d20 EPS3 0x00060722 EPS4 0x00000000 EPS5 0x00000000
[ 6.132790] sof-audio sof-audio: EPS6 0x00000000 EPS7 0x00000000 INTENABL 0x000101d0 INTERRU 0x00000222
be0166d4: 3f29a2 l32i a10, a9, 252
be0166d7: 272a92 l32i a9, a10, 156
be0166da: 022926 beqi a9, 2, be0166e0 <ipc_comp_dai_config+0x44>
be0166dd: db4966 bnei a9, 4, be0166bc <ipc_comp_dai_config+0x20>
Test env:
sof-master: 1024e94
soft-master: c0a881a
kernel sof-dev: 93853c
tplg: sof-apl-nocodec.tplg
mengdonglin commentedon Oct 17, 2018
@keqiaozhang Could you share more analysis info why this is a kernel driver issue?
RanderWang commentedon Oct 19, 2018
I tested latest master FW and kernel with hda topology on WHL, no such issue
12 remaining items
libinyang commentedon Nov 1, 2018
The bug can still be found:
sof-master: 43da169
sof tool: 918e401
kernel sof-dev: 2af9599a + thesofproject/linux#237
tplg: sof-apl-nocodec.tplg
sof-logger.log
sof-logger-t.log
dmesg.log
markyang commentedon Nov 1, 2018
Summary:
DSP panic also exists on UP^2 and on BYT.
both sof-logger and sof-logger -t are empty.
dmesg:
Reproduce step:
1: check dmesg immediately after system booted.
Test env:
sof master: 43da169
sof tool: 4ac9248
kernel debug/v4.19-backport: 925637e3
tplg: sof-byt-rt5651.tplg-4ac9248 on BYT
sof-apl-nocodec.tplg-4ac9248 on APL
dmesg-byt.txt
lgirdwood commentedon Nov 1, 2018
@markyang please also make sure you include the context prior to the crash
This shows the stream tag is wrong for DMA trace which probably causes the panic
[-]DSP panic after system booted[/-][+]Invalid DMA trace stream tag causing FW oops[/+]Jiangxinx commentedon Nov 2, 2018
Summary:
Still can be reproduced on BYT.
dmesg:
Reproduce step:
1: check dmesg immediately after system booted.
Test env:
sof master: 43da169
sof tool: 937032f
kernel topic/sof-dev: a5ddc3d
tplg: sof-byt-rt5651.tplg-937032f on BYT
Log:
dmesg.log
lgirdwood commentedon Nov 2, 2018
@Jiangxinx please use markdown when pasting logs - see my above changes.
ZhendanYang commentedon Nov 5, 2018
Move topology issue on up2 to #539 .
mengdonglin commentedon Nov 5, 2018
#443 is splitted into 2 bugs: one is #443 for stream_tag, another is described in #539
libinyang commentedon Nov 16, 2018
It seems this bug is assigned to me. I will debug this issue after I finish some other issues on UP2.
markyang commentedon Nov 22, 2018
Summary:
This issue can be reproduced on BYT with the following 3 tplgs:
test-ssp2-mclk-0-I2S-volume-s24le-s24le-48k-19200k-codec.tplg
test-ssp2-mclk-0-I2S-volume-s16le-s16le-48k-19200k-codec.tplg
and sof-byt-rt5651.tplg(input s32le output s24le).
Test env:
sof master: 7a79343
sof tool: 6da07a9
kernel sof-dev: b846ee5
mengdonglin commentedon Nov 23, 2018
@markyang could you verify the solution can solve this issue? We expect BYT can boot with the merged PR #613?
markyang commentedon Nov 23, 2018
Summary:
This issue cannot be reproduced on BYT with the following 3 tplgs:
test-ssp2-mclk-0-I2S-volume-s24le-s24le-48k-19200k-codec.tplg
test-ssp2-mclk-0-I2S-volume-s16le-s16le-48k-19200k-codec.tplg
and sof-byt-rt5651.tplg (input s32le output s24le).
Test env:
sof master: e8ef1f2
sof tool: ad3f3e4
kernel sof-dev: 4a3aa5f