[BUG]ERROR ewt in etrace

Describe the bug
Multiple 'ERROR ewt' in etrace log when check-runtime-pm-status on apl-up2-nocodec

To Reproduce
TPLG=sof-apl-nocodec.tplg ~/sof-test/test-case/check-runtime-pm-status.sh -l 15

Reproduction Rate
100%

Environment
Kernel Branch: topic/sof-dev
Kernel Commit: dcea8052
Kernel PR: 2938
SOF Branch: main
SOF Commit: 636cbef

Platform
APL_UP2_NOCODEC

Screenshots or console output

[  8008757921.656386] (           0.000000) c0 dma-trace             src/trace/dma-trace.c:335  ERROR FW ABI 0x3012001 DBG ABI 0x5003000 tag v1.8-rc1-114-g636cbefc62c9 src hash 0x85629837 (ldc hash 0x85629837)
[     7214463.254990] (     7214463.500000) c0 wait                         src/lib/wait.c:45   ERROR ewt
[    15517671.102133] (     8303208.000000) c0 wait                         src/lib/wait.c:45   ERROR ewt
[    16210527.532935] (      692856.437500) c0 wait                         src/lib/wait.c:45   ERROR ewt
[    24186156.747262] (     7975629.000000) c0 wait                         src/lib/wait.c:45   ERROR ewt
[    32125762.994270] (     7939606.000000) c0 wait                         src/lib/wait.c:45   ERROR ewt
[    32455645.481162] (      329882.500000) c0 wait                         src/lib/wait.c:45   ERROR ewt
[    41285201.692807] (     8829556.000000) c0 wait                         src/lib/wait.c:45   ERROR ewt
[    50000448.794411] (     8715247.000000) c0 wait                         src/lib/wait.c:45   ERROR ewt
[    50010666.814838] (       10218.020508) c0 wait                         src/lib/wait.c:45   ERROR ewt
[    58100591.857957] (     8089925.000000) c0 wait                         src/lib/wait.c:45   ERROR ewt
[    66559722.407238] (     8459131.000000) c0 wait                         src/lib/wait.c:45   ERROR ewt

@juimonen

Thanks, this is not a very helpful message and it's the last of the legacy message types.

int poll_for_register_delay(uint32_t reg, uint32_t mask,
			    uint32_t val, uint64_t us)
{
	uint64_t tick = clock_us_to_ticks(PLATFORM_DEFAULT_CLOCK, us);
	uint32_t tries = DEFAULT_TRY_TIMES;
	uint64_t delta = tick / tries;

	if (!delta) {
		delta = us;
		tries = 1;
	}

	while ((io_reg_read(reg) & mask) != val) {
		if (!tries--) {
			tr_err(&wait_tr, "ewt");
			return -EIO;
		}
		wait_delay(delta);
	}
	return 0;
}

@juimonen can you send a quick PR that prints a poll timeout message and includes the register, mask, val and us fields. This will tell us which user (dw-dma, ssp or pm on Intel platforms is showing the issue). I would also make all users check the return value too and print an error.

@iuliana-prodan

This is from the ssp_stop(). the change from @iuliana-prodan to stop ssp dai before stopping DMA introduce this error for us, we don't see this issue on byt/bdw as we configure CONFIG_DMA_SUSPEND_DRAIN there. @juimonen FYI in case you will work on it. The quick fix could be configure CONFIG_DMA_SUSPEND_DRAIN for all Intel platforms, but I am not very happy with that.

#if CONFIG_DMA_SUSPEND_DRAIN
		ret = dma_stop(dd->chan);
		dai_trigger(dd->dai, cmd, dev->direction);
#else
		dai_trigger(dd->dai, cmd, dev->direction);
		ret = dma_stop(dd->chan);
#endif

@lgirdwood

@lgirdwood @keyonjie yeah I can do the prints, but you need to decide what is the actual fix :)

@lgirdwood

@lgirdwood @keyonjie yeah I can do the prints, but you need to decide what is the actual fix :)

@juimonen can you do the trace update and probably @keyonjie can fix the SSP ?

#4251, printing here. Only one (I think) who was not checking return in some way was sai.c

FWIW I've seen this ERROR ewt messages since forever. That's because I've been working on the logger for some time.

EDIT: they were rare though.

I made some enhancements recently but generally speaking sof-test has not cared about errors in the firmware logs:

thesofproject/sof-test#297 (comment)
thesofproject/sof-test#666
etc.

@slawblauciak

Debug PRs merged, so it looks like we are polling on HW change that's not happening and will need inputs from @slawblauciak

haven't seen it in CI for months

@XiaoyunWu6666

@XiaoyunWu6666 we don't detect firmware ERRORS. Did you really spend the enormous time needed to look at every etrace of every test on every platform every day? I doubt it.

thesofproject/sof-test#799

src/lib/wait.c:51 ERROR poll timeout is another, more frequent error we don't catch

https://sof-ci.01.org/sofpr/PR5192/build12056/devicetest/?model=APL_UP2_NOCODEC_ZEPHYR&testcase=multiple-pause-resume-5

Close as this issue cannot be reproduced in recent daily testing.

@mengdonglin

@mengdonglin sof-test still does not look for firmware ERRORs so of course it hasn't been seen.

Navigation Menu

Repository navigation

[BUG]ERROR ewt in etrace #4248

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[BUG]ERROR ewt in etrace #4248

Description

Activity

lgirdwood commented on May 26, 2021

keyonjie commented on May 26, 2021

juimonen commented on May 26, 2021

lgirdwood commented on May 26, 2021

juimonen commented on May 26, 2021

marc-hb commented on May 26, 2021

lgirdwood commented on Jun 3, 2021

XiaoyunWu6666 commented on Jan 24, 2022

marc-hb commented on Jan 24, 2022

marc-hb commented on Feb 16, 2022

mengdonglin commented on Sep 6, 2022

marc-hb commented on Sep 6, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions