Skip to content

[CML][BUG ]Multi-pipeline: all streams killed when DSP core utilisation > 100% #2280

Closed
Closed
[CML][BUG ]Multi-pipeline: all streams killed when DSP core utilisation > 100%#2280
Assignees
sinahuang
Labels
CMLApplies to Comet Lake platformDSP overloadIssue caused by overload of DSPP2Critical bugs or normal featuresSDWSoundWirebugSomething isn't working as expected
Milestone
v1.6

Description

@sinahuang
sinahuang
opened on Jan 13, 2020

Describe the bug
Multi-pipeline display "input/output error" when playback/capture more than 6 piplines at the same time. Aplay/arecord function can't be effected after issue occured. Can recover it if playback/capture not more than 6 piplines.

To Reproduce
1.sudo reboot
2.Run "./multiplex_pipeline-test.sh pcm0p pcm2p pcm3p pcm5p pcm7p pcm1c pcm4c".

scripts:
multiplex_pipeline-test.txt

Reproduce rate
5/5

Expected result
Multi-pipeline should work normally.

Actual result
Multi-pipeline display "input/output error" when playback/capture more than 6 piplines at the same time.
Screenshot from 2020-01-07 03-24-38

sof_error_trace:

CORE  LEVEL      COMP_ID                TIMESTAMP            DELTA                FILE_NAME	CONTENT
    0      1  SCHEDULE_LL         3578224875.260417              nan edule/timer_domain.c:34  	timer_report_delay(): timer 3 delayed by 37 uS 901 ticks

Test recipe

Kernel:https://github.com/thesofproject/linux/tree/integration/soundwire-latest commit: 5e7259b +PR1692
FW: https://github.com/thesofproject/sof/commits/master commit: 418c77e
Tplg: Same with FW branch, sof-cml-rt711-rt1308-rt715.tplg
Platform:CML-H Laptop with codec ALC711, 2 x RT1308, RT715 in SDW mode.

dmesg.log : not have error/failed info in dmesg log
error.log
sof_trace.log

Activity

  1. sinahuang
    added
    bugSomething isn't working as expected
    on Jan 13, 2020
  2. sinahuang
    changed the title [-][CML][BUG]Multipipe display "input/output error" when aplay/arecord is more than 6 pipleline[/-] [+][CML][BUG]Multi-pipeline display "input/output error" when aplay/arecord is more than 6 pipleline[/+] on Jan 13, 2020
  3. YvonneYang2
    changed the title [-][CML][BUG]Multi-pipeline display "input/output error" when aplay/arecord is more than 6 pipleline[/-] [+][CML][BUG]Multi-pipeline display "input/output error" when playback/capture more than 6 piplines at the same time[/+] on Jan 13, 2020
  4. sinahuang
    added
    SDWSoundWire
    CMLApplies to Comet Lake platform
    on Jan 13, 2020

  5. sinahuang commented on Jan 14, 2020

    @sinahuang
    sinahuang
    on Jan 14, 2020
    Author

    This issue is caused by commit 91fb594

  6. ClarexZhou commented on Jan 14, 2020

    @ClarexZhou
    ClarexZhou
    on Jan 14, 2020

    @mmaka1 could you please check first?

  7. paulstelian97 commented on Jan 14, 2020

    @paulstelian97
    paulstelian97
    on Jan 14, 2020
    Collaborator

    Is it possible to dynamically use a different oscillator dependent on load? As in using the higher performance one when there is need.

  8. lgirdwood
    added
    P2Critical bugs or normal features
    on Jan 14, 2020
  9. lgirdwood
    added this to the v1.5 milestone on Jan 14, 2020

  10. lgirdwood commented on Jan 14, 2020

    @lgirdwood
    lgirdwood
    on Jan 14, 2020
    Member

    @plbossart any ideas, nothing obvious in any of the logs. Not sure yet if it's SDW only.

  11. lgirdwood commented on Jan 14, 2020

    @lgirdwood
    lgirdwood
    on Jan 14, 2020
    Member

    @sinahuang can you retry this with 2,3,4,5 streams - it may still be an issue with less streams.

  12. sinahuang commented on Jan 15, 2020

    @sinahuang
    sinahuang
    on Jan 15, 2020
    Author

    Test with 2,3,4,5 streams, issue can't be reproduced. Issue only can be reproduced when more than 6 piplines.

  13. plbossart commented on Jan 15, 2020

    @plbossart
    plbossart
    on Jan 15, 2020
    Member

    on this hardware there are 3 playback devices and 2 capture ones.
    What you are doing is enabling all of them, and then add HDMI.

    I suspect you must run out of resources at some point and underflow. That would explain why there's no error in dmesg, but you consistently cannot play

    Rather than using a scripts, please try manually adding a stream for each of the devices in a different ssh terminal, and send the logs when the error happen (terminal output, dmesg logs, DSP logs).

  14. sinahuang commented on Jan 15, 2020

    @sinahuang
    sinahuang
    on Jan 15, 2020
    Author

    On CML-H Laptop there are 5 playback devices and 2 capture ones.

    **** List of PLAYBACK Hardware Devices ****
card 0: sofsdwrt7111308 [sof-sdw-rt711-1308-715], device 0: Headphone (*) []
  Subdevices: 1/1
  Subdevice #0: subdevice #0
card 0: sofsdwrt7111308 [sof-sdw-rt711-1308-715], device 2: SDW1-speakers (*) []
  Subdevices: 1/1
  Subdevice #0: subdevice #0
card 0: sofsdwrt7111308 [sof-sdw-rt711-1308-715], device 3: SDW2-speakers (*) []
  Subdevices: 1/1
  Subdevice #0: subdevice #0
card 0: sofsdwrt7111308 [sof-sdw-rt711-1308-715], device 5: HDMI1 (*) []
  Subdevices: 1/1
  Subdevice #0: subdevice #0
card 0: sofsdwrt7111308 [sof-sdw-rt711-1308-715], device 6: HDMI2 (*) []
  Subdevices: 1/1
  Subdevice #0: subdevice #0
card 0: sofsdwrt7111308 [sof-sdw-rt711-1308-715], device 7: HDMI3 (*) []
  Subdevices: 1/1
  Subdevice #0: subdevice #0
ubuntu@cd-CML-H-rt711-rt1308-rt715:~$ arecord -l
**** List of CAPTURE Hardware Devices ****
card 0: sofsdwrt7111308 [sof-sdw-rt711-1308-715], device 1: Headset mic (*) []
  Subdevices: 1/1
  Subdevice #0: subdevice #0
card 0: sofsdwrt7111308 [sof-sdw-rt711-1308-715], device 4: DMIC (*) []
  Subdevices: 1/1
  Subdevice #0: subdevice #0
ubuntu@cd-CML-H-rt711-rt1308-rt715:~$

    If I manually adding a stream for each of the devices in a different ssh terminal, the error happened after added the 7th devices.

    dmesg logs:
    dmesg.log
    kern.log
    sof_error_trace.log

     CORE  LEVEL      COMP_ID                TIMESTAMP            DELTA                FILE_NAME	CONTENT
    0      1           SA         439946318.854167 439946304.000000      src/lib/agent.c:54  	validate(), ll drift detected, delta = 25223
    0      1  SCHEDULE_LL         439950019.375000      3700.520752 edule/timer_domain.c:34  	timer_report_delay(): timer 3 delayed by 35 uS 844 ticks

    Terminal output:

    Screenshot from 2020-01-08 22-37-36

  15. 13 remaining items

  16. YvonneYang2 commented on Jan 19, 2020

    @YvonneYang2
    YvonneYang2
    on Jan 19, 2020

    Run 6 multipiplines aplay/arecord at the same time, and switch aplay0,2(RT1308) to background, and back to foreground can also cause IO error.
    May be related to this issue?

    e.g
    Terminal 1: aplay -Dhw:0,0 -r48000 -c2 -fs16_le wav/48California_Dreaming.wav -vv -i
    Terminal 2: aplay -Dhw:0,2 -r48000 -c2 -fs16_le wav/48California_Dreaming.wav -vv -i
    Terminal 3: aplay -Dhw:0,6 -r48000 -c2 -fs16_le wav/48California_Dreaming.wav -vv -i
    Terminal 4: aplay -Dhw:0,7 -r48000 -c2 -fs16_le wav/48California_Dreaming.wav -vv -i
    Terminal 5: arecord -Dhw:0,1 -r48000 -c2 -fs16_le test1.wav -vv -i
    Terminal 6: arecord -Dhw:0,4 -r48000 -c2 -fs16_le test4.wav -vv -i

    Switch aplay0,2(RT1308) to background, and back to foreground in terminal 2, IO error occurred.

    dmesg.log
    sof_trace.log

    Test recipe:
    Kernel: https://github.com/thesofproject/linux/commits/topic/sof-dev (commit: 674afed)
    FW: https://github.com/thesofproject/sof/commits/master commit: 415273f
    Platform: CML-U sdw laptop

  17. lgirdwood commented on Jan 20, 2020

    @lgirdwood
    lgirdwood
    on Jan 20, 2020
    Member

    @YvonneYang2 yep, any EDF overload will currently kill all streams and this is currently showing in low power clock mode. We are looking at EDF resource management.

  18. lgirdwood
    modified the milestones: v1.5, v1.6 on Jan 27, 2020

  19. lgirdwood commented on Jan 27, 2020

    @lgirdwood
    lgirdwood
    on Jan 27, 2020
    Member

    EDF resource management target v1.6

  20. tlauda commented on Mar 10, 2020

    @tlauda
    tlauda
    on Mar 10, 2020
    Contributor

    @lgirdwood @lbetlej What is the expectation here? Pipelines are scheduled with low latency scheduler (timer) and not with EDF, so EDF resource management won't help here. I imagine that the first step would be to review what pipelines are run and measure their cycles. We need to assess whether some of the components are not optimized. Assuming everything will look correct, I think the resource management should be manually set based on worst case scenario and taking into consideration topology and clock settings. If we want to keep lower clock, that's OK, but some of the pipelines should be started on slave cores.

  21. sinahuang commented on Mar 11, 2020

    @sinahuang
    sinahuang
    on Mar 11, 2020
    Author

    Test with sof-dev (commit:d05959) +master (commit:8e5ab9c), issue also can be reproduced on TGL-U RVP with codec ALC711 in SDW mode and RT1308 in I2S mode when playback/capture more than 5 piplines at the same time.

  22. lgirdwood commented on Mar 12, 2020

    @lgirdwood
    lgirdwood
    on Mar 12, 2020
    Member

    @tlauda we do need resource management, i.e. each component has a measured worst case cost per number of samples per platform. e.g volume 24bit cost on platform X is Y cycles. This can then bo worked out at runtime and rejected if new load + current load > budget.

  23. lbetlej commented on Mar 12, 2020

    @lbetlej
    lbetlej
    on Mar 12, 2020
    Collaborator

    @lgirdwood @lauda resource management is an interesting topic itself:

    1. I'm with you on component level worst case compute requirements for each audio format
    2. The measurement process has to account: DSP configuration, SRAM memory access time (waitstates) and cache worst case latency for each audio format (SRC as an example)
    3. The worst case utilization has to be visible at 2 levels:
    • topology so that during topology creation the system integrator can verify if the workload fit's in DSP
    • at FW level to verify if new pipeline is exceeding compute budget
  24. mengdonglin
    added
    DSP overloadIssue caused by overload of DSP
    on Mar 24, 2020

  25. mengdonglin commented on Mar 24, 2020

    @mengdonglin
    mengdonglin
    on Mar 24, 2020
    Collaborator

    Caused by DSP overload, please see comments of @tlauda #2531 (comment)

  26. slawblauciak commented on Apr 7, 2020

    @slawblauciak
    slawblauciak
    on Apr 7, 2020
    Collaborator

    @sinahuang should no longer occur on master, high performance clock is enabled by default

  27. slawblauciak
    assigned
    sinahuang
    on Apr 7, 2020

  28. sinahuang commented on Apr 7, 2020

    @sinahuang
    sinahuang
    on Apr 7, 2020
    Author

    Issue can't be reproduced with sof-dev commit:052e45d and master commit: b4e8b1e on on CML-H and CML-U, so closed it.

  29. sinahuang
    closed this as completedon Apr 7, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

CMLApplies to Comet Lake platformDSP overloadIssue caused by overload of DSPP2Critical bugs or normal featuresSDWSoundWirebugSomething isn't working as expected

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions