Repository navigation
ASoC: SOF: pcm: harden PCM STOP sequence - #1169
Conversation
58120b3 to
e72d4a2
Compare
|
@xiulipan this beats me. Where is this need coming from? Which platform are we seeing the xrun on? |
|
same comment as @ranj063, where does this come from? What issue does this solve and on which platforms? |
|
@ranj063 @plbossart @tlauda @keyonjie |
|
@ranj063 @xiulipan @plbossart Please look at the #1160. I explained potential problem for the HDA Link pipelines, but getting an xrun is also a possibility, if the delay between DMA stop and IPC will be too big. |
|
I am not clear on the suggested solutions here. @xiulipan 's proposal in this PR is to send an IPC Pause followed by an IPC stop. This doesn't address handling of the RUN bits. @tlauda 's suggestion is to send an IPC Stop followed by a DMA stop (with the RUN bits changed by the driver). Pick one please. |
|
@plbossart an extra TRIGGER_PAUSE IPC is from the proposed DMA stop sequence from FW team to avoid xrun happening. With my knowledge the FW side almost handling PAUSE IPC very similar with that of STOP IPC, so will this extra pause IPC introduce any failure at double stopping? As @tlauda suggested, maybe worth to try only exchange the position of the RUN bit setting and TRIGGER_STOP IPC sending solution. |
|
@plbossart @keyonjie Extra PAUSE won't introduce any problems, but actually it's not needed. Changing the sequence of RUN and TRIGGER_STOP is enough. |
|
@plbossart
What @tlauda means is that we should not only have DMA RUN0 then IPC PAUSE/STOP, we may have some gap between and have XRUN risk. So what he meant is we should also harden PCM PAUSE sequence with
I agree with that idea to harden both DMA STOP and PIPELINE STOP/PAUSE sequence. Now the only problem is solved by @tlauda I will try to make the PR update for both PCM PAUSE and STOP |
|
Setting the 'Unclear' label since we don't have an agreed state machine between driver and firmware teams. |
|
@plbossart I've already defined the proper state transition:
@xiulipan We don't need additional SOF_IPC_STREAM_TRIG_PAUSE. |
|
@tlauda @plbossart |
|
@xiulipan Yes, it's the same situation. |
e72d4a2 to
9864dd5
Compare
|
@plbossart @tlauda
|
|
@tlauda |
|
@tlauda |
plbossart
left a comment
There was a problem hiding this comment.
the error handling looks suspicious on stop
|
@xiulipan the commit message is so confusing. You are saying the old sequence is Stop DMA -> Stop IPC. And then you are saying you should change it to be the same? |
9864dd5 to
c455794
Compare
ranj063
left a comment
There was a problem hiding this comment.
@xiulipan also another minor comment about the commit message:
"If delay happen before the step 1 and 2," should be "If delay happen between the steps 1 and 2,"
And in the end, say that the suggested change to avoid the issue is to switch the order of steps 1 and 2 for the stop sequence.
plbossart
left a comment
There was a problem hiding this comment.
please update the error handling and address feedback on naming
c455794 to
e247931
Compare
|
@plbossart @ranj063 |
| ret = sof_ipc_tx_message(sdev->ipc, stream.hdr.cmd, &stream, | ||
| sizeof(stream), &reply, sizeof(reply)); | ||
|
|
||
| if (ipc_first) |
There was a problem hiding this comment.
@xiulipan in case the STOP IPC fails, is this expected that you still call platform_trigger. If yes you should make it explicit and explain why you are ignoring the return value and using it below.
There was a problem hiding this comment.
@plbossart Will add comments. It seems either we fail the IPC or not we must stop the DMA stop.
There was a problem hiding this comment.
@plbossart Another question. If IPC stop fails, do we need to free the PCM to make sure the pipeline can be reused?
There was a problem hiding this comment.
@kv2019i can you look into this? I don't recall the code changes that were committed in July and the test below for the pcm_free case looks suspicious to me.
There was a problem hiding this comment.
@xiulipan @plbossart The fix looks good. The test for pcm_free indeed has the same problem. My assumption was the if the IPC fails for stop, something fatal has happened and no bother sending the dsp_pcm_free(). But if stop will fail in an xrun case like @xiulipan you explained earlier, that changes things. Probably we should use different logic for start and stop -- abort with error if start fails to an intermediate step. For stopping/pausing, we should send all the IPCs still despite errors to ensure we free resources. To me it still sounds a bit fishy that stop would legitimately return an error, but if that's the FW API, then we have to follow it.
That does raise other issue. I tried to think of a nicer way to implement the "ipc_first" logic. If we now have differences in error handling, that will make the two code paths even more different. Maybe time for a different subroutines for starting data transfer and stopping it.:::?
There was a problem hiding this comment.
@kv2019i
So you would suggest we will always send FREE IPC no matter if IPC STOP fail
actually, we will only send IPC FREE after IPC STOP.
Actually I tried with some other way like switch case the start and stop in different code. But @ranj063 suggest to go with some simple flag. And this really looks simple but need a few comments to explain why we need do like that.
e247931 to
65e46f5
Compare
kv2019i
left a comment
There was a problem hiding this comment.
Good patch! Will need some work for merging.
| } | ||
|
|
||
| snd_sof_pcm_platform_trigger(sdev, substream, cmd); | ||
| /* DMA and IPC sequence is different for start and stop. Need to send |
There was a problem hiding this comment.
minor nit: start multi-line comment with "/*\n"
| ret = sof_ipc_tx_message(sdev->ipc, stream.hdr.cmd, &stream, | ||
| sizeof(stream), &reply, sizeof(reply)); | ||
|
|
||
| if (ipc_first) |
There was a problem hiding this comment.
@xiulipan @plbossart The fix looks good. The test for pcm_free indeed has the same problem. My assumption was the if the IPC fails for stop, something fatal has happened and no bother sending the dsp_pcm_free(). But if stop will fail in an xrun case like @xiulipan you explained earlier, that changes things. Probably we should use different logic for start and stop -- abort with error if start fails to an intermediate step. For stopping/pausing, we should send all the IPCs still despite errors to ensure we free resources. To me it still sounds a bit fishy that stop would legitimately return an error, but if that's the FW API, then we have to follow it.
That does raise other issue. I tried to think of a nicer way to implement the "ipc_first" logic. If we now have differences in error handling, that will make the two code paths even more different. Maybe time for a different subroutines for starting data transfer and stopping it.:::?
| if (ipc_first) | ||
| snd_sof_pcm_platform_trigger(sdev, substream, cmd); | ||
|
|
||
| /* only need to free PCM is the pipeline is stopped successfully */ |
There was a problem hiding this comment.
typo: "free PCM if the pipeline"
Otherwise see comments above. This logic may be wrong. It was written with the interpretation of FW API that no non-fatal error can be returned for stop. Apparently this may not be the case, in which case this code needs to be changed.
There was a problem hiding this comment.
I will change it back to IPC FREE with IPC STOP no matter what IPC STOP ret.
65e46f5 to
f8aa7f0
Compare
| if (ipc_first) | ||
| snd_sof_pcm_platform_trigger(sdev, substream, cmd); | ||
|
|
||
| /* free PCM no matter STOP fail or success*/ |
There was a problem hiding this comment.
that comment looks wrong?
if the STOP IPC fails with a non-zero return value, you will not free the PCM.
What I am missing?
There was a problem hiding this comment.
@plbossart @xiulipan Yeah, this is a bit confusing as you ignore the platform_trigger return value but do not ignore the IPC send return value. So maybe call sof_pcm_dsp_pcm_free() whenever reset_hw_params is set, independently of "ret".
To no overwrite error values, you could add "ret/err" logic I added to d6064e8 to overcome a similar situation (this design pattern for collecting error values is at least used elsewhere in ALSA).
There was a problem hiding this comment.
@kv2019i I agree the comment is incorrect but I suppose the sequence is like this for the STOP trigger:
- send STOP IPC
- stop DMA
- free pcm is step 1 is successful and reset_hw_params is set.
So Xiuli's change looks legit to me except for the comment. on line 391 which should really say /* free PCM if reset_hw_params is set and the STOP IPC is successful */
kv2019i
left a comment
There was a problem hiding this comment.
This some issues (see inline).
| if (ipc_first) | ||
| snd_sof_pcm_platform_trigger(sdev, substream, cmd); | ||
|
|
||
| /* free PCM no matter STOP fail or success*/ |
There was a problem hiding this comment.
@plbossart @xiulipan Yeah, this is a bit confusing as you ignore the platform_trigger return value but do not ignore the IPC send return value. So maybe call sof_pcm_dsp_pcm_free() whenever reset_hw_params is set, independently of "ret".
To no overwrite error values, you could add "ret/err" logic I added to d6064e8 to overcome a similar situation (this design pattern for collecting error values is at least used elsewhere in ALSA).
|
@xiulipan can we try and close this PR this week? There are a couple of comments that need to be addressed and the last change was 6 days ago. |
The old STOP sequence is: 1. stop DMA 2. send STOP ipc If delay happen before the steps 1 and 2, the DMA buffer will be empty in short time and cause pipeline xrun then stop the pipeline. Then the step 2 ipc stop will return error as pipeline is already stopped. Suggested change to avoid the issue is to switch the order of steps 1 and 2 for the stop sequence. Signed-off-by: Pan Xiuli <xiuli.pan@linux.intel.com>
f8aa7f0 to
729b470
Compare
|
@plbossart @ranj063 @kv2019i or I will follow the old logic to free only if STOP success for this update. |
|
@xiulipan wrote:
Ack, ok, good for me. |
|
@xiulipan CI is stuck? I'd like to merge with actual results... |
|
@plbossart CI re-runed. And remove the RFC |
With kaslr the kernel image is placed at a random place, so starting the bottom-up allocation with the kernel_end can result in an allocation failure and a warning like this one: hugetlb_cma: reserve 2048 MiB, up to 2048 MiB per node ------------[ cut here ]------------ memblock: bottom-up allocation failed, memory hotremove may be affected WARNING: CPU: 0 PID: 0 at mm/memblock.c:332 memblock_find_in_range_node+0x178/0x25a Modules linked in: CPU: 0 PID: 0 Comm: swapper Not tainted 5.10.0+ #1169 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.14.0-1.fc33 04/01/2014 RIP: 0010:memblock_find_in_range_node+0x178/0x25a Code: e9 6d ff ff ff 48 85 c0 0f 85 da 00 00 00 80 3d 9b 35 df 00 00 75 15 48 c7 c7 c0 75 59 88 c6 05 8b 35 df 00 01 e8 25 8a fa ff <0f> 0b 48 c7 44 24 20 ff ff ff ff 44 89 e6 44 89 ea 48 c7 c1 70 5c RSP: 0000:ffffffff88803d18 EFLAGS: 00010086 ORIG_RAX: 0000000000000000 RAX: 0000000000000000 RBX: 0000000240000000 RCX: 00000000ffffdfff RDX: 00000000ffffdfff RSI: 00000000ffffffea RDI: 0000000000000046 RBP: 0000000100000000 R08: ffffffff88922788 R09: 0000000000009ffb R10: 00000000ffffe000 R11: 3fffffffffffffff R12: 0000000000000000 R13: 0000000000000000 R14: 0000000080000000 R15: 00000001fb42c000 FS: 0000000000000000(0000) GS:ffffffff88f71000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: ffffa080fb401000 CR3: 00000001fa80a000 CR4: 00000000000406b0 Call Trace: memblock_alloc_range_nid+0x8d/0x11e cma_declare_contiguous_nid+0x2c4/0x38c hugetlb_cma_reserve+0xdc/0x128 flush_tlb_one_kernel+0xc/0x20 native_set_fixmap+0x82/0xd0 flat_get_apic_id+0x5/0x10 register_lapic_address+0x8e/0x97 setup_arch+0x8a5/0xc3f start_kernel+0x66/0x547 load_ucode_bsp+0x4c/0xcd secondary_startup_64_no_verify+0xb0/0xbb random: get_random_bytes called from __warn+0xab/0x110 with crng_init=0 ---[ end trace f151227d0b39be70 ]--- At the same time, the kernel image is protected with memblock_reserve(), so we can just start searching at PAGE_SIZE. In this case the bottom-up allocation has the same chances to success as a top-down allocation, so there is no reason to fallback in the case of a failure. All together it simplifies the logic. Link: https://lkml.kernel.org/r/20201217201214.3414100-2-guro@fb.com Fixes: 8fabc62 ("powerpc: Ensure that swiotlb buffer is allocated from low memory") Signed-off-by: Roman Gushchin <guro@fb.com> Reviewed-by: Mike Rapoport <rppt@linux.ibm.com> Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com> Cc: Michal Hocko <mhocko@kernel.org> Cc: Rik van Riel <riel@surriel.com> Cc: Wonhyuk Yang <vvghjk1234@gmail.com> Cc: Thiago Jung Bauermann <bauerman@linux.ibm.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
[ Upstream commit 2dcb396 ] With kaslr the kernel image is placed at a random place, so starting the bottom-up allocation with the kernel_end can result in an allocation failure and a warning like this one: hugetlb_cma: reserve 2048 MiB, up to 2048 MiB per node ------------[ cut here ]------------ memblock: bottom-up allocation failed, memory hotremove may be affected WARNING: CPU: 0 PID: 0 at mm/memblock.c:332 memblock_find_in_range_node+0x178/0x25a Modules linked in: CPU: 0 PID: 0 Comm: swapper Not tainted 5.10.0+ thesofproject#1169 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.14.0-1.fc33 04/01/2014 RIP: 0010:memblock_find_in_range_node+0x178/0x25a Code: e9 6d ff ff ff 48 85 c0 0f 85 da 00 00 00 80 3d 9b 35 df 00 00 75 15 48 c7 c7 c0 75 59 88 c6 05 8b 35 df 00 01 e8 25 8a fa ff <0f> 0b 48 c7 44 24 20 ff ff ff ff 44 89 e6 44 89 ea 48 c7 c1 70 5c RSP: 0000:ffffffff88803d18 EFLAGS: 00010086 ORIG_RAX: 0000000000000000 RAX: 0000000000000000 RBX: 0000000240000000 RCX: 00000000ffffdfff RDX: 00000000ffffdfff RSI: 00000000ffffffea RDI: 0000000000000046 RBP: 0000000100000000 R08: ffffffff88922788 R09: 0000000000009ffb R10: 00000000ffffe000 R11: 3fffffffffffffff R12: 0000000000000000 R13: 0000000000000000 R14: 0000000080000000 R15: 00000001fb42c000 FS: 0000000000000000(0000) GS:ffffffff88f71000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: ffffa080fb401000 CR3: 00000001fa80a000 CR4: 00000000000406b0 Call Trace: memblock_alloc_range_nid+0x8d/0x11e cma_declare_contiguous_nid+0x2c4/0x38c hugetlb_cma_reserve+0xdc/0x128 flush_tlb_one_kernel+0xc/0x20 native_set_fixmap+0x82/0xd0 flat_get_apic_id+0x5/0x10 register_lapic_address+0x8e/0x97 setup_arch+0x8a5/0xc3f start_kernel+0x66/0x547 load_ucode_bsp+0x4c/0xcd secondary_startup_64_no_verify+0xb0/0xbb random: get_random_bytes called from __warn+0xab/0x110 with crng_init=0 ---[ end trace f151227d0b39be70 ]--- At the same time, the kernel image is protected with memblock_reserve(), so we can just start searching at PAGE_SIZE. In this case the bottom-up allocation has the same chances to success as a top-down allocation, so there is no reason to fallback in the case of a failure. All together it simplifies the logic. Link: https://lkml.kernel.org/r/20201217201214.3414100-2-guro@fb.com Fixes: 8fabc62 ("powerpc: Ensure that swiotlb buffer is allocated from low memory") Signed-off-by: Roman Gushchin <guro@fb.com> Reviewed-by: Mike Rapoport <rppt@linux.ibm.com> Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com> Cc: Michal Hocko <mhocko@kernel.org> Cc: Rik van Riel <riel@surriel.com> Cc: Wonhyuk Yang <vvghjk1234@gmail.com> Cc: Thiago Jung Bauermann <bauerman@linux.ibm.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org> Signed-off-by: Sasha Levin <sashal@kernel.org>
The old STOP sequence is: 1. stop DMA 2. send STOP ipc
If delay happen before the step 1 and 2, the DMA buffer will be empty in
short time and cause pipeline xrun then stop the pipeline.
Then the step 2 ipc stop will return error as pipeline is already stopped.
Harden the STOP sequence by send STOP IPC before stop DMA.
Signed-off-by: Pan Xiuli xiuli.pan@linux.intel.com