Skip to content

ASoC: SOF: pcm: fix resource leak in hw_free - #1194

Merged
plbossart merged 1 commit into
thesofproject:topic/sof-devfrom
kv2019i:fix/hw-free-fix
Sep 16, 2019
Merged

plbossart merged 1 commit into
thesofproject:topic/sof-devfrom
kv2019i:fix/hw-free-fix

Conversation

@kv2019i

@kv2019i kv2019i commented Sep 11, 2019

Copy link
Copy Markdown
Collaborator

Fix a bug in sof_pcm_hw_free() where some cleanup actions were
skipped if STREAM_PCM_FREE IPC was already succesfully sent to
DSP when the stream was stopped or suspended. This is incorrect
as hw_free should clean up also other resources, including pcm
lib page allocations, period elapsed work queue and call to
platform hw_free.

Fixes: c29d96c ("ASoC: SOF: reset DMA state in prepare")
Signed-off-by: Kai Vehmanen kai.vehmanen@linux.intel.com

@plbossart plbossart left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Error handling looks suspicious to me

Comment thread sound/soc/sof/pcm.c
Comment thread sound/soc/sof/pcm.c Outdated
if (spcm->prepared[substream->stream]) {
ret = sof_pcm_dsp_pcm_free(substream, sdev, spcm);
if (ret < 0)
return ret;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if ret < 0 you skip the follow-up cleanup...

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and the test below after cancel_work_sync may use an uninitialized value since ret is not set

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@plbossart Oops sorry, my bad, I was meaning to remove the check on L269-270. But, but, I just replied to #1169 and it seems I have had incorrect assumptions about the FW ABI. Reflecting the same discussion here (and e.g. implementation of snd_pcm_hw_free() in pcm_native.c), the correct action would indeed seem to be capture the error but not give any of the following clean-up steps. I will update patch accordingly.

Fix a bug in sof_pcm_hw_free() where some cleanup actions were
skipped if STREAM_PCM_FREE IPC was already successfully sent to
DSP when the stream was stopped or suspended. This is incorrect
as hw_free should clean up also other resources, including pcm
lib page allocations, period elapsed work queue and call to
platform hw_free.

Fixes: c29d96c ("ASoC: SOF: reset DMA state in prepare")
Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
@kv2019i

kv2019i commented Sep 11, 2019

Copy link
Copy Markdown
Collaborator Author

FYI @plbossart . This is pretty bad as we never run the end of hw_params in current code. It seems we have been a bit lucky in that we are not leaking memory due to the preallocated DMA buffers, so the missing call to snd_pcm_lib_free_pages() has had limited impact. I've been testing today with kmemleak and see no reports (without this patch). I also tested running a stress test of aplay and have been monitoring kernel memory stats and they seem normal. Phew. Patch updated with fixes, I'll continue testing tomorrow.

@keyonjie

Copy link
Copy Markdown

FYI @plbossart . This is pretty bad as we never run the end of hw_params in current code. It

Hi @kv2019i do you mean that the invoking to sof_pcm_dsp_params() will return failure in your cases?

seems we have been a bit lucky in that we are not leaking memory due to the preallocated DMA buffers, so the missing call to snd_pcm_lib_free_pages() has had limited impact. I've been testing today with kmemleak and see no reports (without this patch). I also tested running a stress test of aplay and have been monitoring kernel memory stats and they seem normal. Phew. Patch updated with fixes, I'll continue testing tomorrow.

@kv2019i

kv2019i commented Sep 12, 2019

Copy link
Copy Markdown
Collaborator Author

@keyonjie wrote

Hi @kv2019i do you mean that the invoking to sof_pcm_dsp_params() will return failure in
your cases?

No. I mean that in current code, in typical use-case dsp_pcm_free() is already run successfully in sof_pcm_trigger(STOP). When PCM device is closed, code goes to he_free, but as prepared state is false (as dsp_pcm_free() is already run), we skip clearing DSP PCM state, but the problem is that we also skip other actions in hw_free() which is not correct.

@keyonjie

keyonjie commented Sep 12, 2019 •

Copy link
Copy Markdown

@keyonjie wrote

Hi @kv2019i do you mean that the invoking to sof_pcm_dsp_params() will return failure in
your cases?

No. I mean that in current code, in typical use-case dsp_pcm_free() is already run successfully in sof_pcm_trigger(STOP). When PCM device is closed, code goes to he_free, but as prepared state is false (as dsp_pcm_free() is already run), we skip clearing DSP PCM state, but the problem is that we also skip other actions in hw_free() which is not correct.

um, I got your point now, sending PCM_FREE IPC at trigger stop was not good idea IMHO, but I don't recall why we do it in that way (to fix some suspend/resume issue?).

@kv2019i

kv2019i commented Sep 12, 2019

Copy link
Copy Markdown
Collaborator Author

@keyonjie wrote:

um, I got your point now, sending PCM_FREE IPC at trigger stop was not good idea IMHO, but I don't recall why we do it in that way (to fix some suspend/resume issue?).

When ALSA does a stop and start (without close/open cycle), the DMA must be reprogrammed (see my patch "ASoC: SOF: reset DMA state in prepare" from June 19). Thus we need to do PCM_FREE_IPC at stop, so that in the start flow we are in the expected state. This broke start/stop tests on a number of apps. Doing at stop seemed better as it's less time critical here (versus adding more IPC hops to the start path where we'd like to get the stream started as soon as possible).

@keyonjie

Copy link
Copy Markdown

@keyonjie wrote:

um, I got your point now, sending PCM_FREE IPC at trigger stop was not good idea IMHO, but I don't recall why we do it in that way (to fix some suspend/resume issue?).

When ALSA does a stop and start (without close/open cycle), the DMA must be reprogrammed

Why? the hw_params is not changed, why DMA need to be reprogrammed?

(see my patch "ASoC: SOF: reset DMA state in prepare" from June 19). Thus we need to do PCM_FREE_IPC at stop, so that in the start flow we are in the expected state. This broke start/stop tests on a number of apps. Doing at stop seemed better as it's less time critical here (versus adding more IPC hops to the start path where we'd like to get the stream started as soon as possible).

@kv2019i

kv2019i commented Sep 12, 2019

Copy link
Copy Markdown
Collaborator Author

@keyonjie wrote:

When ALSA does a stop and start (without close/open cycle), the DMA must be reprogrammed

Why? the hw_params is not changed, why DMA need to be reprogrammed?

Did you read the patch commit message I referred to? :) It has a lengthy description.

Please check the summary. The ALSA PCM model expects hw_ptr/appl_ptr to be reset when device goes to PREPARED state and back to RUNNING. In Intel DSP, we have no other means to reset the position than to FREE_PCM and then resent the params. And the linked DMA needs to be reset as well, or otherwise playback will fail. PAUSE/UNPAUSE is a different flow.

@plbossart plbossart left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @kv2019i

@plbossart
plbossart merged commit c5dec63 into thesofproject:topic/sof-dev Sep 16, 2019
ujfalusi pushed a commit to ujfalusi/sof-linux that referenced this pull request Jul 29, 2026
If revalidating the zones of a zoned block device with
blk_revalidate_disk_zones() fails during a SCSI disk rescan, the following
lockdep splat is thrown:

[  347.251859] [  T11230] sda: failed to revalidate zones

[  347.261380] [  T11230] ======================================================
[  347.263882] [  T11230] WARNING: possible circular locking dependency detected
[  347.266353] [  T11230] 7.1.0+ thesofproject#1194 Not tainted
[  347.268052] [  T11230] ------------------------------------------------------
[  347.270537] [  T11230] tcsh/11230 is trying to acquire lock:
[  347.272555] [  T11230] ffffffff8f91d400 (wq_pool_mutex){+.+.}-{4:4}, at: destroy_workqueue+0x15d/0x8d0
[  347.275914] [  T11230]
                          but task is already holding lock:
[  347.278646] [  T11230] ffff88812fa1bcc0 (&q->q_usage_counter(io)thesofproject#5){++++}-{0:0}, at: blk_mq_freeze_queue_nomemsave+0x16/0x30
[  347.282503] [  T11230]
                          which lock already depends on the new lock.

[  347.286239] [  T11230]
                          the existing dependency chain (in reverse order) is:
[  347.289408] [  T11230]
                          -> thesofproject#2 (&q->q_usage_counter(io)thesofproject#5){++++}-{0:0}:
[  347.292437] [  T11230]        blk_alloc_queue+0x5ca/0x750
[  347.294379] [  T11230]        blk_mq_alloc_queue+0x14c/0x240
[  347.296375] [  T11230]        scsi_alloc_sdev+0x871/0xd10 [scsi_mod]
[  347.298619] [  T11230]        scsi_probe_and_add_lun+0x600/0xc50 [scsi_mod]
[  347.301056] [  T11230]        __scsi_scan_target+0x187/0x3b0 [scsi_mod]
[  347.303385] [  T11230]        scsi_scan_channel+0xf2/0x180 [scsi_mod]
[  347.305651] [  T11230]        scsi_scan_host_selected+0x20b/0x2d0 [scsi_mod]
[  347.308119] [  T11230]        do_scan_async+0x42/0x420 [scsi_mod]
[  347.310276] [  T11230]        async_run_entry_fn+0x94/0x5a0
[  347.312284] [  T11230]        process_one_work+0x8da/0x1690
[  347.314287] [  T11230]        worker_thread+0x5fe/0x1010
[  347.316216] [  T11230]        kthread+0x358/0x450
[  347.317675] [  T11230]        ret_from_fork+0x5b9/0x8e0
[  347.319181] [  T11230]        ret_from_fork_asm+0x11/0x20
[  347.320778] [  T11230]
                          -> thesofproject#1 (fs_reclaim){+.+.}-{0:0}:
[  347.322890] [  T11230]        fs_reclaim_acquire+0xd5/0x120
[  347.324464] [  T11230]        __kmalloc_cache_node_noprof+0x39/0x620
[  347.326223] [  T11230]        init_rescuer+0x19b/0x560
[  347.327697] [  T11230]        workqueue_init+0x33b/0x6a0
[  347.329224] [  T11230]        kernel_init_freeable+0x2eb/0x600
[  347.330881] [  T11230]        kernel_init+0x1c/0x140
[  347.332334] [  T11230]        ret_from_fork+0x5b9/0x8e0
[  347.333847] [  T11230]        ret_from_fork_asm+0x11/0x20
[  347.335360] [  T11230]
                          -> #0 (wq_pool_mutex){+.+.}-{4:4}:
[  347.337510] [  T11230]        __lock_acquire+0xdea/0x2260
[  347.339030] [  T11230]        lock_acquire+0x187/0x2f0
[  347.340495] [  T11230]        __mutex_lock+0x1ab/0x2600
[  347.341464] [  T11230]        destroy_workqueue+0x15d/0x8d0
[  347.342485] [  T11230]        disk_free_zone_resources+0xd5/0x560
[  347.343577] [  T11230]        blk_revalidate_disk_zones+0x620/0xac7
[  347.344723] [  T11230]        sd_zbc_revalidate_zones+0x1dd/0x790 [sd_mod]
[  347.345938] [  T11230]        sd_revalidate_disk+0xc66/0x8e60 [sd_mod]
[  347.347112] [  T11230]        scsi_rescan_device+0x1f9/0x310 [scsi_mod]
[  347.348318] [  T11230]        store_rescan_field+0x19/0x20 [scsi_mod]
[  347.349507] [  T11230]        kernfs_fop_write_iter+0x3d2/0x5e0
[  347.350565] [  T11230]        vfs_write+0x469/0x1000
[  347.351484] [  T11230]        ksys_write+0x116/0x250
[  347.352403] [  T11230]        do_syscall_64+0xf0/0x6e0
[  347.353361] [  T11230]        entry_SYSCALL_64_after_hwframe+0x4b/0x53
[  347.354533] [  T11230]
                          other info that might help us debug this:

[  347.356432] [  T11230] Chain exists of:
                            wq_pool_mutex --> fs_reclaim --> &q->q_usage_counter(io)thesofproject#5

[  347.358919] [  T11230]  Possible unsafe locking scenario:

[  347.360307] [  T11230]        CPU0                    CPU1
[  347.361327] [  T11230]        ----                    ----
[  347.362340] [  T11230]   lock(&q->q_usage_counter(io)thesofproject#5);
[  347.363344] [  T11230]                                lock(fs_reclaim);
[  347.364526] [  T11230]                                lock(&q->q_usage_counter(io)thesofproject#5);
[  347.365968] [  T11230]   lock(wq_pool_mutex);
[  347.366811] [  T11230]
                           *** DEADLOCK ***

This happens because SCSI disk rescan is executed from a work context
and a failure of blk_revalidate_disk_zones() causes a call to
disk_free_zone_resources() which will free the disk zone write plug
workqueue.

Avoid this by delaying the destruction of the disk zone write plug
workqueue to disk_release(). Do this by introducing the function
disk_release_zone_resources() and using this new function from
disk_release(). This new function destroys the zone write plugs workqueue
and calls disk_free_zone_resources(), thus allowing to remove the call to
destroy_workqueue() from disk_free_zone_resources().
disk_alloc_zone_resources() is modified to not create the disk zone
write plug work queue if it already exists.

Fixes: a8f59e5 ("block: use a per disk workqueue for zone write plugging")
Cc: stable@vger.kernek.org
Signed-off-by: Damien Le Moal <dlemoal@kernel.org>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Hannes Reinecke <hare@kernel.org>
Link: https://patch.msgid.link/20260701082155.1369996-1-dlemoal@kernel.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants