[FEATURE] Dependent PCMs - Userspace handling

@cujomalainey

Is your feature request related to a problem? Please describe.
This is not a FW feature but a userspace feature and here is probably the best place to log it for the moment.

There are now many audio features where we have multiple PCMs and some of these PCMs are dependant on the others.i.e.

PCM0 --> buf0 --> Processing --> buf1 --> demux--+--> buf2 --> DAI
                                                 |
PCM1 <----------------------------------- buf3 --+

PCM1 above is dependent on PCM0 for data

Consider the above example where playback PCM0 is the data source and synchronisation for capture PCM1. Triggering PCM1 without PCM0 being active would mean to data (or synchronisation).

Userspace is currently unaware of this requirement.

Describe the solution you'd like
Make userspace aware of this dependency so it can trigger in the correct sequence for dependent PCMs EDIT updated with comments from discussion.

The UCM tuple for PCM dependence as @cujomalainey and I have mentioned.
Echo ref (and similar PCM) to return -ENODATA on trigger if their dependant PCM is inactive (FW can return this value as part of trigger IPC, driver just passes up the stack).
Update aplay/arecord to print appropriate and helpful error if PCM returns ENODATA.
CRAS to use UCM tuple
Pulseaudio to log appropriate message if PCM trigger returns ENODATA. It could also lookup UCM tuple and use the UCM values in the log message.

Describe alternatives you've considered
Add this logic to FW and send fake zero data if PCM1 is activated first until PCM0 is activated. This is more complex for FW as it means running additional time source and DMA channel on FW. It also means there is no synchronisation between PCM0 and PCM1 (as they can be asynchronously triggered by users). See #2407

@dgreid

@dgreid @cujomalainey any take on CRAS changes here ?
@juimonen @kv2019i any comments on UCM and PA updates here ?
@lbetlej fyi.

@lgirdwood

@lgirdwood this is not enforceable, we have no control on userspace deployment and readiness in downstream distros. Just look how complicated it is just for people to get the firmware and topology files, UCM2 is not even ready.

The only way to do this would be a modification of PCM states so that the dependencies are handled within the kernel. That we can control, though it's still anyone's guess how applications will react if the PCM streaming is delayed or stopped with an error.

The only way to do this would be a modification of PCM states so that the dependencies are handled within the kernel. That we can control, though it's still anyone's guess how applications will react if the PCM streaming is delayed or stopped with an error.

The only thing I can think of today that would be similar in the kernel is USB audio device. i.e.

Active playback on PCM0 causes PCM1 device to be created and udev etc is notified.
Playback stopping on PCM0 causes PCM1 device to be deleted (like USB device removal) and udev etc is notified.

Userspace handles it like USB audio.

This would enforce the ordering and Pulseaudio/CRAS already work well with USB. It would then only require a topology tuple (to set the dependency relationship) and some logic in SOF core to create/delet dynamic PCM devices (like USB does).

@lgirdwood

@lgirdwood we have a notion of Echo Reference devices currently in our UCM. I have been trying to get the definition upstreamed. Its not as general as this use case but seems to serve a similar purpose.

The only way to do this would be a modification of PCM states so that the dependencies are handled within the kernel. That we can control, though it's still anyone's guess how applications will react if the PCM streaming is delayed or stopped with an error.

The only thing I can think of today that would be similar in the kernel is USB audio device. i.e.

Active playback on PCM0 causes PCM1 device to be created and udev etc is notified.

Playback stopping on PCM0 causes PCM1 device to be deleted (like USB device removal) and udev etc is notified.

Userspace handles it like USB audio.

This would enforce the ordering and Pulseaudio/CRAS already work well with USB. It would then only require a topology tuple (to set the dependency relationship) and some logic in SOF core to create/delet dynamic PCM devices (like USB does).

I don't think that's how USB Audio works? PCM streams are mapped to endpoints, they are statically created upon parsing of the descriptors. The streams are also handled at the kernel level when e.g. the capture path is used as a clock to let the driver how many samples it can push (implicit async)

I am not aware of any cases where PCM devices came and go. It's hard enough when a card comes and goes...

@plbossart

I have to agree with @plbossart here, this seems really difficult to make somehow logical and usable in user space. If we have a pcm, you should be able to open and play or capture from it, no matter in what ever some other pcm state is. I would think that it should be accepted that echo reference path is like an extra capture pipeline in FW (needs as many dma's etc.)...

I don't think that's how USB Audio works? PCM streams are mapped to endpoints, they are statically created upon parsing of the descriptors. The streams are also handled at the kernel level when e.g. the capture path is used as a clock to let the driver how many samples it can push (implicit async)

I'm meaning that the PCM device(s) are being created upon USB audio device insertion (as part of USB bus enumeration) and vice versa on USB removal. Pulseaudio is clever enough to switch between removable PCMs upon removal/insertion.

I am not aware of any cases where PCM devices came and go. It's hard enough when a card comes and goes...

Afaik, it's just USB, IIRC HDMI creates them even if there is no monitor. Topology also supports this today, but it's never been tested.

@plbossart

I have to agree with @plbossart here, this seems really difficult to make somehow logical and usable in user space. If we have a pcm, you should be able to open and play or capture from it, no matter in what ever some other pcm state is.

Not really, device state is important. Users cannot be guaranteed to perform device ops if the device is not ready. There are lots of standard errnos for this sort of thing and in this case ENODATA is appropriate if the PCM has no data.

I would think that it should be accepted that echo reference path is like an extra capture pipeline in FW (needs as many dma's etc.)...

It is a "capture" pipeline, but it's data is only meaningful if and only if the playback PCM is running.

The only things that work are
a) creating/removal of devices as part of a plug event
b) static creation of devices based on topology but a jack event that makes them visible/usable (e.g. headset capture or HDMI playback)

I am not aware of PCM devices coming and going otherwise.

@lgirdwood

@lgirdwood There's long history with linking PCM's via snd_pcm_link:
https://www.alsa-project.org/alsa-doc/alsa-lib/group___p_c_m.html#gac6c33091b049985baa6466e8fe93917e

... but describing semantics of the PCM connections, is a much harder problem. Existing infra indeed assumes the application has out-of-band information about PCM-to-PCM links, and when and how to use them. E.g. the

To me it sounds most reasonable approach is to upstream "discovery infra" for a specific usage like echo reference which @cujomalainey mentioned as a UCM token. A token like echo reference is something applications can act on. This will require specific support from CRAS and Pulseaudio.

But PCMs should still work (to the extent possible) also with apps that do not use UCM. E.g. with snd_pcm_link, there is a category of hardware that needs to be started linked to have synchronized playback, but these PCMs do still work without snd_pcm_link calls (you will just have channels drifting). With echo reference, if PCM0 works also with PCM1 not opened, and full support for PCM1 requires an application that understands the echo reference UCM tokens (and opens PCM0 and PCM1 in sync), I think that sounds ok.

A more generic language to describe PCM dependencies is possible, but it doesn't really seem feasible to push this to the wider ALSA application space. The use-cases are just too fringe to get app developer interest, and based on e.g. adoption of snd_pcm_link (that has been there for 15+ years), not worth the effort.

@perexg

There are a lot of possible ways to solve PCM dependence in the kernel, system libraries (alsa-lib, tinyalsa) and userspace (sound servers). I'd like to suggest we initially look at the low hanging fruit here in order to enable more community involvement in improving support throughout the stack.

I've updated proposal in the feature given comments. @perexg any comments on above feature ?

@tiwai

Adding @tiwai to the discussion. This is basically a loopback / slave device requirement. My opinions:

kernel should return an error when the stream is not available as proposed (ENODATA or ENOLINK etc.), if the PCM device is not able to give the silence
the stream presence should be detected using a control (ALSA control API)
UCM value should describe which control can be used for the detection and / or we may standardize this
We should think about an interface to pass the "master" stream parameters... We do something similar in snd-aloop (loopback driver) - the PCM parameters are exported using the control interface, so the application can get the correct parameters for the slave stream. Add this extension to UCM, too.

Speaking of HD-audio HDMI: the dynamic PCM device creation was one of proposals at the time we had worked on HDA HDMI implementation. But this approach wasn't taken because it didn't work with PA: PA expects the devices belonging to the card don't change once after probed. USB-audio does the whole card-level hotplug / unplug, hence it works with the dynamic creation and removal, as suggested in the above. But the device-level hotplug / unplug isn't recognized by PA.

Also, HDA HDMI couldn't return an error at the open (also at the prepare, IIRC) during disconnection, because the device probe at PA startup failed otherwise. The current code returns an error at trigger (-EPIPE like a normal xrun) when started with disconnection, and this was the compromise after experiments. Maybe returning another error code might work nowadays, but this has to be carefully tested through different applications and backends.

I think we can see things separately: some feasible low-hanging fruits and the rest.
A good low-hanging fruit would be the new error code for the insufficient setup. This can be applied to a HDMI device, too, for example. But, as mentioned in the above, we have to test this carefully before actually applying.

For the rest, the PCM stream dependency, there are indeed many different ways to solve it. And it's difficult to generalize. The most generic solution would be an exposed graph like media controller, but I guess it's an overkill for a simple one-level dependency like this case. For the limited one-level dependency, the in-kernel representation as Jaroslav suggested would work. But, if UCM is mandatory in anyway, this looks likely superfluous, too.

[RFC] arecord: add support for -ENODATA handling

@lgirdwood

@lgirdwood @perexg @cujomalainey et al I've confirmed -ENODATA is passed to user-space correctly through kernel driver (I only had to modify FW, no changes to kernel were needed).

With unpatched application, trying to record from echo reference PCM without the master device running, one gets this:
arecord: pcm_read:2103: read error: No data available

I also made RFC patches to alsa-utils and alsa-lib (documentation) only:

alsa-utils:
kv2019i/alsa-utils@a2ba541

alsa-lib:
kv2019i/alsa-lib@87b2981

With these patches, arecord on the echo ref device will reuse aplay.c suspend logic to retry starting the device until -ENODATA is no longer returned. This can happen if the dependent PCMs are started in parallel, or after resume from system suspend. The behaviour with arecord looks like this for the echo reference device (initially the master device is not running so it has to retry a few times:

$ ./arecord -Dhw:0,4 -r48000 -c2 -fS16_LE -v /dev/null
Recording WAVE '/dev/null' : Signed 16 bit Little Endian, Rate 48000 Hz, Stereo
[....]
Suspended. Trying resume. Failed. Restarting stream. Done.
Suspended. Trying resume. Failed. Restarting stream. Done.
####

The "Suspended. Trying.." message come from aplay.c suspend() handle I reused in this RFC patch.

If the generic approach is ok, I can proceed to submit more proper patches for alsa-utils and alsa-lib. I run the test with a custom SOF FW that is forced to return -ENODATA -- proper support for -ENODATA is not yet in master).

@chiang831

@chiang831 apologies, missed this update until Kai posted.

Hi Liam, could you please point me to an example of the token defined in
SND_USE_CASE_MOD_ECHO_REF modifier ?
I could not find the doc in https://www.alsa-project.org/alsa-doc/alsa-lib/group__ucm.html or the existing UCM using that token.

@juimonen can you help here, is this in the UCM2 updates ?

In CRAS use case, CRAS needs to know the echo reference device name for a speaker device.
Ideally, after scanning through UCM file when the card struct and device structs are first added to CRAS, CRAS should know this information.

We use EchoReference variable in SectionDevice of speaker device to specify the associated echo reference device name on the same sound card.

There are two usages for this information:
When enabling speaker device, also enable the associated echo reference device.
When disabling speaker device, also disable the associated echo reference device.
( code in
https://chromium-review.googlesource.com/c/chromiumos/third_party/adhd/+/1180783/7/cras/src/server/cras_iodev_list.c#381 )
Note that CRAS usage here does not start by opening echo reference device first, and see the error code later.
CRAS only opens echo reference device when it knows that there is an associated echo reference device for the speaker device to be opened. So I think the error code passing described earlier in the thread is not used by CRAS.

ok, this is fine. You will need to check your XRUN handling as the echo ref may return -ENODATA if it's starved long enough during XRUN handling on the primary PCM.

If the token in SND_USE_CASE_MOD_ECHO_REF modifier can provide such relation, it is easy for CRAS to replace EchoReference and adopt SND_USE_CASE_MOD_ECHO_REF. CRAS can scan through the UCM file and add this information to speaker device struct.
Hope this explanation helps!

Yes it does thanks.

@kv2019i

@kv2019i ok, that's great - did the kernel need any update for PCM to PCM topology ?

@juimonen

@juimonen can you help here, is this in the UCM2 updates ?

I think the ECHO_REF token is an oldie in alsa-lib, but I've not seen it any conf actually using it...
@perexg you know any conf using that or other examples?

@chiang831

Yeah, @chiang831 @juimonen , I don't think ECHO_REF has been used. It's a modifier in UCM, so in theory it would go like this (modifying the examples in use-case.h):

Set verb to SND_USE_CASE_VERB_VOICECALL (for a call)
Call snd_use_case_get("CapturePCM") and snd_use_case_get("PlaybackPCM") to get ALSA PCM names for speaker and microphone
Set verb to SND_USE_CASE_VERB_VOICECALL (for a call)
Set modifier to SND_USE_CASE_MOD_ECHO_REF
Call snd_use_case_get("CapturePCM") to get the reference PCM name

But it's a bit clumsy as echo reference is really part of the main use-case, so having to do a separate step with the modified does not seem right. Any idea @perexg why it was defined like this and any known users (I did a quick search and could not find any)?

So to me it would be better to add a new variable, e.g. "EchoReferencePCM". When you set up a verb, you could then query for "PlaybackPCM", "CapturePCM" and "EchoReferencePCM", and have all information you need to set up correct devices. If "EchoReferencePCM" is empty (e.g. routed to headset), then you know to disable. FYI to @cujomalainey and @lgirdwood as well

Note that UCM was designed mostly by Liam, so I am a bit surprised, why the ball is on my side (Liam's commit introduced this). The modifiers is something which was introduced in the beginning. It's true that they are not used to this time.

Modifier is the runtime thing. The idea is to allow the additional configuration when you activate such functions like the echo audio reference. You can just think about this like an extra device for the special purpose which can be activated when the "master" device is running. You can define own Enable/DisableSequence and values. Thus, just work with it like with another device like:

SectionModifier."Echo Reference" {
  SupportedDevices [
     "Mic"
  ]
  EnableSequence [
  ]
  DisableSequence [
  ]
  Values {
     CapturePCM "hw:${CardID},99"
  }
}

And the UCM query string is: "CapturePCM/Echo Reference/HiFi" . Don't forget to call _enamod to activate the modifier (run the EnableSequence) before use and _dismod (DisableSequence).

@perexg

@perexg wrote:

Note that UCM was designed mostly by Liam, so I am a bit surprised, why the ball is on my side (Liam's commit introduced this).

:) My bad completely, I just noted you mentioned ECHO_REF modifier in this thread, and didn't bother to checkup the code history at all. Thanks a lot for the reply. So this exists, but just hasn't been used yet. Please others take a look -- do we just go with the existing echo reference modifier, or add something new.

@perexg

@perexg I think the ball was passed in your direction since you mentioned using the modifier first :-)
In general, this was the intention of the modifier. It was to cover any extra optional configuration needed for the use case that would not fit elsewhere. It would work as intended in this case.

Thanks for the example and explanation.
I am still concerned on the usage and whether it can scale.
I listed my question below with the example by Jaroslav:

How to associate this echo reference modifier (and device) to the output device.
On one sound card, there are one speaker and one headphone output device.
Usually we know echo reference device should be associated with the speaker device.
Can we use a Value to make it explicitly defined ?
This can save the futile call to open echo reference device when headphone is in use.
The alternative solution is to always assume echo reference is available when there is output, and handle the error code returned. But this seems not ideal and we really prefer to specify the device usage explicitly in UCM.
How to scale when there are multiple output devices and one echo reference devices.
A possible use case is like:
There are two output devices. One for low latency, and one for high latency.
SOF will mix them and output to speaker.
There is one echo reference device for the speaker output.
So this echo reference device is associated with both low latency output and high latency output.
If we have EchoReference value in output device, we can put this value in low latency output and high latency output, pointing to the same echo reference device.
But if we define the relationship in SectionModifier of echo reference using a Value (like 1.), we will need to use a list to hold low latency output and high latency output. Which is still a feasible solution although a little complicated.
What if there are multiple echo reference devices ?
There seems to be no common use case like this but theoretically this can happen.
For example, on a sound card, there are speaker device and line out device.
Both devices contain some post processing effect so echo references for both devices are needed respectively (and might be supported on the sound card). We might need a new naming for SectionModifier."Echo Reference" to support multiple echo references.
What does SupportedDevices mean ? In the example, there is a device name "Mic" here. Not sure what its usage is.
Kai mentioned that "If "EchoReferencePCM" is empty (e.g. routed to headset), then you know to disable". Could you please explain a bit more how to achieve this ? Speaker device and headset device can output at the same time. Can we still query "EchoReferencePCM" to get the echo reference device name for speaker in this case ? (Note that this still have the drawback that the relationship is defined implicitly)

SectionModifier."Echo Reference" {   ----> Question 3: How to handle the case where there are multiple echo reference devices ?
  SupportedDevices [
     "Mic"  ----> Question 4: What does this name mean ?
  ]
  EnableSequence [
  ]
  DisableSequence [
  ]
  Values {
     CapturePCM "hw:${CardID},99"
      -----> Question 1, 2: Can we add another value to explicitly associate this echo reference device to its corresponding output device (may be multiple output devices) ?
  }
}

Thanks for taking a look!

The associated master devices are listed in SupportedDevices [] list. In the above example, the "Mic" device has the "Echo Reference" modifier. Multiple devices can be listed here, but it would also mean, that only one of those devices can be used at runtime (the SupportedDevice/ConflictingDevice lists should describe that correctly).

For multiple instances, I think that we should follow the recent device naming update (add indexes to the end of the name). So if the name is without index, one instance exists, if it begins with 1, there may be more instances like: "Echo Reference 1" "Echo Reference 2" etc. in the continuous index range. The application should determine which modifier belongs to the device using SupportedDevices/ConflictingDevices lists.

@perexg

Hi @perexg, thanks for the explanation.
I am not quite sure why a "Mic" device is associated with a "Echo Reference" modifier.
Maybe you mean the echo reference device is named "Mic."?

But since multiple devices can be put into SupportedDevice list, I guess my use case can be like:

SectionModifier."Echo Reference" {
  SupportedDevices [
     "Speaker",
  ]
  EnableSequence [
  ]
  DisableSequence [
  ]
  Values {
     CapturePCM "hw:${CardID},99"
  }
}

If we can use this format, then CRAS can identify that there is a echo reference device at hw:${CardID},99 to be used, and it is associated with "Speaker" device.
By "associated" it means this echo reference device provides what is played on speaker.
I think that would work. We can make related changes in CRAS to align the usage between CRAS and upstream.

Hi @cujomalainey, there are not many boards ChromeOS boards having echo reference device. It should not be too complicated to make the change and verify all of them.

Thanks!

@perexg

Thanks @perexg and @chiang831 . So it seems the existing UCM infra actually can cover all the needs. As it is already there, I'd say let's start by using it.

No activity in 5 years, closing. Please reopen if still relevant.

Navigation Menu

Repository navigation

[FEATURE] Dependent PCMs - Userspace handling #2564

10 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[FEATURE] Dependent PCMs - Userspace handling #2564

Description

Activity

lgirdwood commented on Mar 17, 2020

plbossart commented on Mar 17, 2020

lgirdwood commented on Mar 17, 2020

cujomalainey commented on Mar 17, 2020

plbossart commented on Mar 17, 2020

juimonen commented on Mar 17, 2020

lgirdwood commented on Mar 17, 2020

lgirdwood commented on Mar 17, 2020

plbossart commented on Mar 17, 2020

kv2019i commented on Mar 18, 2020

lgirdwood commented on Mar 18, 2020

perexg commented on Mar 18, 2020

tiwai commented on Mar 19, 2020

tiwai commented on Mar 19, 2020

10 remaining items

kv2019i commented on Apr 2, 2020

lgirdwood commented on Apr 2, 2020

lgirdwood commented on Apr 2, 2020

juimonen commented on Apr 3, 2020

kv2019i commented on Apr 3, 2020

perexg commented on Apr 3, 2020

kv2019i commented on Apr 3, 2020

lgirdwood commented on Apr 3, 2020

chiang831 commented on Apr 3, 2020

perexg commented on Apr 3, 2020

chiang831 commented on Apr 4, 2020

kv2019i commented on Apr 6, 2020

kv2019i commented on Dec 17, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions