soundwire: intel: exit clock-stop mode before system suspend by plbossart · Pull Request #3006 · thesofproject/linux

plbossart · 2021-06-22T04:05:50Z

Intel validation reported an issue where the HW_RST self-clearing bit
is not cleared in hardware, which as a ripple effect creates issues
with the clock stop mode.

This happens is a specific sequence where the Intel manager is
pm_runtime suspended with the clock-stop mode enabled. During the
system suspend, we currently do nothing, which can lead to potential
issues on system resume and the following pm_runtime suspend,
depending on the hardware state.

This patch suggests a full resume if the clock-stop mode is used. This
may require extra time but will make the suspend/resume flows
completely symmetric. This also removes a race condition where we
could not access SHIM registers if the parent was suspended as
well. Resuming the link also resumes the parent by construction.

BugLink: #2606
Signed-off-by: Pierre-Louis Bossart pierre-louis.bossart@linux.intel.com

bardliao · 2021-06-22T07:43:26Z

SOFCI TEST

RanderWang · 2021-06-23T09:41:21Z

@plbossart copy your comments in #2606 (comment)

"I have a possible explanation: this problem happens when the link is already pm_runtime suspended, so we are in clock-stop
mode. But when we do a system suspend, we don't resume the link out of clock stop. And when we do a system resume, we
may be in a zombie state. It could very well be that we have to do a pm_runtime resume first, and do a full system suspend."

if we do a pm_runtime resume first, then we will find sdw controller is not in clock stop0 mode in intel_resume_runtime function for this case since sdw controller is powered off and clock stop0 state is not sustained. And it will go normal path like intel_resume for s3. I don't see any difference between intel_resume vs intel_resume_runtime + intel_resume

bardliao · 2021-06-23T09:52:48Z

@plbossart It is interesting that rt711_dev_system_suspend was not invoked when I ran "sudo rtcwake -m mem -s 5" command. Did I miss something?

plbossart · 2021-06-23T13:14:24Z

@plbossart It is interesting that rt711_dev_system_suspend was not invoked when I ran "sudo rtcwake -m mem -s 5" command. Did I miss something?

that was the case on the earlier version but that's no longer the case in the update, please double-check.

plbossart · 2021-06-23T13:17:15Z

@plbossart copy your comments in #2606 (comment)

"I have a possible explanation: this problem happens when the link is already pm_runtime suspended, so we are in clock-stop
mode. But when we do a system suspend, we don't resume the link out of clock stop. And when we do a system resume, we
may be in a zombie state. It could very well be that we have to do a pm_runtime resume first, and do a full system suspend."

if we do a pm_runtime resume first, then we will find sdw controller is not in clock stop0 mode in intel_resume_runtime function for this case since sdw controller is powered off and clock stop0 state is not sustained. And it will go normal path like intel_resume for s3. I don't see any difference between intel_resume vs intel_resume_runtime + intel_resume

@RanderWang the change is on the suspend path. The proposal is to bring the devices back to full power, i.e. exit the clock stop mode before doing the system suspend.
We don't know what the state of the hardware is when we do a system resume, and we obviously have real issues with the HW_RST bit that isn't cleared, so when in doubt the proposal is to avoid any dependency on a state that is related to clock stop.

plbossart · 2021-06-23T18:38:47Z

@bardliao @RanderWang I am not fully happy with this PR, there's got to be a way to force the devices to resume at the bus level, maybe with a .prepare step.

plbossart · 2021-06-23T21:16:48Z

@bardliao @RanderWang I think the update "should" provide a better solution, if only I found why the sdw_slave_suspend() callback doesn't seem to be called... Gah.

bardliao · 2021-06-24T06:01:59Z

@plbossart It is interesting that rt711_dev_system_suspend was not invoked when I ran "sudo rtcwake -m mem -s 5" command. Did I miss something?

that was the case on the earlier version but that's no longer the case in the update, please double-check.

@plbossart I double checked it with your latest update. I can only see rt711 sdw:1:025d:0711:00: plb resume start, but no "plb system suspend start or plb sdw_slave_suspend start
dmesg_pr3006_0624.txt
I have to do some investigation.

bardliao · 2021-06-24T09:24:57Z

@bardliao @RanderWang I think the update "should" provide a better solution, if only I found why the sdw_slave_suspend() callback doesn't seem to be called... Gah.

@plbossart The reason why sdw_slave_suspend() isn't called is because dev->power.direct_complete is true.
dev->power.direct_complete is assigned at device_prepare. It is true due to ret > 0 where ret = callback(dev);.
We have dev->pm_domain for the sdw Slave devices so the callback is dev->pm_domain->ops.prepare which is acpi_subsys_prepare.
And acpi_dev_needs_resume will return false for pm_runtime_suspended(dev) is true.
So sdw_slave_suspend() and rt711_dev_system_suspend() will be called if you do system suspend in playback.
ie.

Start a playback
Use another terminal to run the sudo rtcwake -m mem -s 5 command.

plbossart · 2021-06-24T23:36:00Z

@bardliao @RanderWang I think the update "should" provide a better solution, if only I found why the sdw_slave_suspend() callback doesn't seem to be called... Gah.

@plbossart The reason why sdw_slave_suspend() isn't called is because dev->power.direct_complete is true.
dev->power.direct_complete is assigned at device_prepare. It is true due to ret > 0 where ret = callback(dev);.
We have dev->pm_domain for the sdw Slave devices so the callback is dev->pm_domain->ops.prepare which is acpi_subsys_prepare.
And acpi_dev_needs_resume will return false for pm_runtime_suspended(dev) is true.
So sdw_slave_suspend() and rt711_dev_system_suspend() will be called if you do system suspend in playback.
ie.

Start a playback

Use another terminal to run the sudo rtcwake -m mem -s 5 command.

Nice analysis @bardliao, you saved me a lot of time here.

I think we can just set the NO_DIRECT_COMPLETE flag and prevent all this

dev->power.direct_complete = state.event == PM_EVENT_SUSPEND &&
		(ret > 0 || dev->power.no_pm_callbacks) &&
		!dev_pm_test_driver_flags(dev, DPM_FLAG_NO_DIRECT_COMPLETE);

if we set this flag the result is logically false. Testing this now.

plbossart · 2021-06-25T00:03:54Z

@bardliao @RanderWang I think the update "should" provide a better solution, if only I found why the sdw_slave_suspend() callback doesn't seem to be called... Gah.

@plbossart The reason why sdw_slave_suspend() isn't called is because dev->power.direct_complete is true.
dev->power.direct_complete is assigned at device_prepare. It is true due to ret > 0 where ret = callback(dev);.
We have dev->pm_domain for the sdw Slave devices so the callback is dev->pm_domain->ops.prepare which is acpi_subsys_prepare.
And acpi_dev_needs_resume will return false for pm_runtime_suspended(dev) is true.
So sdw_slave_suspend() and rt711_dev_system_suspend() will be called if you do system suspend in playback.
ie.

Start a playback

Use another terminal to run the sudo rtcwake -m mem -s 5 command.

Nice analysis @bardliao, you saved me a lot of time here.

I think we can just set the NO_DIRECT_COMPLETE flag and prevent all this
dev->power.direct_complete = state.event == PM_EVENT_SUSPEND &&
		(ret > 0 || dev->power.no_pm_callbacks) &&
		!dev_pm_test_driver_flags(dev, DPM_FLAG_NO_DIRECT_COMPLETE); 
if we set this flag the result is logically false. Testing this now.

No luck @bardliao , with this change the direct_complete is false but still we don't call sdw_slave_suspend...

bardliao · 2021-06-25T11:56:45Z

No luck @bardliao , with this change the direct_complete is false but still we don't call sdw_slave_suspend...

@plbossart I must be wrong with my last comment. sdw_slave_suspend should not be called by sdw devices. See __device_suspend line 1653 to line 1681.
dev->pm_domain is the highest priority and dev->bus->pm is the lowest priority. Slave devices has dev->pm_domain which is ACPI domain. So dev->bus->pm will never be called.
I don't know if change pm priority make sense or not. If yes, we can add pm ops in sdw_slave_type. If no, we may need to call pm_runtime_resume from each codec's prepare pm ops. BTW, I think it seems be more reasonable to call pm_runtime_resume from prepare instead of suspend.

plbossart · 2021-06-25T16:09:45Z

Good explanation @bardliao, I had no idea how power domains worked ;-)

It seems the issue is that "rtcwake -m mem" translates to S0 state, which means no resume:

linux/drivers/acpi/device_pm.c

Line 999 in bf96c09

if (sys_target == ACPI_STATE_S0)

Maybe that means that instead of forcing the codec to resume to work-around the clock stop issue, we need to deal with the intel_suspend differently and keep the clock stop mode and wakes in S0. @RanderWang I wonder if what we did assumed a transition to S3.

bardliao · 2021-06-28T11:07:45Z

SOFCI TEST

bardliao · 2021-06-28T12:26:12Z

@plbossart I like this version since all changes are in intel specific code. But we have to test if (pm_runtime_enabled(dev)) before calling pm_runtime_resume(dev). That should be able to fix the pr-device-test issue.
For example

static int intel_resume_child_device(struct device *dev, void *data)
{
        int ret = 0;

        if (pm_runtime_enabled(dev))
                ret = pm_runtime_resume(dev);

        return ret;
}

plbossart · 2021-06-28T15:29:21Z

@plbossart I like this version since all changes are in intel specific code. But we have to test if (pm_runtime_enabled(dev)) before calling pm_runtime_resume(dev). That should be able to fix the pr-device-test issue.
For example
static int intel_resume_child_device(struct device *dev, void *data)
{
        int ret = 0;

        if (pm_runtime_enabled(dev))
                ret = pm_runtime_resume(dev);

        return ret;
}

@bardliao are you referring to this error?

[ 1037.444421] kernel: PM: device_prepare(): intel_pm_prepare+0x0/0x60 [soundwire_intel] returns -13
[ 1037.444439] kernel: soundwire_intel soundwire_intel.link.0: PM: not prepared for power transition: code -13

I don't see it in my local tests, and I don't understand why it happens either. If something disables pm_runtime(), then it's not clear to me how we would resume the device... Oh well, this needs more work obviously.

plbossart · 2021-06-28T17:16:37Z

update with one test to discard 'ghost' codec devices and trace in case of pm_runtime_resume errors

plbossart · 2021-06-28T23:21:44Z

Amazing how many bad configs this PR exposes. Now we have a case where we suspend/resume even when we are not using the SoundWire IP https://sof-ci.01.org/linuxpr/PR3006/build5945/devicetest/?model=TGLH_RVP_HDA&testcase=check-suspend-resume-5

[  821.104721] kernel: soundwire sdw:0:025d:0701:00: intel_resume_child_device: skipping device, no probed driver
[  821.104729] kernel: rt700 sdw:0:025d:0700:00: intel_resume_child_device: skipping device, never detected on bus
[  821.104739] kernel: soundwire sdw:1:025d:0701:00: intel_resume_child_device: skipping device, no probed driver
[  821.104742] kernel: rt700 sdw:1:025d:0700:00: intel_resume_child_device: skipping device, never detected on bus
[  821.105343] kernel: soundwire_intel soundwire_intel.link.1: intel_link_power_down: Unbalanced power-up/down calls
[  821.105346] kernel: soundwire_intel soundwire_intel.link.1: intel_link_power_down: powering down all links
[  821.106775] kernel: soundwire_intel soundwire_intel.link.1: intel_link_power_down: could not power down link
[  821.106778] kernel: soundwire_intel soundwire_intel.link.1: Link power down failed: -11

added a test to only do PM/wake sequences if we started the link.

bardliao · 2021-06-29T06:05:46Z

Amazing how many bad configs this PR exposes. Now we have a case where we suspend/resume even when we are not using the SoundWire IP https://sof-ci.01.org/linuxpr/PR3006/build5945/devicetest/?model=TGLH_RVP_HDA&testcase=check-suspend-resume-5

[  821.104721] kernel: soundwire sdw:0:025d:0701:00: intel_resume_child_device: skipping device, no probed driver
[  821.104729] kernel: rt700 sdw:0:025d:0700:00: intel_resume_child_device: skipping device, never detected on bus
[  821.104739] kernel: soundwire sdw:1:025d:0701:00: intel_resume_child_device: skipping device, no probed driver
[  821.104742] kernel: rt700 sdw:1:025d:0700:00: intel_resume_child_device: skipping device, never detected on bus
[  821.105343] kernel: soundwire_intel soundwire_intel.link.1: intel_link_power_down: Unbalanced power-up/down calls
[  821.105346] kernel: soundwire_intel soundwire_intel.link.1: intel_link_power_down: powering down all links
[  821.106775] kernel: soundwire_intel soundwire_intel.link.1: intel_link_power_down: could not power down link
[  821.106778] kernel: soundwire_intel soundwire_intel.link.1: Link power down failed: -11

added a test to only do PM/wake sequences if we started the link.

@plbossart This didn't fix the issue. The problem is that the link is enabled but all of its child are 'ghost'. And pm runtime will not be enabled for the 'ghost' devices. As a result, soundwire link device will not be resumed in the prepare process. My proposal is that we can check if we can find any 'real' child device before calling intel_resume_child_device().
See bardliao@fdbde71

plbossart · 2021-07-15T14:34:47Z

Complete failure on Intel test 5283, on TGLU_SKU0A32_SDCA we see random power_down/power_up sequences leading to a bus clash. Gah.

[  170.199758] kernel: soundwire_intel soundwire_intel.link.2: intel_link_power_down: powering down all links
[  170.199771] kernel: soundwire_intel soundwire_intel.link.1: intel_link_power_up: powering up all links
[  170.199771] kernel: soundwire_intel soundwire_intel.link.1: intel_link_power_up: first link up, programming SYNCPRD
[  170.200118] kernel: rt1316-sdca sdw:1:025d:1316:01: sdw_modify_slave_status: initializing enumeration and init completion for Slave 1
[  170.200971] kernel: soundwire_intel soundwire_intel.link.1: Slave status change: 0x2
[  170.200975] kernel: soundwire sdw-master-1: Slave attached, programming device number
[  170.201141] kernel: soundwire sdw-master-1: SDW Slave Addr: 31025d131601
[  170.201142] kernel: soundwire sdw-master-1: SDW Slave class_id 0x01, mfg_id 0x025d, part_id 0x1316, unique_id 0x1, version 0x3
[  170.201143] kernel: soundwire sdw-master-1: Slave already registered, reusing dev_num:1
[  170.201370] kernel: soundwire_intel soundwire_intel.link.1: Msg ignored for Slave 0
[  170.201371] kernel: soundwire sdw-master-1: No more devices to enumerate
[  170.201395] kernel: soundwire_intel soundwire_intel.link.1: Slave status change: 0x21
[  170.201397] kernel: rt1316-sdca sdw:1:025d:1316:01: sdw_modify_slave_status: signaling enumeration completion for Slave 1
[  170.201573] kernel: rt1316-sdca sdw:1:025d:1316:01: Configured bus base 1, scale 3, mclk 19200000, curr_freq 4800000
[  170.214595] kernel: rt1316-sdca sdw:1:025d:1316:01: rt1316_io_init hw_init complete
[  170.214596] kernel: rt1316-sdca sdw:1:025d:1316:01: sdw_handle_slave_status: signaling initialization completion for Slave 1
[  170.214624] kernel: soundwire_intel soundwire_intel.link.1: Slave status change: 0x40
[  170.224398] kernel: soundwire_intel soundwire_intel.link.1: intel_link_power_down: powering down all links
[  170.224405] kernel: soundwire_intel soundwire_intel.link.0: intel_link_power_up: powering up all links
[  170.224406] kernel: soundwire_intel soundwire_intel.link.0: intel_link_power_up: first link up, programming SYNCPRD
[  170.224460] kernel: soundwire_intel soundwire_intel.link.1: Bus clash for control word

ranj063 · 2021-07-15T16:40:06Z

Complete failure on Intel test 5283, on TGLU_SKU0A32_SDCA we see random power_down/power_up sequences leading to a bus clash. Gah.

@plbossart I think doing the resume for the codecs in their prepare/suspend routine sychronously might be a better option. Worth trying I think

plbossart · 2021-07-15T16:54:14Z

Complete failure on Intel test 5283, on TGLU_SKU0A32_SDCA we see random power_down/power_up sequences leading to a bus clash. Gah.

@plbossart I think doing the resume for the codecs in their prepare/suspend routine sychronously might be a better option. Worth trying I think

It'd be horrible, we only need to do this when the clock-stop mode is used, and that's a property of the parent. Not to mention that we would push an Intel-specific requirement to ALL codecs, that would be problematic if a codec is also used on a non-Intel product.

The topologies used for sof-tgl-rt711-rt1308 and sof-tgl-max98357a-rt5682 are almost equivalent. Conditionally add support for the amplifier reference dailink and BT offload, so that this topology can be used as is on Google Volteer. This will help root-cause suspend/resume issues we see only on TGL_RVP_SDW Related kernel PR: thesofproject/linux#3006 Signed-off-by: Pierre-Louis Bossart <pierre-louis.bossart@linux.intel.com>

ranj063 · 2021-07-15T18:26:40Z

It'd be horrible, we only need to do this when the clock-stop mode is used, and that's a property of the parent. Not to mention that we would push an Intel-specific requirement to ALL codecs, that would be problematic if a codec is also used on a non-Intel product.

@plbossart how about adding a flag to the child device that can be set from the parent to indicate if it should resume or not?

plbossart · 2021-07-15T18:34:30Z

that was my initial PR @ranj063, but it still requires codec modifications. I still want to get to the bottom of the 2 issues we found, already fixed one issue on CML and one to go on TGL

plbossart · 2021-07-16T11:59:57Z

@bardliao we still have a race condition somewhere that's exposed with the changes in this PR.

See
dmesg.txt, there is a sequence where we power-off link1, but still get an interrupt for link1 that throws a bus clash/parity error interrupt.

[ 5104.524004] kernel: soundwire_intel soundwire_intel.link.1: intel_suspend: start plb
[ 5104.524543] kernel: soundwire_intel soundwire_intel.link.1: intel_link_power_down: powering down all links
[ 5104.524549] kernel: soundwire_intel soundwire_intel.link.1: intel_suspend: done plb

At this point link1.link_up == false

[ 5104.524552] kernel: soundwire_intel soundwire_intel.link.0: intel_resume_runtime: start plb
[ 5104.524553] kernel: soundwire_intel soundwire_intel.link.0: intel_link_power_up: powering up all links
[ 5104.524554] kernel: soundwire_intel soundwire_intel.link.0: intel_link_power_up: first link up, programming SYNCPRD

But we have interrupts that are only handled when link1.link_up == true

[ 5104.524629] kernel: soundwire_intel soundwire_intel.link.1: Parity error
[ 5104.524639] kernel: soundwire_intel soundwire_intel.link.1: Bus clash for control word

the code in the power_down sequence is weird

	mutex_unlock(sdw->link_res->shim_lock);

	if (ret < 0) {
		dev_err(sdw->cdns.dev, "%s: could not power down link\n", __func__);

		return ret;
	}

	sdw->cdns.link_up = false;	
        return 0;

Do you remember if this is intentional? we added this code in commit 4a17c44,
wondering if we need to move the link_up assignment under the mutex_lock protection - though I still don't see how this would change the interrupt handling.

Very odd sequence indeed.

plbossart · 2021-07-16T18:48:38Z

@bardliao @RanderWang with the update on the last commit to protect the link_up flag, I don't see any errors any more:

https://sof-ci.sh.intel.com/#/result/planresultdetail/5332

I'll run more tests over the week-end with thousands of cycles but that last commit does seem to have an effect.

On some platforms with an external HDaudio codec, the DSDT reports the presence of SoundWire devices. Pin-mux restrictions and board reworkds usually prevent coexistence between the two types of links, let's prevent unnecessary operations from starting. In the case of a single iDISP codec being detected, we still start the links even if no SoundWire machine configuration was detected, so that we can double-check what the hardware is and add the missing configuration if applicable. Signed-off-by: Pierre-Louis Bossart <pierre-louis.bossart@linux.intel.com>

The power down sequence sets the link_up flag as false outside of the mutex_lock. This is potentially unsafe. In additional the flow in that sequence can be improved by first testing if the link was powered, setting the link_up flag as false and proceeding with the power down. In case the CPA bits cannot be cleared, we only flag an error since we cannot deal with interrupts any longer. Signed-off-by: Pierre-Louis Bossart <pierre-louis.bossart@linux.intel.com>

On some HDaudio platforms, SoundWire devices are described in the DSDT but never used. This patch adds a boolean status flag to skip all suspend/resume/wake sequences for this configuration. Signed-off-by: Pierre-Louis Bossart <pierre-louis.bossart@linux.intel.com>

Intel validation reported an issue where the HW_RST self-clearing bit is not cleared in hardware, which as a ripple effect creates issues with the clock stop mode. This happens is a specific sequence where the Intel manager is pm_runtime suspended with the clock-stop mode enabled. During the system suspend, we currently do nothing, which can lead to potential issues on system resume and the following pm_runtime suspend, depending on the hardware state. This patch suggests a full resume (parent+child devices) if the clock-stop mode is used. This may require extra time but will make the suspend/resume flows completely symmetric. This also removes a race condition where we could not access SHIM registers if the parent was suspended as well. Resuming the link also resumes the parent by construction. BugLink: thesofproject#2606 Signed-off-by: Pierre-Louis Bossart <pierre-louis.bossart@linux.intel.com>

Since we've introduced a .prepare callback that brings all devices to full power on S state transitions if the clock-stop mode was used, we no longer need to special-case when the device was pm_runtime suspended. Signed-off-by: Pierre-Louis Bossart <pierre-louis.bossart@linux.intel.com>

The topologies used for sof-tgl-rt711-rt1308 and sof-tgl-max98357a-rt5682 are almost equivalent. Conditionally add support for the amplifier reference dailink and BT offload, so that this topology can be used as is on Google Volteer. This will help root-cause suspend/resume issues we see only on TGL_RVP_SDW Related kernel PR: thesofproject/linux#3006 Signed-off-by: Pierre-Louis Bossart <pierre-louis.bossart@linux.intel.com>

bardliao · 2021-07-19T10:34:19Z

Do you remember if this is intentional? we added this code in commit 4a17c44,
wondering if we need to move the link_up assignment under the mutex_lock protection - though I still don't see how this would change the interrupt handling.

I am not sure if it is intentional. But I think the idea was that we use mutex to protect the HW process and set sdw->cdns.link_up = false; when the the link is physically down. But I agree with you that we should set sdw->cdns.link_up = false; first and don't set it back to true if CPA bit is not cleared since we don't want to handle the interrupt right after intel_link_power_down is called.

plbossart · 2021-07-19T15:22:00Z

@bardliao @RanderWang @ranj063 I ran a bunch of tests and don't see any regression with the last patches. Let's try and merge this with your reviews and test further.

ranj063

LGTM @plbossart.

plbossart requested review from RanderWang, bardliao and ranj063 June 22, 2021 04:06

plbossart mentioned this pull request Jun 22, 2021

[BUG][SDW][CML] clock stop prepare issue - Speaker cannot work after suspend/resume #2606

Closed

plbossart force-pushed the sdw/fix-suspend branch 2 times, most recently from 95238b3 to 59ffdf7 Compare June 22, 2021 20:34

plbossart force-pushed the sdw/fix-suspend branch from 59ffdf7 to 924c989 Compare June 23, 2021 21:16

plbossart force-pushed the sdw/fix-suspend branch from 924c989 to 11ef23d Compare June 25, 2021 00:03

plbossart force-pushed the sdw/fix-suspend branch from 11ef23d to 608a8b1 Compare June 25, 2021 17:18

plbossart force-pushed the sdw/fix-suspend branch from 608a8b1 to 7b4adbf Compare June 28, 2021 17:15

plbossart force-pushed the sdw/fix-suspend branch from 7b4adbf to fb28014 Compare June 28, 2021 21:00

plbossart dismissed stale reviews from RanderWang, bardliao, and ranj063 via 84ac174 July 15, 2021 19:24

plbossart force-pushed the sdw/fix-suspend branch from f2202c5 to 84ac174 Compare July 15, 2021 19:24

plbossart force-pushed the sdw/fix-suspend branch from 84ac174 to 52e7d41 Compare July 16, 2021 13:34

plbossart added 5 commits July 16, 2021 16:31

plbossart force-pushed the sdw/fix-suspend branch from 52e7d41 to b07c6d7 Compare July 16, 2021 21:42

plbossart requested review from RanderWang, bardliao and ranj063 July 19, 2021 15:20

ranj063 approved these changes Jul 19, 2021

View reviewed changes

bardliao approved these changes Jul 20, 2021

View reviewed changes

plbossart merged commit d54b780 into thesofproject:topic/sof-dev Jul 20, 2021

plbossart mentioned this pull request Jul 20, 2021

[SoundWire] intel_resume: MCP_CONTROL_HW_RST not cleared at iteration (dev_err logs) #3012

Open

Conversation

plbossart commented Jun 22, 2021

Uh oh!

bardliao commented Jun 22, 2021

Uh oh!

RanderWang commented Jun 23, 2021

Uh oh!

bardliao commented Jun 23, 2021

Uh oh!

plbossart commented Jun 23, 2021

Uh oh!

plbossart commented Jun 23, 2021

Uh oh!

plbossart commented Jun 23, 2021

Uh oh!

plbossart commented Jun 23, 2021

Uh oh!

bardliao commented Jun 24, 2021

Uh oh!

bardliao commented Jun 24, 2021

Uh oh!

plbossart commented Jun 24, 2021

Uh oh!

plbossart commented Jun 25, 2021

Uh oh!

bardliao commented Jun 25, 2021

Uh oh!

plbossart commented Jun 25, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bardliao commented Jun 28, 2021

Uh oh!

bardliao commented Jun 28, 2021

Uh oh!

plbossart commented Jun 28, 2021

Uh oh!

plbossart commented Jun 28, 2021

Uh oh!

plbossart commented Jun 28, 2021

Uh oh!

bardliao commented Jun 29, 2021

Uh oh!

plbossart commented Jul 15, 2021

Uh oh!

ranj063 commented Jul 15, 2021

Uh oh!

plbossart commented Jul 15, 2021

Uh oh!

ranj063 commented Jul 15, 2021

Uh oh!

plbossart commented Jul 15, 2021

Uh oh!

plbossart commented Jul 16, 2021

Uh oh!

plbossart commented Jul 16, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bardliao commented Jul 19, 2021

Uh oh!

plbossart commented Jul 19, 2021

Uh oh!

ranj063 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

plbossart commented Jun 25, 2021 •

edited

Loading

plbossart commented Jul 16, 2021 •

edited

Loading