Add initial support for i.MX8 SOF driver by dbaluta · Pull Request #1048 · thesofproject/linux

dbaluta · 2019-06-18T16:44:19Z

Hello @plbossart @ranj063 @lgirdwood
This is version 4 of the patches that add the initial DSP Linux driver for i.MX8 into SOF.

This version only contains SOF related patches and fixes review comments from @plbossart and Lucas Stach from arm mailing list:

https://lore.kernel.org/linux-arm-kernel/20190723084104.12639-1-daniel.baluta@nxp.com/

This patches would require IMX DSP interface driver (which I hope to be soon merged by Shawn Guo - imx maintainer).

https://lkml.org/lkml/2019/8/1/260

Now there is the tricky part: how do we bring all these patches together because SOF Linux driver
depends on IMX DSP. I have handled this through KConfig depenencies. So, one cannot select SOF Linux driver compilation without imx dsp.

Hello @lgirdwood @ranj063 @plbossart
This is version 3 of the patch to add the initial DSP Linux driver for i.MX8 into SOF.

For compilation you need to use the following kconfig fragment:
thesofproject/kconfig#13

I have added all patches related to DSP here in order to be able to do a proper compilation.

these patches are under review with arm64/imx community
ASoC: SOF: Add DT DSP device support
dt-bindings: dsp: fsl: Add DSP core binding support
arm64: dts: imx8qxp: Add DSP DT node

these patches need to be reviewed by SOF community
ASoC: SOF: Add DT DSP device support
ASoC: SOF: topology: Add dummy support for i.MX8 DAIs
ASoC: SOF: imx: Add i.MX8 HW support

this patch is under review:
firmware: imx: Add DSP IPC protocol interface

Patches below are already accepted into Shawn's Guo tree for i.mx8 arm.

clk: imx8: Add DSP related clocks
firmware: imx: scu-pd: Add IRQSTR_DSP PD range
firmware: imx: scu-pd: Add mu13 b side PD range
firmware: imx: scu-pd: Rename mu PD range to mu_a
arm64: dts: imx8qxp: sort LSIO subsystem devices
arm64: dts: imx8qxp: Add lsio_mu13 node

Version 2:
Hello veryone @ranj063 @plbossart @lgirdwood. This is version 2 of the PR trying to add basic support for i.MX8.

What changed since v1:

i tried to address all your comments. We now support only no-codec mode
i removed the patches related to dsp ipc / dts / pd because I have sent them separately to corresponding maintainers.

All patches are here, together with some patches for serial console to work.
https://github.com/dbaluta/linux/commits/topic/sof-dev

All this patches were sent to review, as follows:

power domains related changes:

https://lkml.org/lkml/2019/7/18/252

DSP IPC interface

https://lkml.org/lkml/2019/7/18/123

DSP DT node

@plbossart @lgirdwood what do you think would be the appropriate way to have this reviewed with the alsa community? I guess starting the patches to alsa-devel would be the right way to do?

Version 1:
This patch series add basic support for i.MX8 SOF driver.

First 3 patches introduce DSP IPC protocol driver (review with community ongoing - https://lkml.org/lkml/2019/6/14/165)

505a33c dt-bindings: arm: fsl: Add DSP IPC binding support
00a5e98 firmware: imx: Add DSP IPC protocol driver
d2a410a mailbox: imx: Clear GIEn bit at shutdown

Next 3 patches add DT nodes corresponding to DSP. Review will be started with community soon.

15ccaaa arm64: dts: imx8qxp: Add DSP DT node
d4dfab8 arm64: dts: imx8qxp: Add DSP MU node
76760c1 clk: imx8qxp: Add DSP clocks

Next 2 patches are small SOF cleanups that can be merged as they are:

7310ddc ASoC: SOF: loader: Dont ignore SRAM block types
23159b1 ASoC: SOF: loader: Use the BAR provided by FW

Last 3 patches are the actual support for i.MX8 SOF driver.

70dd8da ASoC: SOF: Add DT DSP device support
5b28a5e ASoC: SOF: topology: Add dummy support for i.MX8 DAIs
b79c490 ASoC: SOF: imx: Add i.MX8 HW support

Would like to see some review for last 5 patches.

plbossart

Thanks @dbaluta, this is really cool.
I have a couple of comments below, mostly to make sure the license choice is intentional and on the machine driver load. This will likely need a bit more work to reconcile the DT and ACPI ways of dealing with the machine driver.
But yes, nice work!

plbossart

Looks good @dbaluta. Can you fix the Makefile conflicts (see comment below) and let's merge.
Thanks!

Add support for device tree based SOF DSP devices. Signed-off-by: Daniel Baluta <daniel.baluta@nxp.com>

plbossart

Thanks @dbaluta
I'd like a second reviewer, I guess it'll be one of the usual suspects @ranj063 @kv2019i :-)

ranj063 · 2019-08-09T22:31:48Z

+static int imx8_run(struct snd_sof_dev *sdev)
+{
+	int ret;
+	struct imx8_priv *dsp_priv = (struct imx8_priv *)sdev->private;


@dbaluta only minor but for the sake of consistency can you please switch the 2 statements so the struct goes first.

ranj063 · 2019-08-09T22:34:03Z

+static int imx8_remove(struct snd_sof_dev *sdev)
+{
+	int i;
+	struct imx8_priv *priv = (struct imx8_priv *)sdev->private;


ranj063 · 2019-08-09T22:41:38Z

@dbaluta just wondering if you forgot to include a header file? where are the functions like imx_ring_doorbell() defined?

Add support for the audio DSP hardware found on NXP i.MX8 platform. Signed-off-by: Daniel Baluta <daniel.baluta@nxp.com>

Add dummy support for SAI/ESAI digital audio interface IPs found on i.MX8 boards. Signed-off-by: Daniel Baluta <daniel.baluta@nxp.com>

dbaluta · 2019-08-11T13:15:30Z

@ranj063 imx_ring_doorbell prototype is defined in #include <linux/firmware/imx/dsp.h>. This header is already included at the top of imx8.c file

dsp.h file will reach linux tree via imx8 maintainer. The patch was already send and reviewed here:

https://patchwork.kernel.org/patch/11048657/

Now, theoretically this PR doesn't compile on its own. It needs the patch above. Anyhow, the imx8 SOF driver has a Kconfig dependency on IMX DSP so practically we won't be able to select it for compilation until all changes get merged into the same tree.

dbaluta · 2019-08-12T07:23:08Z

SOFCI TEST

plbossart · 2019-08-12T14:21:39Z

ok let's merge and start including i.MX support!

dbaluta · 2019-08-12T15:25:17Z

@plbossart thanks! IMX DSP protocol interface was accepted through Shawn's tree:

https://lkml.org/lkml/2019/8/12/605

We are still waiting for DTS file review and hopefully with your next rebase of sof-dev we will have a working imx initial support!

Thanks again for support and review!

plbossart · 2019-08-12T15:38:13Z

On 8/12/19 10:25 AM, Daniel Baluta wrote: @plbossart <https://github.com/plbossart> thanks! IMX DSP protocol interface was accepted through Shawn's tree [1]

Do you want to cherry-pick the required patches and submit them as well for the SOF tree? Or do we wait for after the 5.4 merge window where all of this will be available?

plbossart · 2019-08-12T16:01:15Z

@dbaluta I just did a rebase on upstream, do you want me to send the patches directly to Mark?
see the topic/sof-dev-rebase branch.

dbaluta · 2019-08-12T16:16:13Z

@plbossart yes, please send them to Mark. Also, for patches merged via other trees would be nice to cherry pick them if it isnt too stressful for you. Will provide with a pointer to linux-next once the patches are there.

plbossart · 2019-08-12T17:06:35Z

On 8/12/19 11:16 AM, Daniel Baluta wrote: @plbossart <https://github.com/plbossart> yes, please send them to Mark. Also, for patches merged via other trees would be nice to cherry pick them if it isnt too stressful for you. Will provide with a pointer to linux-next once the patches are there.

if you want to cherry-pick on top of topic/sof-dev, and push a PR then I can just merge the new PR and we are done.

dbaluta · 2019-08-12T18:59:11Z

@plbossart thanks! Done.

#1125

What is the difference between topic/sof-dev and topic/sof-dev/rebase ?

Another question: are you also shipping SOF with 4.19 kernel?

plbossart · 2019-08-12T21:14:10Z

What is the difference between topic/sof-dev and topic/sof-dev/rebase ?

hint: the name tells you the difference :-) the -rebase version doesn't keep the history and will be rebased on top of the latest upstream code (Mark Brown's for-next + Takashi's for-next + soundWire stuff). the regular version will in contrast be updated with a set of regular merges from upstream. Doing merges is nice for developers, and keeps history. The flip side is that it makes it really hard to figure out the differences with upstream, and gitk goes all over the place, while in the -rebase case you can literally see the patches to be queued in linear order (and we can even squash/reorder for simplicity). The flow is everything takes place in topic/sof-dev. When we do upstream merges, we update topic/sof-dev-rebase as well, make sure the two branches are exactly identical, then I upstream from topic/sof-dev-rebase.

Another question: are you also shipping SOF with 4.19 kernel?

There are SOF-based products using 4.19 and 5.0, but the kernels are no longer directly maintained by the SOF team. We used to maintain backports initially for Intel-based solutions but we've started to let our partners (be they internal or internal) do the work. I don't think a single team can do both upstream and downstream work at this point. And unfortunately for ARM platforms the scripts we used are probably completely useless since we essentially ignored those platforms...

Wolfram reports that his R-Car H2-based Lager board can no longer be rebooted in v5.11-rc1, as it crashes with an imprecise external abort. The issue can be reproduced on other boards (e.g. Koelsch with R-Car M2-W) too, if CONFIG_IP_PNP is disabled, and the Ethernet interface is down at reboot time: Unhandled fault: imprecise external abort (0x1406) at 0x00000000 pgd = (ptrval) [00000000] *pgd=422b6835, *pte=00000000, *ppte=00000000 Internal error: : 1406 [#1] ARM Modules linked in: CPU: 0 PID: 1105 Comm: init Tainted: G W 5.10.0-rc1-00402-ge2f016cf7751 #1048 Hardware name: Generic R-Car Gen2 (Flattened Device Tree) PC is at sh_mdio_ctrl+0x44/0x60 LR is at sh_mmd_ctrl+0x20/0x24 ... Backtrace: [<c0451f30>] (sh_mdio_ctrl) from [<c0451fd4>] (sh_mmd_ctrl+0x20/0x24) r7:0000001f r6:00000020 r5:00000002 r4:c22a1dc4 [<c0451fb4>] (sh_mmd_ctrl) from [<c044fc18>] (mdiobb_cmd+0x38/0xa8) [<c044fbe0>] (mdiobb_cmd) from [<c044feb8>] (mdiobb_read+0x58/0xdc) r9:c229f844 r8:c0c329dc r7:c221e000 r6:00000001 r5:c22a1dc4 r4:00000001 [<c044fe60>] (mdiobb_read) from [<c044c854>] (__mdiobus_read+0x74/0xe0) r7:0000001f r6:00000001 r5:c221e000 r4:c221e000 [<c044c7e0>] (__mdiobus_read) from [<c044c9d8>] (mdiobus_read+0x40/0x54) r7:0000001f r6:00000001 r5:c221e000 r4:c221e458 [<c044c998>] (mdiobus_read) from [<c044d678>] (phy_read+0x1c/0x20) r7:ffffe000 r6:c221e470 r5:00000200 r4:c229f800 [<c044d65c>] (phy_read) from [<c044d94c>] (kszphy_config_intr+0x44/0x80) [<c044d908>] (kszphy_config_intr) from [<c044694c>] (phy_disable_interrupts+0x44/0x50) r5:c229f800 r4:c229f800 [<c0446908>] (phy_disable_interrupts) from [<c0449370>] (phy_shutdown+0x18/0x1c) r5:c229f800 r4:c229f804 [<c0449358>] (phy_shutdown) from [<c040066c>] (device_shutdown+0x168/0x1f8) [<c0400504>] (device_shutdown) from [<c013de44>] (kernel_restart_prepare+0x3c/0x48) r9:c22d2000 r8:c0100264 r7:c0b0d034 r6:00000000 r5:4321fedc r4:00000000 [<c013de08>] (kernel_restart_prepare) from [<c013dee0>] (kernel_restart+0x1c/0x60) [<c013dec4>] (kernel_restart) from [<c013e1d8>] (__do_sys_reboot+0x168/0x208) r5:4321fedc r4:01234567 [<c013e070>] (__do_sys_reboot) from [<c013e2e8>] (sys_reboot+0x18/0x1c) r7:00000058 r6:00000000 r5:00000000 r4:00000000 [<c013e2d0>] (sys_reboot) from [<c0100060>] (ret_fast_syscall+0x0/0x54) As of commit e2f016c ("net: phy: add a shutdown procedure"), system reboot calls phy_disable_interrupts() during shutdown. As this happens unconditionally, the PHY registers may be accessed while the device is suspended, causing undefined behavior, which may crash the system. Fix this by wrapping the PHY bitbang accessors in the sh_eth driver by wrappers that take care of Runtime PM, to resume the device when needed. Reported-by: Wolfram Sang <wsa+renesas@sang-engineering.com> Suggested-by: Andrew Lunn <andrew@lunn.ch> Signed-off-by: Geert Uytterhoeven <geert+renesas@glider.be> Tested-by: Wolfram Sang <wsa+renesas@sang-engineering.com> Reviewed-by: Florian Fainelli <f.fainelli@gmail.com> Reviewed-by: Andrew Lunn <andrew@lunn.ch> Signed-off-by: Jakub Kicinski <kuba@kernel.org>

This will add eBPF JIT support to the 32-bit ARCv2 processors. The implementation is qualified by running the BPF tests on a Synopsys HSDK board with "ARC HS38 v2.1c at 500 MHz" as the 4-core CPU. The test_bpf.ko reports 2-10 fold improvements in execution time of its tests. For instance: test_bpf: thesofproject#33 tcpdump port 22 jited:0 704 1766 2104 PASS test_bpf: thesofproject#33 tcpdump port 22 jited:1 120 224 260 PASS test_bpf: thesofproject#141 ALU_DIV_X: 4294967295 / 4294967295 = 1 jited:0 238 PASS test_bpf: thesofproject#141 ALU_DIV_X: 4294967295 / 4294967295 = 1 jited:1 23 PASS test_bpf: thesofproject#776 JMP32_JGE_K: all ... magnitudes jited:0 2034681 PASS test_bpf: thesofproject#776 JMP32_JGE_K: all ... magnitudes jited:1 1020022 PASS Deployment and structure ------------------------ The related codes are added to "arch/arc/net": - bpf_jit.h -- The interface that a back-end translator must provide - bpf_jit_core.c -- Knows how to handle the input eBPF byte stream - bpf_jit_arcv2.c -- The back-end code that knows the translation logic The bpf_int_jit_compile() at the end of bpf_jit_core.c is the entrance to the whole process. Normally, the translation is done in one pass, namely the "normal pass". In case some relocations are not known during this pass, some data (arc_jit_data) is allocated for the next pass to come. This possible next (and last) pass is called the "extra pass". 1. Normal pass # The necessary pass 1a. Dry run # Get the whole JIT length, epilogue offset, etc. 1b. Emit phase # Allocate memory and start emitting instructions 2. Extra pass # Only needed if there are relocations to be fixed 2a. Patch relocations Support status -------------- The JIT compiler supports BPF instructions up to "cpu=v4". However, it does not yet provide support for: - Tail calls - Atomic operations - 64-bit division/remainder - BPF_PROBE_MEM* (exception table) The result of "test_bpf" test suite on an HSDK board is: hsdk-lnx# insmod test_bpf.ko test_suite=test_bpf test_bpf: Summary: 863 PASSED, 186 FAILED, [851/851 JIT'ed] All the failing test cases are due to the ones that were not JIT'ed. Categorically, they can be represented as: .-----------.------------.-------------. | test type | opcodes | # of cases | |-----------+------------+-------------| | atomic | 0xC3, 0xDB | 149 | | div64 | 0x37, 0x3F | 22 | | mod64 | 0x97, 0x9F | 15 | `-----------^------------+-------------| | (total) 186 | `-------------' Setup: build config ------------------- The following configs must be set to have a working JIT test: CONFIG_BPF_JIT=y CONFIG_BPF_JIT_ALWAYS_ON=y CONFIG_TEST_BPF=m The following options are not necessary for the tests module, but are good to have: CONFIG_DEBUG_INFO=y # prerequisite for below CONFIG_DEBUG_INFO_BTF=y # so bpftool can generate vmlinux.h CONFIG_FTRACE=y # CONFIG_BPF_SYSCALL=y # all these options lead to CONFIG_KPROBE_EVENTS=y # having CONFIG_BPF_EVENTS=y CONFIG_PERF_EVENTS=y # Some BPF programs provide data through /sys/kernel/debug: CONFIG_DEBUG_FS=y arc# mount -t debugfs debugfs /sys/kernel/debug Setup: elfutils --------------- The libdw.{so,a} library that is used by pahole for processing the final binary must come from elfutils 0.189 or newer. The support for ARCv2 [1] has been added since that version. [1] https://sourceware.org/git/?p=elfutils.git;a=commit;h=de3d46b3e7 Setup: pahole ------------- The line below in linux/scripts/Makefile.btf must be commented out: pahole-flags-$(call test-ge, $(pahole-ver), 121) += --btf_gen_floats Or else, the build will fail: $ make V=1 ... BTF .btf.vmlinux.bin.o pahole -J --btf_gen_floats \ -j --lang_exclude=rust \ --skip_encoding_btf_inconsistent_proto \ --btf_gen_optimized .tmp_vmlinux.btf Complex, interval and imaginary float types are not supported Encountered error while encoding BTF. ... BTFIDS vmlinux ./tools/bpf/resolve_btfids/resolve_btfids vmlinux libbpf: failed to find '.BTF' ELF section in vmlinux FAILED: load BTF from vmlinux: No data available This is due to the fact that the ARC toolchains generate "complex float" DIE entries in libgcc and at the moment, pahole can't handle such entries. Running the tests ----------------- host$ scp /bld/linux/lib/test_bpf.ko arc: arc # sysctl net.core.bpf_jit_enable=1 arc # insmod test_bpf.ko test_suite=test_bpf ... test_bpf: thesofproject#1048 Staggered jumps: JMP32_JSLE_X jited:1 697811 PASS test_bpf: Summary: 863 PASSED, 186 FAILED, [851/851 JIT'ed] Acknowledgments --------------- - Claudiu Zissulescu for his unwavering support - Yuriy Kolerov for testing and troubleshooting - Vladimir Isaev for the pahole workaround - Sergey Matyukevich for paving the road by adding the interpreter support Signed-off-by: Shahab Vahedi <shahab@synopsys.com> Link: https://lore.kernel.org/r/20240430145604.38592-1-list+bpf@vahedi.org Signed-off-by: Alexei Starovoitov <ast@kernel.org>

Call to ieee80211_color_collision_detection_work() needs wiphy lock to be held (see lockdep assert in cfg80211_bss_color_notify()). Not locking wiphy causes the following lockdep error: WARNING: CPU: 2 PID: 42 at net/wireless/nl80211.c:19505 cfg80211_bss_color_notify+0x1a4/0x25c Modules linked in: CPU: 2 PID: 42 Comm: kworker/u8:3 Tainted: G W 6.4.0-02327-g36c6cb260481 thesofproject#1048 Hardware name: Workqueue: phy1 ieee80211_color_collision_detection_work pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--) pc : cfg80211_bss_color_notify+0x1a4/0x25c lr : cfg80211_bss_color_notify+0x1a0/0x25c sp : ffff000002947d00 x29: ffff000002947d00 x28: ffff800008e1a000 x27: ffff000002bd4705 x26: ffff00000d034000 x25: ffff80000903cf40 x24: 0000000000000000 x23: ffff00000cb70720 x22: 0000000000800000 x21: ffff800008dfb008 x20: 000000000000008d x19: ffff00000d035fa8 x18: 0000000000000010 x17: 0000000000000001 x16: 000003564b1ce96a x15: 000d69696d057970 x14: 000000000000003b x13: 0000000000000001 x12: 0000000000040000 x11: 0000000000000001 x10: ffff80000978f9c0 x9 : ffff0000028d3174 x8 : ffff800008e30000 x7 : 0000000000000000 x6 : 0000000000000028 x5 : 000000000002f498 x4 : ffff00000d034a80 x3 : 0000000000800000 x2 : ffff800016143000 x1 : 0000000000000000 x0 : 0000000000000000 Call trace: cfg80211_bss_color_notify+0x1a4/0x25c ieee80211_color_collision_detection_work+0x20/0x118 process_one_work+0x294/0x554 worker_thread+0x70/0x440 kthread+0xf4/0xf8 ret_from_fork+0x10/0x20 irq event stamp: 77372 hardirqs last enabled at (77371): [<ffff800008a346fc>] _raw_spin_unlock_irq+0x2c/0x4c hardirqs last disabled at (77372): [<ffff800008a28754>] el1_dbg+0x20/0x48 softirqs last enabled at (77350): [<ffff8000089e120c>] batadv_send_outstanding_bcast_packet+0xb8/0x120 softirqs last disabled at (77348): [<ffff8000089e11d4>] batadv_send_outstanding_bcast_packet+0x80/0x120 The wiphy lock cannot be taken directly from color collision detection delayed work (ieee80211_color_collision_detection_work()) because this work is cancel_delayed_work_sync() under this wiphy lock causing a potential deadlock( see [0] for details). To fix that ieee80211_color_collision_detection_work() could be converted to a wiphy work and cancel_delayed_work_sync() can be simply replaced by wiphy_delayed_work_cancel() serving the same purpose under wiphy lock. This could potentially fix [1]. [0]: https://lore.kernel.org/linux-wireless/D4A40Q44OAY2.W3SIF6UEPBUN@freebox.fr/ [1]: https://lore.kernel.org/lkml/000000000000612f290618eee3e5@google.com/ Reported-by: Nicolas Escande <nescande@freebox.fr> Signed-off-by: Remi Pommarel <repk@triplefau.lt> Link: https://patch.msgid.link/20240924192805.13859-3-repk@triplefau.lt Signed-off-by: Johannes Berg <johannes.berg@intel.com>

dbaluta requested review from bardliao, juimonen, keyonjie, lgirdwood, libinyang, lyakh, plbossart, ranj063 and singalsu as code owners June 18, 2019 16:44

ranj063 reviewed Jun 18, 2019

View reviewed changes

Comment thread sound/soc/sof/topology.c Outdated

ranj063 reviewed Jun 18, 2019

View reviewed changes

Comment thread sound/soc/sof/sof-dt-dev.c

plbossart requested changes Jun 18, 2019

View reviewed changes

dbaluta requested a review from plbossart July 23, 2019 15:00

dbaluta mentioned this pull request Jul 30, 2019

imx-defconfig: Add support to build SOF for imx8 thesofproject/kconfig#13

Merged

dbaluta changed the title ~~[RFC] Add initial support for i.MX8 SOF driver~~ Add initial support for i.MX8 SOF driver Aug 7, 2019

plbossart reviewed Aug 9, 2019

View reviewed changes

Comment thread sound/soc/sof/Makefile

ASoC: SOF: Add OF DSP device support

40aa230

Add support for device tree based SOF DSP devices. Signed-off-by: Daniel Baluta <daniel.baluta@nxp.com>

plbossart previously approved these changes Aug 9, 2019

View reviewed changes

ranj063 reviewed Aug 9, 2019

View reviewed changes

dbaluta added 2 commits August 11, 2019 16:07

ASoC: SOF: imx: Add i.MX8 HW support

cf00c7c

Add support for the audio DSP hardware found on NXP i.MX8 platform. Signed-off-by: Daniel Baluta <daniel.baluta@nxp.com>

ASoC: SOF: topology: Add dummy support for i.MX8 DAIs

e39c57a

Add dummy support for SAI/ESAI digital audio interface IPs found on i.MX8 boards. Signed-off-by: Daniel Baluta <daniel.baluta@nxp.com>

dbaluta dismissed plbossart’s stale review via e39c57a August 11, 2019 13:08

plbossart approved these changes Aug 12, 2019

View reviewed changes

plbossart merged commit c9da397 into thesofproject:topic/sof-dev Aug 12, 2019

dbaluta deleted the sof-dev-3 branch August 12, 2019 15:37

Conversation

dbaluta commented Jun 18, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

plbossart left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

plbossart left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

plbossart left a comment

Choose a reason for hiding this comment

Uh oh!

ranj063 Aug 9, 2019

Choose a reason for hiding this comment

Uh oh!

dbaluta Aug 11, 2019

Choose a reason for hiding this comment

Uh oh!

ranj063 Aug 9, 2019

Choose a reason for hiding this comment

Uh oh!

dbaluta Aug 11, 2019

Choose a reason for hiding this comment

Uh oh!

ranj063 commented Aug 9, 2019

Uh oh!

dbaluta commented Aug 11, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dbaluta commented Aug 12, 2019

Uh oh!

plbossart commented Aug 12, 2019

Uh oh!

dbaluta commented Aug 12, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

plbossart commented Aug 12, 2019 via email

Uh oh!

plbossart commented Aug 12, 2019

Uh oh!

dbaluta commented Aug 12, 2019

Uh oh!

plbossart commented Aug 12, 2019 via email

Uh oh!

dbaluta commented Aug 12, 2019

Uh oh!

plbossart commented Aug 12, 2019 via email

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dbaluta commented Jun 18, 2019 •

edited

Loading

dbaluta commented Aug 11, 2019 •

edited

Loading

dbaluta commented Aug 12, 2019 •

edited

Loading