Improve IPC flood error reports #2204

Activity

1. plbossart

   assigned

   ranj063

   on Jun 16, 2020
2. ranj063 commented on Jun 22, 2020

   

   ranj063

   on Jun 22, 2020

   Collaborator

   More actions

   @plbossart I'm not sure this is a kernel issue. I think this is a problem in the test script. The error should have been reported for the IPC flood test as a failure because the device couldnt suspend after the IPC flood test completed. And of course its a different issue as to why the device failed to idle in the first place.
3. fredoh9 commented on Jun 22, 2020

   

   fredoh9

   on Jun 22, 2020

   Last edited by fredoh9

   Collaborator

   More actions

   @plbossart @ranj063 I will assign this to myself.

   First ipc_flood_test didn't go through. But it returned PASS. Looks I need to udpate IPC Flood test script.

   2020-06-16 14:49:33 UTC [REMOTE_INFO] Dumping test logs!
   2020-06-16 14:49:33 UTC [REMOTE_INFO] Test Result: PASS!
   

   in good case,

   2020-06-16 14:48:00 UTC [REMOTE_INFO] Dumping test logs!
   [   70.091602] sof-audio-pci 0000:00:1f.3: IPC Flood count: 10000, Avg response time: 50780ns
   [   70.091603] sof-audio-pci 0000:00:1f.3: Max response time: 452093ns
   [   70.091603] sof-audio-pci 0000:00:1f.3: Min response time: 32028ns
   2020-06-16 14:48:00 UTC [REMOTE_INFO] Test Result: PASS!
   

   2nd, the device failed to idle. Need to debug why so.

   kernel: [ 96.248808] sof-audio-pci 0000:00:1f.3: error: debugfs write failed to idle -16
   
4. fredoh9

   assigned

   fredoh9

   and unassigned

   ranj063

   on Jun 22, 2020
5. fredoh9 commented on Jun 23, 2020

   

   fredoh9

   on Jun 23, 2020

   Collaborator

   More actions

   Correction: above logs difference is due to dynamic log verbosity difference. We need to match same verbosity for all CI devices. I will submit a issue.

   This error is treated to ignore for a long time.
   [ 96.248808] sof-audio-pci 0000:00:1f.3: error: debugfs write failed to idle -16

   __pm_runtime_suspend() return -16, Device or resource busy. @ranj063 @plbossart not sure we should worry about this error. Looks we don't have any issue with this error.

   For one failure in CI report: https://sof-ci.01.org/linuxpr/PR2201/build3950/devicetest/
   I don't think it is related with debugfs write failure.

   16:Write error: -32,Broken pipe
   2020-06-16 14:54:40 UTC [REMOTE_ERROR] speaker test failed
   
6. fredoh9 commented on Jul 24, 2020

   

   fredoh9

   on Jul 24, 2020

   Last edited by fredoh9

   Collaborator

   More actions

   Sorry for lazy update on this issue. This issue have evolved to dynamic message filter and log filtering/scanning.

   1. Same sof-dyndbg.conf is deployed to all test DUTs, this is done and up and running
   2. Remove sof-audio before scanning kernel log, Include errors outside sof-audio before scanning dmesg for errors, ignore only specific known issues sof-test#290 (+1 approved)
   3. Giving a warning when sof-dyndbg.conf is not available in /etc/modprobe.d/, env-check: add sof-dyndbg.conf file check sof-test#260 (waiting for approval)

   I'm planning to close this when 2nd item is merged.
7. fredoh9 commented on Jul 27, 2020

   

   fredoh9

   on Jul 27, 2020

   Collaborator

   More actions

   thesofproject/sof-test#290 is merged. I will monitor test results for next a few days.
8. fredoh9

   closed this as completedon Jul 27, 2020