Skip to content
This repository was archived by the owner on Jan 15, 2026. It is now read-only.
This repository was archived by the owner on Jan 15, 2026. It is now read-only.

Idle SSH connections can trigger unwanted activity on the target #646

Open

Description

@douglas-raillard-arm

Idle SSH connections maintained for concurrent command execution can end up creating unwanted activity on the target.
This typically is very small amount, but x50 and with "help" from the kernel scheduler, this ends up being boosted and create frequency increase spikes.

The issue was observed with OpenSSH on the target, less so (or not at all) using dropbear.

As a result, we need to introduce an API to close unused connections, in the form of a context manager that people can use if they are running some experiments that needs as little disturbance as possible.

Here is a graph of utilization spikes resulting from this activity.
image

TODO:

  • add a context manager to silence SSH as much as possible (generic, should be exposed as target level and then possibly a no-op for some connection types).
  • figure out if we can make SSH a bit more silent (I suspect some sort of watchdog mechanism is triggering activity at regular interval even if the connection is unsed)

Activity

  1. marcbonnici commented on Sep 6, 2023

    @marcbonnici
    Collaborator

    This is an interesting finding. Out of curiosity do you know if there are any default keepalive settings enabled on the server side (I'm assuming this is server side as this is not observed with dropbear rather than a paramiko default config)?

  2. douglas-raillard-arm commented on Sep 18, 2023

    @douglas-raillard-arm
    CollaboratorAuthor

    I could not find anything enabled by default, but the default is probably irrelevant since distros typically ship a non-default config. The kinds of keepalive I found are:

    • keepalive in the ssh protocol, implemented by the servers and clients.
    • TCP layer keepalive, dealt with by the libc/kernel.

    We don't know exactly what is happening here

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions