feature request: PSIJ submission via ssh #342

Activity

1. andre-merzky

   self-assigned this

   on Feb 3, 2023
2. andre-merzky

   added

   enhancementNew feature or requestNew feature or request

   on Feb 3, 2023
3. hategan commented on Feb 3, 2023

   

   hategan

   on Feb 3, 2023

   Collaborator

   More actions

   Question: are we talking about running a remote command through SSH or remote submission (e.g., to SLURM) via SSH?
4. andre-merzky commented on Feb 3, 2023

   

   andre-merzky

   on Feb 3, 2023

   CollaboratorAuthor

   More actions

   Speaking from the RCT perspective: we ultimately need the latter (remote submission to batch), but I currently would favor the approach to use the ssh executor to bootstrap and launch a PSIJ Rest or ZMQ service on the target machine and then proxy all executor requests to that service. So the ssh executor would need to only run commands really. As such, it would be basically a duplication of the local fork executor doing the same things over paramiko.

   Any thoughts on this?

   FWIW, we are happy with psij over ZMQ and REST at the moment. The ssh bootstrapping is a nice to have and I want to sink some time into this over the next weeks, thus the issue to track this activity, but it's not a showstopper for us ATM.
5. hategan commented on Feb 3, 2023

   

   hategan

   on Feb 3, 2023

   Collaborator

   More actions

   Speaking from the RCT perspective: we ultimately need the latter (remote submission to batch), but I currently would favor the approach to use the ssh executor to bootstrap and launch a PSIJ Rest or ZMQ service on the target machine and then proxy all executor requests to that service. So the ssh executor would need to only run commands really. As such, it would be basically a duplication of the local fork executor doing the same things over paramiko.

   Any thoughts on this?

   FWIW, we are happy with psij over ZMQ and REST at the moment. The ssh bootstrapping is a nice to have and I want to sink some time into this over the next weeks, thus the issue to track this activity, but it's not a showstopper for us ATM.

   That's precisely what Coasters do/did in Swift/K, and it was a nice turn-key solution that required no interaction from the user (except a one-time setting up of credentials).
   Given that we both had tried this and seen it to work well, I'm in favor of it.
   It was also useful for setting up poor man's clusters.

   The Swift SSH provider was a bit more complex than local, since it used on-demand SSH channels over cached connections to avoid initiating fresh connections for each job. We may not want to do that in a first iteration, but it did prove more robust for high-throughput situations.

   👍React with 👍1Reacted by Andre Merzky
6. andre-merzky commented on Mar 31, 2023

   

   andre-merzky

   on Mar 31, 2023

   CollaboratorAuthor

   More actions

   The Swift SSH provider was a bit more complex than local, since it used on-demand SSH channels over cached connections to avoid initiating fresh connections for each job. We may not want to do that in a first iteration, but it did prove more robust for high-throughput situations.

   That reflects what SAGA does did with surprising precision. Ack on both: this being potentially much faster and resource friendly, but also more complex and with low priority (for now).