Submission on LSF fails (Sanger cluster) #523

Activity

1. hategan commented on Jul 2, 2025

   

   hategan

   on Jul 2, 2025

   Collaborator

   More actions

   Thank you for your report. I do not have access to an LSF machine, so I might need some help in figuring this out. What happens when you pass -o /dev/null directly to bsub? In other words, does the following work or fail:

   bsub -o /dev/null generated_script.sh
2. jan-engelmann commented on Jul 3, 2025

   

   jan-engelmann

   on Jul 3, 2025

   Author

   More actions

   Thank you for looking into this @hategan!

   bsub -o /dev/null generated_script.sh

   works.

   How can I specify this using the psij package?

   Also, I'm struggling to set the right job specs.
   How would you translate the following command for Psij?

   bsub -o log.txt -G GROUP_NAME -M1000 -R"select[mem>1000] rusage[mem=1000] span[hosts=1]" -n 8 -q QUEUE_NAME /bin/date

   I tried as below but couldnt quite get it right

   import logging
   
   from psij import Job, JobSpec, JobExecutor, ResourceSpecV1
   
   ex = JobExecutor.get_instance("lsf")
   
   spec = JobSpec(
       executable="/bin/date",
       resources=ResourceSpecV1(
           node_count=1,
           processes_per_node=1,
           memory=1_024 * 1024,
       ),
   )
   spec.attributes.queue_name = "normal"
   spec.attributes.custom_attributes = {}
   spec.attributes.custom_attributes['lsf.R'] = "select[mem>1024] rusage[mem=1024] span[hosts=1]"
   
   job = Job(spec)
   
   self = ex
   
   self._ensure_work_dir()
   
   self._check_job(job)
   
   context = self._create_script_context(job)
   
   # assumes job ids are unique
   submit_file_path = self.work_directory / (job.id + ".job")
   with submit_file_path.open("w") as submit_file:
       self.generate_submit_script(job, context, submit_file)
   
   

   this outputs:

   #!/bin/bash
   #BSUB -env all
   #BSUB -nnodes 1
   #BSUB -n 1
   #BSUB -M 1024KB
   #BSUB -W 0:10
   #BSUB -q "normal"
   #BSUB -R "'select[mem>1024] rusage[mem=1024] span[hosts=1]'"
   #BSUB -e /dev/null
   #BSUB -o /dev/null
   PSIJ_NODEFILE="$LSB_HOSTS"
   export PSIJ_NODEFILE
   exec &>> "/nfs/users/nfs_j/USER/.psij/work/lsf/$LSB_JOBID.out"
   echo "$?" > "/nfs/users/nfs_j/USER/.psij/work/lsf/$LSB_JOBID.ec"
   
3. hategan commented on Jul 3, 2025

   

   hategan

   on Jul 3, 2025

   Collaborator

   More actions

   Thank you for looking into this @hategan!

   bsub -o /dev/null generated_script.sh

   works.

   That is odd. What is the contents of generated_script.sh?

   How can I specify this using the psij package?

   That is specified through the #BSUB -o /dev/null line(s). The fact that somehow it doesn't work is a bit puzzling.

   Also, I'm struggling to set the right job specs. How would you translate the following command for Psij?

   This might get a bit verbose, so it might be useful to submit a separate issue in the future if you have a question/issue different than the original one.

   bsub -o log.txt -G GROUP_NAME -M1000 -R"select[mem>1000] rusage[mem=1000] span[hosts=1]" -n 8 -q QUEUE_NAME /bin/date

   I tried as below but couldnt quite get it right

   import logging
   
   from psij import Job, JobSpec, JobExecutor, ResourceSpecV1
   
   ex = JobExecutor.get_instance("lsf")
   
   spec = JobSpec(
       executable="/bin/date",
       resources=ResourceSpecV1(
           node_count=1,
           processes_per_node=1,
           memory=1_024 * 1024,
       ),
   )
   spec.attributes.queue_name = "normal"
   spec.attributes.custom_attributes = {}
   spec.attributes.custom_attributes['lsf.R'] = "select[mem>1024] rusage[mem=1024] span[hosts=1]"
   
   job = Job(spec)
   
   self = ex
   
   self._ensure_work_dir()
   
   self._check_job(job)
   
   context = self._create_script_context(job)
   
   # assumes job ids are unique
   submit_file_path = self.work_directory / (job.id + ".job")
   with submit_file_path.open("w") as submit_file:
       self.generate_submit_script(job, context, submit_file)
   

   this outputs:

   #!/bin/bash
   #BSUB -env all
   #BSUB -nnodes 1
   #BSUB -n 1
   #BSUB -M 1024KB
   #BSUB -W 0:10
   #BSUB -q "normal"
   #BSUB -R "'select[mem>1024] rusage[mem=1024] span[hosts=1]'"
   #BSUB -e /dev/null
   #BSUB -o /dev/null
   PSIJ_NODEFILE="$LSB_HOSTS"
   export PSIJ_NODEFILE
   exec &>> "/nfs/users/nfs_j/USER/.psij/work/lsf/$LSB_JOBID.out"
   echo "$?" > "/nfs/users/nfs_j/USER/.psij/work/lsf/$LSB_JOBID.ec"
   

   I am not sure I fully follow. What is it that is different from what you were expecting?
   Also, the launch line seems to be missing. Did you remove it by accident when reformatting the output?