polling for jobs removed from the queue does not properly mark them as completed #401

Activity

1. Andrew-S-Rosen commented on Aug 3, 2023

   

   Andrew-S-Rosen

   on Aug 3, 2023

   Last edited by Andrew-S-Rosen

   Contributor

   More actions

   I can now confirm that this is indeed an issue.

   On Perlmutter at NERSC, I ran the following code-block to submit a simple toy job.

   from psij import Job, JobAttributes, JobExecutor, JobSpec, ResourceSpecV1
   import time
   
   job_executor = JobExecutor.get_instance("slurm")
   job = Job(
       JobSpec(
           name="test",
           executable="/bin/date",
           resources=ResourceSpecV1(node_count = 1),
           attributes=JobAttributes(project_name = 'matgen', custom_attributes = {'slurm.constraint': 'cpu', 'slurm.qos': 'debug'}),
           launcher="single",
       )
   )
   job_executor.submit(job)
   native_id = job.native_id
   print(native_id)

   I then manually monitored the queue (i.e. with squeue) until the job finished and was no longer visible in the squeue history.

   Then I ran the following block:

   from psij import JobExecutor, Job
   
   job_executor = JobExecutor.get_instance("slurm")
   job = Job()
   job_executor.attach(job, "13203806") # native ID from above
   print(job.status.state)

   It returned NEW every time I printed the state. It also reported an error in logging like what follows, but this did not hinder the return of job.status.state as NEW.

   Job 651f97ae-c467-4c4d-9edd-f16bf2371d89: failed clean submit script: [Errno 2] No such file or directory: '/global/homes/r/rosen/.psij/work/slurm/651f97ae-c467-4c4d-9edd-f16bf2371d89.job'
   

   I was never able to get it to return COMPLETED. Unfortunately, the Slurm squeue job states are cleared very quickly on Perlmutter and so this makes it impractical to retrieve a status of COMPLETED when using .attach (unless you happen to get lucky).
2. hategan commented on Aug 10, 2023

   

   hategan

   on Aug 10, 2023

   Last edited by hategan

   CollaboratorAuthor

   More actions

   I am getting the error above, and that should be fixed (clarification: we should fix that, it's not fixed yet), but if I add a wait() or sleep(3) after attach, the job shows as COMPLETED. So I think the main issue goes back to the fact that it takes a bit for the job status to be updated after attach().
3. hategan commented on Aug 10, 2023

   

   hategan

   on Aug 10, 2023

   CollaboratorAuthor

   More actions

   PR #403 fixes the "failed to clean submit script" error.
4. hategan commented on Sep 12, 2023

   

   hategan

   on Sep 12, 2023

   CollaboratorAuthor

   More actions

   @Andrew-S-Rosen, can I close this?
5. Andrew-S-Rosen commented on Sep 12, 2023

   

   Andrew-S-Rosen

   on Sep 12, 2023

   Contributor

   More actions

   @hategan: Yes, thank you for asking. Go ahead and close the issue.
6. hategan commented on Sep 12, 2023

   

   hategan

   on Sep 12, 2023

   CollaboratorAuthor

   More actions

   Thank you!

   For reference, the "failed to clean submit script" error was fixed by #403 and the COMPLETED state should be reachable with the requisite wait() after attach().
7. hategan

   closed this as completedon Sep 12, 2023