Repository navigation
Submitting tasks #300
Description
Hello, is there a way to improve the speed at which tasks are submitted across nodes in a pool?
I have a cluster with 1024 cores but it takes about 03:30 to submit all tasks. There's very little data from my local R session which needs to be uploaded, so I don't think it's a bandwidth issue. I'm finding that 75% of the time to complete each iteration of my model is accounted for by submitting the tasks (merging is much, much faster) so any performance increases here will make a big difference to my workflow.
Additionally, will the speed at which tasks are submitted increase/decrease depending on where you're based and where your cluster is located? i.e. will it take longer for jobs to submit in 'South Central US' if you're based somewhere in Europe?
Thanks.
Activity
brnleehng commentedon Sep 5, 2018
Hi Simon,
For submitting a large number of tasks, I'll look into increasing the throughput of task submission similar to the other Azure Batch SDKs.
https://docs.microsoft.com/en-us/azure/batch/large-number-tasks#increase-throughput-of-task-submission
Thanks,
Brian
simon-tarr commentedon Sep 10, 2018
Hi Brian Hoang (@brnleehng), thanks for the reply. It sounds like it could be a great enhancement to the package.
simon-tarr commentedon Sep 14, 2018
Hi Brian,
I was just wondering how complex this is to implement? Is it possible to get a rough ETA on when this could be expected to go live, assuming it's possible to increase the throughput in the first place? In addition, is this a back-end update that will just start working after submitting a job, rather than needing a package update within doAzureParallel?
Many thanks,
Simon
darylmsft commentedon Sep 14, 2018
We should be able to get the task factory to call AddTaskCollection and handle its complexity by the end of Oct. That will get about 100x improvement.
simon-tarr commentedon Sep 15, 2018
Wow, 100x is a serious improvement.
I guess this leaves me to figure out whether waiting until the end of October and then running all my tasks will be quicker than letting everything chug along slowly as things currently stand. Currently about 75% of the time each iteration takes to run can be explained by the time taken to submit tasks but if this gets improved 100x, there will essentially be no waiting. Decisions...
simon-tarr commentedon Sep 27, 2018
simon-tarr commentedon Nov 5, 2018
Hello, any news on the status of this feature? Thanks!
Pullarg commentedon Nov 7, 2018
would be excellent to know, running 800k jobs 12 hours and still submitting, at this point spinning up a 64core machine and using doparallel would be better
simon-tarr commentedon Nov 8, 2018
I reckon half of the $20,000 I've spent over the last 6 weeks has been on waiting for tasks to submit...