dsub version: v0-5-2
I’m using dsub for a job where each task writes a batch of output files (names unknown at launch) and a semaphore so I don't retry the same task over and over once it's successfully completed. To me this seemed like a good use for --output-recursive (for my large number of output files), and then I write a simple --output for my semaphore file so I can --skip. E.g.:
--output-recursive OUTPUT_DIR=gs://bucket/path/output/
--output semaphore_file=gs://bucket/path/semaphore/batch_001.done
The destination bucket contains millions of files. I didn't really think anything of this because my access patterns make this work just fine, but I was noticing very bad delocalization performance towards the end of the job. Going over the code, I think that the issue is that dsub seems to delocalize --output-recursive with gsutil rsync -r. Once the destination output prefix has hundreds of thousands of files from previous batches, later tasks with only a small number of local output files can end up running rsync against a very large existing destination prefix. At least, this is my working diagnosis for why delocalization becomes slower over time in this job.
This can actually cause more severe problems, because if the semaphore delocalizes first, and if the delocalization of my handful of output files takes many minutes, then the pre-emptible termination probability becomes substantial. This can cause the semaphore to be written, indicating that the job is done, but the output files may not have delocalized. I appreciate that this is always the risk (since delocalization order is not formalized as being --output-recursive before --output), but I expected copying over a few hundred kilobytes to take seconds instead of tens of minutes and so the actual risk was much higher than I had anticipated. (Arguably, I do think it could make sense to consider formalizing that the --output-recursive steps should run before --output steps, too, to make semaphore-style approaches safer.)
Ultimately, I am not sure that I understand the upside of rsync here. It seems like the useful behavior of --output-recursive is "copy everything under this local directory to this remote prefix" which could just use recursive copy semantics instead of rsync without the risk of incurring this cost. (Also possible that I'm not reading the dsub code correctly...)
dsub version: v0-5-2I’m using dsub for a job where each task writes a batch of output files (names unknown at launch) and a semaphore so I don't retry the same task over and over once it's successfully completed. To me this seemed like a good use for
--output-recursive(for my large number of output files), and then I write a simple--outputfor my semaphore file so I can--skip. E.g.:The destination bucket contains millions of files. I didn't really think anything of this because my access patterns make this work just fine, but I was noticing very bad delocalization performance towards the end of the job. Going over the code, I think that the issue is that dsub seems to delocalize
--output-recursivewithgsutil rsync -r. Once the destination output prefix has hundreds of thousands of files from previous batches, later tasks with only a small number of local output files can end up running rsync against a very large existing destination prefix. At least, this is my working diagnosis for why delocalization becomes slower over time in this job.This can actually cause more severe problems, because if the semaphore delocalizes first, and if the delocalization of my handful of output files takes many minutes, then the pre-emptible termination probability becomes substantial. This can cause the semaphore to be written, indicating that the job is done, but the output files may not have delocalized. I appreciate that this is always the risk (since delocalization order is not formalized as being
--output-recursivebefore--output), but I expected copying over a few hundred kilobytes to take seconds instead of tens of minutes and so the actual risk was much higher than I had anticipated. (Arguably, I do think it could make sense to consider formalizing that the--output-recursivesteps should run before--outputsteps, too, to make semaphore-style approaches safer.)Ultimately, I am not sure that I understand the upside of
rsynchere. It seems like the useful behavior of--output-recursiveis "copy everything under this local directory to this remote prefix" which could just use recursive copy semantics instead ofrsyncwithout the risk of incurring this cost. (Also possible that I'm not reading thedsubcode correctly...)