{"id":839,"date":"2013-04-12T15:47:20","date_gmt":"2013-04-12T13:47:20","guid":{"rendered":"http:\/\/oldblogs.uct.ac.za\/blog\/big-bytes\/2013\/04\/12\/array-jobs"},"modified":"2015-08-25T10:17:16","modified_gmt":"2015-08-25T08:17:16","slug":"array-jobs","status":"publish","type":"post","link":"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/04\/12\/array-jobs\/","title":{"rendered":"Array Jobs"},"content":{"rendered":"Recently we had been given a challenge by a student who needed to analyze thousands of input files with an application but found it cumbersome to submit each input file individually.\r\n

\r\nWe started looking at a \"for loop\" to run through the input files with the PBS directive \"#PBS -v TASKS\".
\r\nfor ((i=0; i < TASKS; i++)); do
\r\ncommand < input.$i
\r\ndone <\/strong>

\r\n.. and exporting the TASKS variable at CLI with a value equal to the number of inputs needed to be analyzed. The problem with this \"for loop\" was that it would run through each input at a time and not execute all of them in parallel. To get around this problem you have to add a ampersand (&) to the end of the command you are executing and a \"wait\" option as seen below.
\r\nfor ((i=0; i < TASKS; i++)); do
\r\ncommand < input.$i &
\r\ndone
\r\nwait <\/strong>
\r\n
\r\nThe above will execute nicely but it doesn't scale and will discover another stumbling block. Issuing 20 input files will effectively use 20 concurrent CPUs but should you submit 1000 input files you will be limited to the number of cores available on the cluster. If the cluster only has 500 cores available, it will cause the job to be rejected. Then we discovered that PBS have directives for these special cases. Enter PBS directive \"#PBS -t\" and \"${PBS_ARRAYID}\".
\r\n
\r\n#PBS -N inputs
\r\n#PBS -l nodes=1:series600:ppn=1
\r\n#PBS -q UCTlong
\r\ncd \/home\/username\/application\/
\r\n.\/command < \/home\/username\/application\/inputs\/input.${PBS_ARRAYID}<\/strong>
\r\n

\r\nThe job submission script would look something like the above. To submit to the cluster you would use \" qsub -t 0-100 job.sh \" with 0-100 being the range of input files with their respective extentions input.0, input.1, input.2 etc etc .... input.100 in a directory. The \" -t 0-100\" will parse to the PBS_ARRAYID and 100 jobs will be spawned on the cluster with some running and some being queued. To checkup on the status of a array job execute \" qstat -t \"","protected":false},"excerpt":{"rendered":"

Recently we had been given a challenge by a student who needed to analyze thousands of input files with an application but found it cumbersome to submit each input file individually. <\/p>\n

We started looking at a “for loop” to run through the input files with the PBS directive “#PBS -v TASKS”.
for ((i=0; i < TASKS; i++)); do
command < input.$i
done <\/strong>
.. and exporting the TASKS variable at CLI with a value equal to the number of inputs needed to be analyzed. The problem with this “for loop” was that it would run through each input at a time and not execute all of them in parallel. To get around this problem you have to add a ampersand (&) to the end of the command you are executing and a “wait” option as seen below.
for ((i=0; i < TASKS; i++)); do
command < input.$i &
done
wait <\/strong><\/p>\n

The above will execute nicely but it doesn’t scale and will discover another stumbling block. Issuing 20 input files will effectively use 20 concurrent CPUs but should you submit 1000 input files you will be limited to the number of cores available on the cluster. If the cluster only has 500 cores available, it will cause the job to be rejected. Then we discovered that PBS have directives for these special cases. Enter PBS directive “#PBS -t” and “${PBS_ARRAYID}”. <\/p>\n

#PBS -N inputs
#PBS -l nodes=1:series600:ppn=1
#PBS -q UCTlong  
cd \/home\/username\/application\/
.\/command < \/home\/username\/application\/inputs\/input.${PBS_ARRAYID}<\/strong><\/p>\n

The job submission script would look something like the above. To submit to the cluster you would use ” qsub -t 0-100 job.sh ” with 0-100 being the range of input files with their respective extentions input.0, input.1, input.2 etc etc …. input.100 in a directory. The ” -t 0-100″ will parse to the PBS_ARRAYID and 100 jobs will be spawned on the cluster with some running and some being queued. To checkup on the status of a array job execute ” qstat -t ” <\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[4],"tags":[],"yoast_head":"\nArray Jobs - UCT HPC<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/04\/12\/array-jobs\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Array Jobs - UCT HPC\" \/>\n<meta property=\"og:description\" content=\"Recently we had been given a challenge by a student who needed to analyze thousands of input files with an application but found it cumbersome to submit each input file individually. We started looking at a "for loop" to run through the input files with the PBS directive "#PBS -v TASKS". for ((i=0; i < TASKS; i++)); do command < input.$i done .. and exporting the TASKS variable at CLI with a value equal to the number of inputs needed to be analyzed. The problem with this "for loop" was that it would run through each input at a time and not execute all of them in parallel. To get around this problem you have to add a ampersand (&) to the end of the command you are executing and a "wait" option as seen below. for ((i=0; i < TASKS; i++)); do command < input.$i & done wait The above will execute nicely but it doesn't scale and will discover another stumbling block. Issuing 20 input files will effectively use 20 concurrent CPUs but should you submit 1000 input files you will be limited to the number of cores available on the cluster. If the cluster only has 500 cores available, it will cause the job to be rejected. Then we discovered that PBS have directives for these special cases. Enter PBS directive "#PBS -t" and "${PBS_ARRAYID}". #PBS -N inputs#PBS -l nodes=1:series600:ppn=1#PBS -q UCTlong  cd \/home\/username\/application\/.\/command < \/home\/username\/application\/inputs\/input.${PBS_ARRAYID}The job submission script would look something like the above. To submit to the cluster you would use " qsub -t 0-100 job.sh " with 0-100 being the range of input files with their respective extentions input.0, input.1, input.2 etc etc .... input.100 in a directory. The " -t 0-100" will parse to the PBS_ARRAYID and 100 jobs will be spawned on the cluster with some running and some being queued. To checkup on the status of a array job execute " qstat -t "\" \/>\n<meta property=\"og:url\" content=\"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/04\/12\/array-jobs\/\" \/>\n<meta property=\"og:site_name\" content=\"UCT HPC\" \/>\n<meta property=\"article:published_time\" content=\"2013-04-12T13:47:20+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2015-08-25T08:17:16+00:00\" \/>\n<meta name=\"author\" content=\"Timothy Carr\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Timothy Carr\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/04\/12\/array-jobs\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/04\/12\/array-jobs\/\"},\"author\":{\"name\":\"Timothy Carr\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#\/schema\/person\/41f6cd039836d7741f2b82a7b7cfe8d0\"},\"headline\":\"Array Jobs\",\"datePublished\":\"2013-04-12T13:47:20+00:00\",\"dateModified\":\"2015-08-25T08:17:16+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/04\/12\/array-jobs\/\"},\"wordCount\":332,\"publisher\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#organization\"},\"articleSection\":[\"hpc\"],\"inLanguage\":\"en-ZA\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/04\/12\/array-jobs\/\",\"url\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/04\/12\/array-jobs\/\",\"name\":\"Array Jobs - UCT HPC\",\"isPartOf\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#website\"},\"datePublished\":\"2013-04-12T13:47:20+00:00\",\"dateModified\":\"2015-08-25T08:17:16+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/04\/12\/array-jobs\/#breadcrumb\"},\"inLanguage\":\"en-ZA\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/04\/12\/array-jobs\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/04\/12\/array-jobs\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/ucthpc.uct.ac.za\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Array Jobs\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#website\",\"url\":\"https:\/\/ucthpc.uct.ac.za\/\",\"name\":\"UCT HPC\",\"description\":\"University of Cape Town High Performance Computing\",\"publisher\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/ucthpc.uct.ac.za\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-ZA\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#organization\",\"name\":\"University of Cape Town High Performance Computing\",\"url\":\"https:\/\/ucthpc.uct.ac.za\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-ZA\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/ucthpc.uct.ac.za\/wp-content\/uploads\/2015\/09\/logocircless.png\",\"contentUrl\":\"https:\/\/ucthpc.uct.ac.za\/wp-content\/uploads\/2015\/09\/logocircless.png\",\"width\":450,\"height\":423,\"caption\":\"University of Cape Town High Performance Computing\"},\"image\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#\/schema\/person\/41f6cd039836d7741f2b82a7b7cfe8d0\",\"name\":\"Timothy Carr\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-ZA\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/7e94dcf3a408e6ada008042fc29d4b15?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/7e94dcf3a408e6ada008042fc29d4b15?s=96&d=mm&r=g\",\"caption\":\"Timothy Carr\"},\"sameAs\":[\"http:\/\/ucthpc.uct.ac.za\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Array Jobs - UCT HPC","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/04\/12\/array-jobs\/","og_locale":"en_US","og_type":"article","og_title":"Array Jobs - UCT HPC","og_description":"Recently we had been given a challenge by a student who needed to analyze thousands of input files with an application but found it cumbersome to submit each input file individually. We started looking at a \"for loop\" to run through the input files with the PBS directive \"#PBS -v TASKS\". for ((i=0; i < TASKS; i++)); do command < input.$i done .. and exporting the TASKS variable at CLI with a value equal to the number of inputs needed to be analyzed. The problem with this \"for loop\" was that it would run through each input at a time and not execute all of them in parallel. To get around this problem you have to add a ampersand (&) to the end of the command you are executing and a \"wait\" option as seen below. for ((i=0; i < TASKS; i++)); do command < input.$i & done wait The above will execute nicely but it doesn't scale and will discover another stumbling block. Issuing 20 input files will effectively use 20 concurrent CPUs but should you submit 1000 input files you will be limited to the number of cores available on the cluster. If the cluster only has 500 cores available, it will cause the job to be rejected. Then we discovered that PBS have directives for these special cases. Enter PBS directive \"#PBS -t\" and \"${PBS_ARRAYID}\". #PBS -N inputs#PBS -l nodes=1:series600:ppn=1#PBS -q UCTlong  cd \/home\/username\/application\/.\/command < \/home\/username\/application\/inputs\/input.${PBS_ARRAYID}The job submission script would look something like the above. To submit to the cluster you would use \" qsub -t 0-100 job.sh \" with 0-100 being the range of input files with their respective extentions input.0, input.1, input.2 etc etc .... input.100 in a directory. The \" -t 0-100\" will parse to the PBS_ARRAYID and 100 jobs will be spawned on the cluster with some running and some being queued. To checkup on the status of a array job execute \" qstat -t \"","og_url":"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/04\/12\/array-jobs\/","og_site_name":"UCT HPC","article_published_time":"2013-04-12T13:47:20+00:00","article_modified_time":"2015-08-25T08:17:16+00:00","author":"Timothy Carr","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Timothy Carr","Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/04\/12\/array-jobs\/#article","isPartOf":{"@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/04\/12\/array-jobs\/"},"author":{"name":"Timothy Carr","@id":"https:\/\/ucthpc.uct.ac.za\/#\/schema\/person\/41f6cd039836d7741f2b82a7b7cfe8d0"},"headline":"Array Jobs","datePublished":"2013-04-12T13:47:20+00:00","dateModified":"2015-08-25T08:17:16+00:00","mainEntityOfPage":{"@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/04\/12\/array-jobs\/"},"wordCount":332,"publisher":{"@id":"https:\/\/ucthpc.uct.ac.za\/#organization"},"articleSection":["hpc"],"inLanguage":"en-ZA"},{"@type":"WebPage","@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/04\/12\/array-jobs\/","url":"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/04\/12\/array-jobs\/","name":"Array Jobs - UCT HPC","isPartOf":{"@id":"https:\/\/ucthpc.uct.ac.za\/#website"},"datePublished":"2013-04-12T13:47:20+00:00","dateModified":"2015-08-25T08:17:16+00:00","breadcrumb":{"@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/04\/12\/array-jobs\/#breadcrumb"},"inLanguage":"en-ZA","potentialAction":[{"@type":"ReadAction","target":["https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/04\/12\/array-jobs\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/04\/12\/array-jobs\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/ucthpc.uct.ac.za\/"},{"@type":"ListItem","position":2,"name":"Array Jobs"}]},{"@type":"WebSite","@id":"https:\/\/ucthpc.uct.ac.za\/#website","url":"https:\/\/ucthpc.uct.ac.za\/","name":"UCT HPC","description":"University of Cape Town High Performance Computing","publisher":{"@id":"https:\/\/ucthpc.uct.ac.za\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/ucthpc.uct.ac.za\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-ZA"},{"@type":"Organization","@id":"https:\/\/ucthpc.uct.ac.za\/#organization","name":"University of Cape Town High Performance Computing","url":"https:\/\/ucthpc.uct.ac.za\/","logo":{"@type":"ImageObject","inLanguage":"en-ZA","@id":"https:\/\/ucthpc.uct.ac.za\/#\/schema\/logo\/image\/","url":"https:\/\/ucthpc.uct.ac.za\/wp-content\/uploads\/2015\/09\/logocircless.png","contentUrl":"https:\/\/ucthpc.uct.ac.za\/wp-content\/uploads\/2015\/09\/logocircless.png","width":450,"height":423,"caption":"University of Cape Town High Performance Computing"},"image":{"@id":"https:\/\/ucthpc.uct.ac.za\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/ucthpc.uct.ac.za\/#\/schema\/person\/41f6cd039836d7741f2b82a7b7cfe8d0","name":"Timothy Carr","image":{"@type":"ImageObject","inLanguage":"en-ZA","@id":"https:\/\/ucthpc.uct.ac.za\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/7e94dcf3a408e6ada008042fc29d4b15?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/7e94dcf3a408e6ada008042fc29d4b15?s=96&d=mm&r=g","caption":"Timothy Carr"},"sameAs":["http:\/\/ucthpc.uct.ac.za"]}]}},"_links":{"self":[{"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/posts\/839"}],"collection":[{"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/comments?post=839"}],"version-history":[{"count":4,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/posts\/839\/revisions"}],"predecessor-version":[{"id":2360,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/posts\/839\/revisions\/2360"}],"wp:attachment":[{"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/media?parent=839"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/categories?post=839"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/tags?post=839"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}