{"id":753,"date":"2014-04-30T11:42:51","date_gmt":"2014-04-30T09:42:51","guid":{"rendered":"http:\/\/oldblogs.uct.ac.za\/blog\/big-bytes\/2014\/04\/30\/computational-hours-and-mpi-issues"},"modified":"2015-08-14T11:24:02","modified_gmt":"2015-08-14T09:24:02","slug":"computational-hours-and-mpi-issues","status":"publish","type":"post","link":"https:\/\/ucthpc.uct.ac.za\/index.php\/2014\/04\/30\/computational-hours-and-mpi-issues\/","title":{"rendered":"Computational hours and MPI issues"},"content":{"rendered":"Our clusters are currently computing more active hours that were processed in the first two years in total. Today we exceeded 300,000 CPU hours of work, that's more than 34 computational years being processed currently.\r\n\r\n&nbsp;\r\n\r\n<img src=\"https:\/\/ucthpc.uct.ac.za\/wp-content\/uploads\/2015\/07\/cumulative.jpg\" alt=\"cumulative\" border=\"0\" \/>\r\n\r\nThere are several MPI issues which users need to keep in mind. \u00a0Nodes 604 and 605 are not completely infiniband enabled. \u00a0This means that some selections of servers will not work with infiniband. \u00a0We would recommend that advanced MPI users steer clear of these nodes by making use of the -l directives to select specific servers:\r\n\r\n<span style=\"font-size: xx-small;\">#PBS -l nodes=srvslshpc600:ppn=2+srvslshpc601:ppn=2+srvslshpc602:ppn=2<\/span>\r\n\r\nwill start a 6 core job over 3 servers with working IB cables. \u00a0It's recommended that users monitor the dashboard to determine which nodescores are free prior to starting an MPI job. \u00a0We apologize for this inconvenience and we will have this sorted out during the June maintenance slot.","protected":false},"excerpt":{"rendered":"<p>Our clusters are currently computing more active hours that were processed in the first two years in total. Today we exceeded 300,000 CPU hours of work, that&#8217;s more than 34 computational years being processed currently.<\/p>\n<p>&nbsp;<\/p>\n<p><img decoding=\"async\" src=\"http:\/\/blogs.uct.ac.za\/gallery\/1253\/cumulative.jpg\" border=\"0\" alt=\"cumulative\">&nbsp;<\/p>\n<p> There are several MPI issues which users need to keep in mind. &nbsp;Nodes 604 and 605 are not completely infiniband enabled. &nbsp;This means that some selections of servers will not work with infiniband. &nbsp;We would recommend that advanced MPI users steer clear of these nodes by making use of the -l directives to select specific servers:<\/p>\n<p><span>#PBS -l nodes=srvslshpc600:ppn=2+srvslshpc601:ppn=2+srvslshpc602:ppn=2<\/span><\/p>\n<p>will start a 6 core job over 3 servers with working IB cables. &nbsp;It&#8217;s recommended that users monitor the dashboard to determine which nodescores are free prior to starting an MPI job. &nbsp;We apologize for this inconvenience and we will have this sorted out during the June maintenance slot.&nbsp;<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[9,4,10],"tags":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v21.4 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Computational hours and MPI issues - UCT HPC<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/ucthpc.uct.ac.za\/index.php\/2014\/04\/30\/computational-hours-and-mpi-issues\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Computational hours and MPI issues - UCT HPC\" \/>\n<meta property=\"og:description\" content=\"Our clusters are currently computing more active hours that were processed in the first two years in total. Today we exceeded 300,000 CPU hours of work, that&#039;s more than 34 computational years being processed currently.&nbsp;&nbsp; There are several MPI issues which users need to keep in mind. &nbsp;Nodes 604 and 605 are not completely infiniband enabled. &nbsp;This means that some selections of servers will not work with infiniband. &nbsp;We would recommend that advanced MPI users steer clear of these nodes by making use of the -l directives to select specific servers:#PBS -l nodes=srvslshpc600:ppn=2+srvslshpc601:ppn=2+srvslshpc602:ppn=2will start a 6 core job over 3 servers with working IB cables. &nbsp;It&#039;s recommended that users monitor the dashboard to determine which nodescores are free prior to starting an MPI job. &nbsp;We apologize for this inconvenience and we will have this sorted out during the June maintenance slot.&nbsp;\" \/>\n<meta property=\"og:url\" content=\"https:\/\/ucthpc.uct.ac.za\/index.php\/2014\/04\/30\/computational-hours-and-mpi-issues\/\" \/>\n<meta property=\"og:site_name\" content=\"UCT HPC\" \/>\n<meta property=\"article:published_time\" content=\"2014-04-30T09:42:51+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2015-08-14T09:24:02+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/ucthpc.uct.ac.za\/wp-content\/uploads\/2015\/07\/cumulative.jpg\" \/>\n<meta name=\"author\" content=\"Andrew Lewis\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Andrew Lewis\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2014\/04\/30\/computational-hours-and-mpi-issues\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2014\/04\/30\/computational-hours-and-mpi-issues\/\"},\"author\":{\"name\":\"Andrew Lewis\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#\/schema\/person\/c183ad1c0a1063124a72d63963ae9c7e\"},\"headline\":\"Computational hours and MPI issues\",\"datePublished\":\"2014-04-30T09:42:51+00:00\",\"dateModified\":\"2015-08-14T09:24:02+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2014\/04\/30\/computational-hours-and-mpi-issues\/\"},\"wordCount\":149,\"publisher\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#organization\"},\"articleSection\":[\"application\",\"hpc\",\"MPI\"],\"inLanguage\":\"en-ZA\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2014\/04\/30\/computational-hours-and-mpi-issues\/\",\"url\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2014\/04\/30\/computational-hours-and-mpi-issues\/\",\"name\":\"Computational hours and MPI issues - UCT HPC\",\"isPartOf\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#website\"},\"datePublished\":\"2014-04-30T09:42:51+00:00\",\"dateModified\":\"2015-08-14T09:24:02+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2014\/04\/30\/computational-hours-and-mpi-issues\/#breadcrumb\"},\"inLanguage\":\"en-ZA\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/ucthpc.uct.ac.za\/index.php\/2014\/04\/30\/computational-hours-and-mpi-issues\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2014\/04\/30\/computational-hours-and-mpi-issues\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/ucthpc.uct.ac.za\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Computational hours and MPI issues\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#website\",\"url\":\"https:\/\/ucthpc.uct.ac.za\/\",\"name\":\"UCT HPC\",\"description\":\"University of Cape Town High Performance Computing\",\"publisher\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/ucthpc.uct.ac.za\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-ZA\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#organization\",\"name\":\"University of Cape Town High Performance Computing\",\"url\":\"https:\/\/ucthpc.uct.ac.za\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-ZA\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/ucthpc.uct.ac.za\/wp-content\/uploads\/2015\/09\/logocircless.png\",\"contentUrl\":\"https:\/\/ucthpc.uct.ac.za\/wp-content\/uploads\/2015\/09\/logocircless.png\",\"width\":450,\"height\":423,\"caption\":\"University of Cape Town High Performance Computing\"},\"image\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#\/schema\/person\/c183ad1c0a1063124a72d63963ae9c7e\",\"name\":\"Andrew Lewis\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-ZA\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/9652c9c73beeab594b8dc2383a880048?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/9652c9c73beeab594b8dc2383a880048?s=96&d=mm&r=g\",\"caption\":\"Andrew Lewis\"},\"sameAs\":[\"http:\/\/blogs.uct.ac.za\/blog\/big-bytes\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Computational hours and MPI issues - UCT HPC","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/ucthpc.uct.ac.za\/index.php\/2014\/04\/30\/computational-hours-and-mpi-issues\/","og_locale":"en_US","og_type":"article","og_title":"Computational hours and MPI issues - UCT HPC","og_description":"Our clusters are currently computing more active hours that were processed in the first two years in total. Today we exceeded 300,000 CPU hours of work, that's more than 34 computational years being processed currently.&nbsp;&nbsp; There are several MPI issues which users need to keep in mind. &nbsp;Nodes 604 and 605 are not completely infiniband enabled. &nbsp;This means that some selections of servers will not work with infiniband. &nbsp;We would recommend that advanced MPI users steer clear of these nodes by making use of the -l directives to select specific servers:#PBS -l nodes=srvslshpc600:ppn=2+srvslshpc601:ppn=2+srvslshpc602:ppn=2will start a 6 core job over 3 servers with working IB cables. &nbsp;It's recommended that users monitor the dashboard to determine which nodescores are free prior to starting an MPI job. &nbsp;We apologize for this inconvenience and we will have this sorted out during the June maintenance slot.&nbsp;","og_url":"https:\/\/ucthpc.uct.ac.za\/index.php\/2014\/04\/30\/computational-hours-and-mpi-issues\/","og_site_name":"UCT HPC","article_published_time":"2014-04-30T09:42:51+00:00","article_modified_time":"2015-08-14T09:24:02+00:00","og_image":[{"url":"https:\/\/ucthpc.uct.ac.za\/wp-content\/uploads\/2015\/07\/cumulative.jpg"}],"author":"Andrew Lewis","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Andrew Lewis","Est. reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/2014\/04\/30\/computational-hours-and-mpi-issues\/#article","isPartOf":{"@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/2014\/04\/30\/computational-hours-and-mpi-issues\/"},"author":{"name":"Andrew Lewis","@id":"https:\/\/ucthpc.uct.ac.za\/#\/schema\/person\/c183ad1c0a1063124a72d63963ae9c7e"},"headline":"Computational hours and MPI issues","datePublished":"2014-04-30T09:42:51+00:00","dateModified":"2015-08-14T09:24:02+00:00","mainEntityOfPage":{"@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/2014\/04\/30\/computational-hours-and-mpi-issues\/"},"wordCount":149,"publisher":{"@id":"https:\/\/ucthpc.uct.ac.za\/#organization"},"articleSection":["application","hpc","MPI"],"inLanguage":"en-ZA"},{"@type":"WebPage","@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/2014\/04\/30\/computational-hours-and-mpi-issues\/","url":"https:\/\/ucthpc.uct.ac.za\/index.php\/2014\/04\/30\/computational-hours-and-mpi-issues\/","name":"Computational hours and MPI issues - UCT HPC","isPartOf":{"@id":"https:\/\/ucthpc.uct.ac.za\/#website"},"datePublished":"2014-04-30T09:42:51+00:00","dateModified":"2015-08-14T09:24:02+00:00","breadcrumb":{"@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/2014\/04\/30\/computational-hours-and-mpi-issues\/#breadcrumb"},"inLanguage":"en-ZA","potentialAction":[{"@type":"ReadAction","target":["https:\/\/ucthpc.uct.ac.za\/index.php\/2014\/04\/30\/computational-hours-and-mpi-issues\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/2014\/04\/30\/computational-hours-and-mpi-issues\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/ucthpc.uct.ac.za\/"},{"@type":"ListItem","position":2,"name":"Computational hours and MPI issues"}]},{"@type":"WebSite","@id":"https:\/\/ucthpc.uct.ac.za\/#website","url":"https:\/\/ucthpc.uct.ac.za\/","name":"UCT HPC","description":"University of Cape Town High Performance Computing","publisher":{"@id":"https:\/\/ucthpc.uct.ac.za\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/ucthpc.uct.ac.za\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-ZA"},{"@type":"Organization","@id":"https:\/\/ucthpc.uct.ac.za\/#organization","name":"University of Cape Town High Performance Computing","url":"https:\/\/ucthpc.uct.ac.za\/","logo":{"@type":"ImageObject","inLanguage":"en-ZA","@id":"https:\/\/ucthpc.uct.ac.za\/#\/schema\/logo\/image\/","url":"https:\/\/ucthpc.uct.ac.za\/wp-content\/uploads\/2015\/09\/logocircless.png","contentUrl":"https:\/\/ucthpc.uct.ac.za\/wp-content\/uploads\/2015\/09\/logocircless.png","width":450,"height":423,"caption":"University of Cape Town High Performance Computing"},"image":{"@id":"https:\/\/ucthpc.uct.ac.za\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/ucthpc.uct.ac.za\/#\/schema\/person\/c183ad1c0a1063124a72d63963ae9c7e","name":"Andrew Lewis","image":{"@type":"ImageObject","inLanguage":"en-ZA","@id":"https:\/\/ucthpc.uct.ac.za\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/9652c9c73beeab594b8dc2383a880048?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/9652c9c73beeab594b8dc2383a880048?s=96&d=mm&r=g","caption":"Andrew Lewis"},"sameAs":["http:\/\/blogs.uct.ac.za\/blog\/big-bytes"]}]}},"_links":{"self":[{"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/posts\/753"}],"collection":[{"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/comments?post=753"}],"version-history":[{"count":3,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/posts\/753\/revisions"}],"predecessor-version":[{"id":2048,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/posts\/753\/revisions\/2048"}],"wp:attachment":[{"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/media?parent=753"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/categories?post=753"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/tags?post=753"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}