{"id":791,"date":"2013-12-09T17:36:58","date_gmt":"2013-12-09T15:36:58","guid":{"rendered":"http:\/\/oldblogs.uct.ac.za\/blog\/big-bytes\/2013\/12\/09\/cluster-upgrades"},"modified":"2015-08-14T11:39:13","modified_gmt":"2015-08-14T09:39:13","slug":"cluster-upgrades","status":"publish","type":"post","link":"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/12\/09\/cluster-upgrades\/","title":{"rendered":"Cluster upgrades"},"content":{"rendered":"
Both clusters are back up again after the upgrade, we'd like to thank our users for their cooperation and patience over the weekend. \u00a0Apart from numerous software patches we also transfered various data volumes to the new Netapp as our old HP EVA 8000 SAN is due for decomissioning. \u00a0One minor hiccup experienced during the upgrade was the Infiniband drivers requiring a change to OpenMPI. \u00a0While mpi jobs ran they completed with the following cosmetic error:<\/div>\r\n
--------------------------------------------------------------------------<\/span><\/div>\r\n
Open MPI failed to open the \/dev\/knem device due to a local error.\u00a0<\/span>Please check with your system administrator to get the problem fixed,\u00a0<\/span>or set the btl_sm_use_knem MCA parameter to 0 to run without \/dev\/knem\u00a0<\/span>support.<\/span><\/div>\r\n
\u00a0Local host: srvslshpc601<\/span><\/div>\r\n
\u00a0Errno: \u00a0 \u00a0 \u00a02 (No such file or directory)<\/span><\/div>\r\n
--------------------------------------------------------------------------<\/span><\/div>\r\n
[srvslshpc601:24784] 3 more processes have sent help message help-mpi-btl-sm.txt \/ knem fail open<\/span><\/div>\r\n
[srvslshpc601:24784] Set MCA parameter \"orte_base_help_aggregate\" to 0 to see all help \/ error messages<\/span><\/div>\r\n
This required adding btl_sm_use_knem = 0<\/span> to \/usr\/mpi\/gcc\/openmpi-1.6.5\/etc\/openmpi-mca-params.conf<\/span><\/div>","protected":false},"excerpt":{"rendered":"
Both clusters are back up again after the upgrade, we’d like to thank our users for their cooperation and patience over the weekend.  Apart from numerous software patches we also transfered various data volumes to the new Netapp as our old HP EVA 8000 SAN is due for decomissioning.  One minor hiccup experienced during the upgrade was the Infiniband drivers requiring a change to OpenMPI.  While mpi jobs ran they completed with the following cosmetic error:<\/div>\n
————————————————————————–<\/span><\/div>\n
Open MPI failed to open the \/dev\/knem device due to a local error. <\/span>Please check with your system administrator to get the problem fixed, <\/span>or set the btl_sm_use_knem MCA parameter to 0 to run without \/dev\/knem <\/span>support.<\/span><\/div>\n
 Local host: srvslshpc601<\/span><\/div>\n
 Errno:      2 (No such file or directory)<\/span><\/div>\n
————————————————————————–<\/span><\/div>\n
[srvslshpc601:24784] 3 more processes have sent help message help-mpi-btl-sm.txt \/ knem fail open<\/span><\/div>\n
[srvslshpc601:24784] Set MCA parameter “orte_base_help_aggregate” to 0 to see all help \/ error messages<\/span><\/div>\n
This required adding btl_sm_use_knem = 0<\/span> to \/usr\/mpi\/gcc\/openmpi-1.6.5\/etc\/openmpi-mca-params.conf<\/span><\/div>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[4],"tags":[],"yoast_head":"\nCluster upgrades - UCT HPC<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/12\/09\/cluster-upgrades\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Cluster upgrades - UCT HPC\" \/>\n<meta property=\"og:description\" content=\"Both clusters are back up again after the upgrade, we'd like to thank our users for their cooperation and patience over the weekend.  Apart from numerous software patches we also transfered various data volumes to the new Netapp as our old HP EVA 8000 SAN is due for decomissioning.  One minor hiccup experienced during the upgrade was the Infiniband drivers requiring a change to OpenMPI.  While mpi jobs ran they completed with the following cosmetic error:--------------------------------------------------------------------------Open MPI failed to open the \/dev\/knem device due to a local error. Please check with your system administrator to get the problem fixed, or set the btl_sm_use_knem MCA parameter to 0 to run without \/dev\/knem support. Local host: srvslshpc601 Errno:      2 (No such file or directory)--------------------------------------------------------------------------[srvslshpc601:24784] 3 more processes have sent help message help-mpi-btl-sm.txt \/ knem fail open[srvslshpc601:24784] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help \/ error messagesThis required adding btl_sm_use_knem = 0 to \/usr\/mpi\/gcc\/openmpi-1.6.5\/etc\/openmpi-mca-params.conf\" \/>\n<meta property=\"og:url\" content=\"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/12\/09\/cluster-upgrades\/\" \/>\n<meta property=\"og:site_name\" content=\"UCT HPC\" \/>\n<meta property=\"article:published_time\" content=\"2013-12-09T15:36:58+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2015-08-14T09:39:13+00:00\" \/>\n<meta name=\"author\" content=\"Andrew Lewis\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Andrew Lewis\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/12\/09\/cluster-upgrades\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/12\/09\/cluster-upgrades\/\"},\"author\":{\"name\":\"Andrew Lewis\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#\/schema\/person\/c183ad1c0a1063124a72d63963ae9c7e\"},\"headline\":\"Cluster upgrades\",\"datePublished\":\"2013-12-09T15:36:58+00:00\",\"dateModified\":\"2015-08-14T09:39:13+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/12\/09\/cluster-upgrades\/\"},\"wordCount\":171,\"publisher\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#organization\"},\"articleSection\":[\"hpc\"],\"inLanguage\":\"en-ZA\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/12\/09\/cluster-upgrades\/\",\"url\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/12\/09\/cluster-upgrades\/\",\"name\":\"Cluster upgrades - UCT HPC\",\"isPartOf\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#website\"},\"datePublished\":\"2013-12-09T15:36:58+00:00\",\"dateModified\":\"2015-08-14T09:39:13+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/12\/09\/cluster-upgrades\/#breadcrumb\"},\"inLanguage\":\"en-ZA\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/12\/09\/cluster-upgrades\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/12\/09\/cluster-upgrades\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/ucthpc.uct.ac.za\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Cluster upgrades\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#website\",\"url\":\"https:\/\/ucthpc.uct.ac.za\/\",\"name\":\"UCT HPC\",\"description\":\"University of Cape Town High Performance Computing\",\"publisher\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/ucthpc.uct.ac.za\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-ZA\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#organization\",\"name\":\"University of Cape Town High Performance Computing\",\"url\":\"https:\/\/ucthpc.uct.ac.za\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-ZA\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/ucthpc.uct.ac.za\/wp-content\/uploads\/2015\/09\/logocircless.png\",\"contentUrl\":\"https:\/\/ucthpc.uct.ac.za\/wp-content\/uploads\/2015\/09\/logocircless.png\",\"width\":450,\"height\":423,\"caption\":\"University of Cape Town High Performance Computing\"},\"image\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#\/schema\/person\/c183ad1c0a1063124a72d63963ae9c7e\",\"name\":\"Andrew Lewis\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-ZA\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/9652c9c73beeab594b8dc2383a880048?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/9652c9c73beeab594b8dc2383a880048?s=96&d=mm&r=g\",\"caption\":\"Andrew Lewis\"},\"sameAs\":[\"http:\/\/blogs.uct.ac.za\/blog\/big-bytes\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Cluster upgrades - UCT HPC","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/12\/09\/cluster-upgrades\/","og_locale":"en_US","og_type":"article","og_title":"Cluster upgrades - UCT HPC","og_description":"Both clusters are back up again after the upgrade, we'd like to thank our users for their cooperation and patience over the weekend.  Apart from numerous software patches we also transfered various data volumes to the new Netapp as our old HP EVA 8000 SAN is due for decomissioning.  One minor hiccup experienced during the upgrade was the Infiniband drivers requiring a change to OpenMPI.  While mpi jobs ran they completed with the following cosmetic error:--------------------------------------------------------------------------Open MPI failed to open the \/dev\/knem device due to a local error. Please check with your system administrator to get the problem fixed, or set the btl_sm_use_knem MCA parameter to 0 to run without \/dev\/knem support. Local host: srvslshpc601 Errno:      2 (No such file or directory)--------------------------------------------------------------------------[srvslshpc601:24784] 3 more processes have sent help message help-mpi-btl-sm.txt \/ knem fail open[srvslshpc601:24784] Set MCA parameter \"orte_base_help_aggregate\" to 0 to see all help \/ error messagesThis required adding btl_sm_use_knem = 0 to \/usr\/mpi\/gcc\/openmpi-1.6.5\/etc\/openmpi-mca-params.conf","og_url":"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/12\/09\/cluster-upgrades\/","og_site_name":"UCT HPC","article_published_time":"2013-12-09T15:36:58+00:00","article_modified_time":"2015-08-14T09:39:13+00:00","author":"Andrew Lewis","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Andrew Lewis","Est. reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/12\/09\/cluster-upgrades\/#article","isPartOf":{"@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/12\/09\/cluster-upgrades\/"},"author":{"name":"Andrew Lewis","@id":"https:\/\/ucthpc.uct.ac.za\/#\/schema\/person\/c183ad1c0a1063124a72d63963ae9c7e"},"headline":"Cluster upgrades","datePublished":"2013-12-09T15:36:58+00:00","dateModified":"2015-08-14T09:39:13+00:00","mainEntityOfPage":{"@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/12\/09\/cluster-upgrades\/"},"wordCount":171,"publisher":{"@id":"https:\/\/ucthpc.uct.ac.za\/#organization"},"articleSection":["hpc"],"inLanguage":"en-ZA"},{"@type":"WebPage","@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/12\/09\/cluster-upgrades\/","url":"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/12\/09\/cluster-upgrades\/","name":"Cluster upgrades - UCT HPC","isPartOf":{"@id":"https:\/\/ucthpc.uct.ac.za\/#website"},"datePublished":"2013-12-09T15:36:58+00:00","dateModified":"2015-08-14T09:39:13+00:00","breadcrumb":{"@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/12\/09\/cluster-upgrades\/#breadcrumb"},"inLanguage":"en-ZA","potentialAction":[{"@type":"ReadAction","target":["https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/12\/09\/cluster-upgrades\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/12\/09\/cluster-upgrades\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/ucthpc.uct.ac.za\/"},{"@type":"ListItem","position":2,"name":"Cluster upgrades"}]},{"@type":"WebSite","@id":"https:\/\/ucthpc.uct.ac.za\/#website","url":"https:\/\/ucthpc.uct.ac.za\/","name":"UCT HPC","description":"University of Cape Town High Performance Computing","publisher":{"@id":"https:\/\/ucthpc.uct.ac.za\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/ucthpc.uct.ac.za\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-ZA"},{"@type":"Organization","@id":"https:\/\/ucthpc.uct.ac.za\/#organization","name":"University of Cape Town High Performance Computing","url":"https:\/\/ucthpc.uct.ac.za\/","logo":{"@type":"ImageObject","inLanguage":"en-ZA","@id":"https:\/\/ucthpc.uct.ac.za\/#\/schema\/logo\/image\/","url":"https:\/\/ucthpc.uct.ac.za\/wp-content\/uploads\/2015\/09\/logocircless.png","contentUrl":"https:\/\/ucthpc.uct.ac.za\/wp-content\/uploads\/2015\/09\/logocircless.png","width":450,"height":423,"caption":"University of Cape Town High Performance Computing"},"image":{"@id":"https:\/\/ucthpc.uct.ac.za\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/ucthpc.uct.ac.za\/#\/schema\/person\/c183ad1c0a1063124a72d63963ae9c7e","name":"Andrew Lewis","image":{"@type":"ImageObject","inLanguage":"en-ZA","@id":"https:\/\/ucthpc.uct.ac.za\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/9652c9c73beeab594b8dc2383a880048?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/9652c9c73beeab594b8dc2383a880048?s=96&d=mm&r=g","caption":"Andrew Lewis"},"sameAs":["http:\/\/blogs.uct.ac.za\/blog\/big-bytes"]}]}},"_links":{"self":[{"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/posts\/791"}],"collection":[{"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/comments?post=791"}],"version-history":[{"count":2,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/posts\/791\/revisions"}],"predecessor-version":[{"id":2066,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/posts\/791\/revisions\/2066"}],"wp:attachment":[{"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/media?parent=791"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/categories?post=791"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/tags?post=791"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}