{"id":837,"date":"2013-04-17T15:20:14","date_gmt":"2013-04-17T13:20:14","guid":{"rendered":"http:\/\/oldblogs.uct.ac.za\/blog\/big-bytes\/2013\/04\/17\/spawning-obscure-pythons"},"modified":"2015-08-14T11:42:31","modified_gmt":"2015-08-14T09:42:31","slug":"spawning-obscure-pythons","status":"publish","type":"post","link":"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/04\/17\/spawning-obscure-pythons\/","title":{"rendered":"Spawning obscure pythons"},"content":{"rendered":"
One of our users has been having difficulty migrating his code to our new cluster. \u00a0We'd installed a newer version of python (2.7.3) and it would appear that in conjunction with\u00a0mpi4py<\/a>\u00a0and OpenMPI the\u00a0--mca run time tuning and debugging argument<\/a>\u00a0is ignored. \u00a0He is using the fork() call and while MPI discourages this due to possible memory corruption it really has no bearing as the processing is done by a seperately called executable. \u00a0The exact same code runs fine on our Scientific Linux cluster with OpenMPI 1.4 but dies on our new cluster with OpenMPI 1.6. \u00a0It also dies when running on the CHPC SUN cluster which also uses OpenMPI 1.4 so the version of OpenMPI is not the issue. \u00a0The job was dying with the following error as soon as fork() was called:<\/div>\r\n
<\/div>\r\n
An MPI process has executed an operation involving a call to the \"fork()\" system call to create a child process. \u00a0Open MPI is currently operating in a condition that could result in memory corruption or other system errors; your MPI job may hang, crash, or produce silent data corruption. \u00a0The use of fork() (or system() or other calls that create child processes) is strongly discouraged.<\/span><\/div>\r\n
The process that invoked fork was:<\/span><\/div>\r\n
\u00a0 Local host: \u00a0 \u00a0 \u00a0 \u00a0 \u00a0srvslshpc600 (PID 38708)<\/span><\/div>\r\n
\u00a0 MPI_COMM_WORLD rank: 0<\/span><\/div>\r\n
If you are *absolutely sure* that your application will successfully and correctly survive a call to fork(), you may disable this warning by setting the mpi_warn_on_fork MCA parameter to 0.<\/span><\/div>\r\n
Adding --mca mpi_warn_on_fork 0 to the mpirun statement resulted in:<\/div>\r\n
mpiexec has exited due to process rank 0 with PID 47536 on node srvslshpc600 exiting improperly. There are two reasons this could occur:<\/span><\/div>\r\n
1. this process did not call \"init\" before exiting, but others in the job did. This can cause a job to hang indefinitely while it waits for all processes to call \"init\". By rule, if one process calls \"init\", then ALL processes must call \"init\" prior to termination.<\/span><\/div>\r\n
2. this process called \"init\", but exited without calling \"finalize\". By rule, all processes that call \"init\" MUST call \"finalize\" prior to exiting or it will be considered an \"abnormal termination\"<\/span><\/div>\r\n
This may have caused other processes in the application to be terminated by signals sent by mpiexec (as reported here).<\/span><\/div>\r\n
While it would probably be best for him to have recoded his software from scratch we found that by installing Python 2.7.2 the code runs successfully. We have no idea why exactly but the answer would probably be obscure and annoying so we'll leave it at that.<\/div>","protected":false},"excerpt":{"rendered":"
One of our users has been having difficulty migrating his code to our new cluster.  We’d installed a newer version of python (2.7.3) and it would appear that in conjunction with mpi4py<\/a> and OpenMPI the –mca run time tuning and debugging argument<\/a> is ignored.  He is using the fork() call and while MPI discourages this due to possible memory corruption it really has no bearing as the processing is done by a seperately called executable.  The exact same code runs fine on our Scientific Linux cluster with OpenMPI 1.4 but dies on our new cluster with OpenMPI 1.6.  It also dies when running on the CHPC SUN cluster which also uses OpenMPI 1.4 so the version of OpenMPI is not the issue.  The job was dying with the following error as soon as fork() was called:<\/div>\n
<\/div>\n
An MPI process has executed an operation involving a call to the “fork()” system call to create a child process.  Open MPI is currently operating in a condition that could result in memory corruption or other system errors; your MPI job may hang, crash, or produce silent data corruption.  The use of fork() (or system() or other calls that create child processes) is strongly discouraged.<\/span><\/div>\n
The process that invoked fork was:<\/span><\/div>\n
  Local host:          srvslshpc600 (PID 38708)<\/span><\/div>\n
  MPI_COMM_WORLD rank: 0<\/span><\/div>\n
If you are *absolutely sure* that your application will successfully and correctly survive a call to fork(), you may disable this warning by setting the mpi_warn_on_fork MCA parameter to 0.<\/span><\/div>\n
Adding –mca mpi_warn_on_fork 0 to the mpirun statement resulted in:<\/div>\n
mpiexec has exited due to process rank 0 with PID 47536 on node srvslshpc600 exiting improperly. There are two reasons this could occur:<\/span><\/div>\n
1. this process did not call “init” before exiting, but others in the job did. This can cause a job to hang indefinitely while it waits for all processes to call “init”. By rule, if one process calls “init”, then ALL processes must call “init” prior to termination.<\/span><\/div>\n
2. this process called “init”, but exited without calling “finalize”. By rule, all processes that call “init” MUST call “finalize” prior to exiting or it will be considered an “abnormal termination”<\/span><\/div>\n
This may have caused other processes in the application to be terminated by signals sent by mpiexec (as reported here).<\/span><\/div>\n
While it would probably be best for him to have recoded his software from scratch we found that by installing Python 2.7.2 the code runs successfully. We have no idea why exactly but the answer would probably be obscure and annoying so we’ll leave it at that.<\/div>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[4],"tags":[],"yoast_head":"\nSpawning obscure pythons - UCT HPC<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/04\/17\/spawning-obscure-pythons\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Spawning obscure pythons - UCT HPC\" \/>\n<meta property=\"og:description\" content=\"One of our users has been having difficulty migrating his code to our new cluster.  We'd installed a newer version of python (2.7.3) and it would appear that in conjunction with mpi4py and OpenMPI the --mca run time tuning and debugging argument is ignored.  He is using the fork() call and while MPI discourages this due to possible memory corruption it really has no bearing as the processing is done by a seperately called executable.  The exact same code runs fine on our Scientific Linux cluster with OpenMPI 1.4 but dies on our new cluster with OpenMPI 1.6.  It also dies when running on the CHPC SUN cluster which also uses OpenMPI 1.4 so the version of OpenMPI is not the issue.  The job was dying with the following error as soon as fork() was called:An MPI process has executed an operation involving a call to the "fork()" system call to create a child process.  Open MPI is currently operating in a condition that could result in memory corruption or other system errors; your MPI job may hang, crash, or produce silent data corruption.  The use of fork() (or system() or other calls that create child processes) is strongly discouraged.The process that invoked fork was:  Local host:          srvslshpc600 (PID 38708)  MPI_COMM_WORLD rank: 0If you are *absolutely sure* that your application will successfully and correctly survive a call to fork(), you may disable this warning by setting the mpi_warn_on_fork MCA parameter to 0.Adding --mca mpi_warn_on_fork 0 to the mpirun statement resulted in:mpiexec has exited due to process rank 0 with PID 47536 on node srvslshpc600 exiting improperly. There are two reasons this could occur:1. this process did not call "init" before exiting, but others in the job did. This can cause a job to hang indefinitely while it waits for all processes to call "init". By rule, if one process calls "init", then ALL processes must call "init" prior to termination.2. this process called "init", but exited without calling "finalize". By rule, all processes that call "init" MUST call "finalize" prior to exiting or it will be considered an "abnormal termination"This may have caused other processes in the application to be terminated by signals sent by mpiexec (as reported here).While it would probably be best for him to have recoded his software from scratch we found that by installing Python 2.7.2 the code runs successfully. We have no idea why exactly but the answer would probably be obscure and annoying so we'll leave it at that.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/04\/17\/spawning-obscure-pythons\/\" \/>\n<meta property=\"og:site_name\" content=\"UCT HPC\" \/>\n<meta property=\"article:published_time\" content=\"2013-04-17T13:20:14+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2015-08-14T09:42:31+00:00\" \/>\n<meta name=\"author\" content=\"Andrew Lewis\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Andrew Lewis\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/04\/17\/spawning-obscure-pythons\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/04\/17\/spawning-obscure-pythons\/\"},\"author\":{\"name\":\"Andrew Lewis\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#\/schema\/person\/c183ad1c0a1063124a72d63963ae9c7e\"},\"headline\":\"Spawning obscure pythons\",\"datePublished\":\"2013-04-17T13:20:14+00:00\",\"dateModified\":\"2015-08-14T09:42:31+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/04\/17\/spawning-obscure-pythons\/\"},\"wordCount\":423,\"publisher\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#organization\"},\"articleSection\":[\"hpc\"],\"inLanguage\":\"en-ZA\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/04\/17\/spawning-obscure-pythons\/\",\"url\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/04\/17\/spawning-obscure-pythons\/\",\"name\":\"Spawning obscure pythons - UCT HPC\",\"isPartOf\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#website\"},\"datePublished\":\"2013-04-17T13:20:14+00:00\",\"dateModified\":\"2015-08-14T09:42:31+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/04\/17\/spawning-obscure-pythons\/#breadcrumb\"},\"inLanguage\":\"en-ZA\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/04\/17\/spawning-obscure-pythons\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/04\/17\/spawning-obscure-pythons\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/ucthpc.uct.ac.za\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Spawning obscure pythons\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#website\",\"url\":\"https:\/\/ucthpc.uct.ac.za\/\",\"name\":\"UCT HPC\",\"description\":\"University of Cape Town High Performance Computing\",\"publisher\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/ucthpc.uct.ac.za\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-ZA\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#organization\",\"name\":\"University of Cape Town High Performance Computing\",\"url\":\"https:\/\/ucthpc.uct.ac.za\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-ZA\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/ucthpc.uct.ac.za\/wp-content\/uploads\/2015\/09\/logocircless.png\",\"contentUrl\":\"https:\/\/ucthpc.uct.ac.za\/wp-content\/uploads\/2015\/09\/logocircless.png\",\"width\":450,\"height\":423,\"caption\":\"University of Cape Town High Performance Computing\"},\"image\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#\/schema\/person\/c183ad1c0a1063124a72d63963ae9c7e\",\"name\":\"Andrew Lewis\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-ZA\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/9652c9c73beeab594b8dc2383a880048?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/9652c9c73beeab594b8dc2383a880048?s=96&d=mm&r=g\",\"caption\":\"Andrew Lewis\"},\"sameAs\":[\"http:\/\/blogs.uct.ac.za\/blog\/big-bytes\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Spawning obscure pythons - UCT HPC","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/04\/17\/spawning-obscure-pythons\/","og_locale":"en_US","og_type":"article","og_title":"Spawning obscure pythons - UCT HPC","og_description":"One of our users has been having difficulty migrating his code to our new cluster.  We'd installed a newer version of python (2.7.3) and it would appear that in conjunction with mpi4py and OpenMPI the --mca run time tuning and debugging argument is ignored.  He is using the fork() call and while MPI discourages this due to possible memory corruption it really has no bearing as the processing is done by a seperately called executable.  The exact same code runs fine on our Scientific Linux cluster with OpenMPI 1.4 but dies on our new cluster with OpenMPI 1.6.  It also dies when running on the CHPC SUN cluster which also uses OpenMPI 1.4 so the version of OpenMPI is not the issue.  The job was dying with the following error as soon as fork() was called:An MPI process has executed an operation involving a call to the \"fork()\" system call to create a child process.  Open MPI is currently operating in a condition that could result in memory corruption or other system errors; your MPI job may hang, crash, or produce silent data corruption.  The use of fork() (or system() or other calls that create child processes) is strongly discouraged.The process that invoked fork was:  Local host:          srvslshpc600 (PID 38708)  MPI_COMM_WORLD rank: 0If you are *absolutely sure* that your application will successfully and correctly survive a call to fork(), you may disable this warning by setting the mpi_warn_on_fork MCA parameter to 0.Adding --mca mpi_warn_on_fork 0 to the mpirun statement resulted in:mpiexec has exited due to process rank 0 with PID 47536 on node srvslshpc600 exiting improperly. There are two reasons this could occur:1. this process did not call \"init\" before exiting, but others in the job did. This can cause a job to hang indefinitely while it waits for all processes to call \"init\". By rule, if one process calls \"init\", then ALL processes must call \"init\" prior to termination.2. this process called \"init\", but exited without calling \"finalize\". By rule, all processes that call \"init\" MUST call \"finalize\" prior to exiting or it will be considered an \"abnormal termination\"This may have caused other processes in the application to be terminated by signals sent by mpiexec (as reported here).While it would probably be best for him to have recoded his software from scratch we found that by installing Python 2.7.2 the code runs successfully. We have no idea why exactly but the answer would probably be obscure and annoying so we'll leave it at that.","og_url":"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/04\/17\/spawning-obscure-pythons\/","og_site_name":"UCT HPC","article_published_time":"2013-04-17T13:20:14+00:00","article_modified_time":"2015-08-14T09:42:31+00:00","author":"Andrew Lewis","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Andrew Lewis","Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/04\/17\/spawning-obscure-pythons\/#article","isPartOf":{"@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/04\/17\/spawning-obscure-pythons\/"},"author":{"name":"Andrew Lewis","@id":"https:\/\/ucthpc.uct.ac.za\/#\/schema\/person\/c183ad1c0a1063124a72d63963ae9c7e"},"headline":"Spawning obscure pythons","datePublished":"2013-04-17T13:20:14+00:00","dateModified":"2015-08-14T09:42:31+00:00","mainEntityOfPage":{"@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/04\/17\/spawning-obscure-pythons\/"},"wordCount":423,"publisher":{"@id":"https:\/\/ucthpc.uct.ac.za\/#organization"},"articleSection":["hpc"],"inLanguage":"en-ZA"},{"@type":"WebPage","@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/04\/17\/spawning-obscure-pythons\/","url":"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/04\/17\/spawning-obscure-pythons\/","name":"Spawning obscure pythons - UCT HPC","isPartOf":{"@id":"https:\/\/ucthpc.uct.ac.za\/#website"},"datePublished":"2013-04-17T13:20:14+00:00","dateModified":"2015-08-14T09:42:31+00:00","breadcrumb":{"@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/04\/17\/spawning-obscure-pythons\/#breadcrumb"},"inLanguage":"en-ZA","potentialAction":[{"@type":"ReadAction","target":["https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/04\/17\/spawning-obscure-pythons\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/04\/17\/spawning-obscure-pythons\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/ucthpc.uct.ac.za\/"},{"@type":"ListItem","position":2,"name":"Spawning obscure pythons"}]},{"@type":"WebSite","@id":"https:\/\/ucthpc.uct.ac.za\/#website","url":"https:\/\/ucthpc.uct.ac.za\/","name":"UCT HPC","description":"University of Cape Town High Performance Computing","publisher":{"@id":"https:\/\/ucthpc.uct.ac.za\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/ucthpc.uct.ac.za\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-ZA"},{"@type":"Organization","@id":"https:\/\/ucthpc.uct.ac.za\/#organization","name":"University of Cape Town High Performance Computing","url":"https:\/\/ucthpc.uct.ac.za\/","logo":{"@type":"ImageObject","inLanguage":"en-ZA","@id":"https:\/\/ucthpc.uct.ac.za\/#\/schema\/logo\/image\/","url":"https:\/\/ucthpc.uct.ac.za\/wp-content\/uploads\/2015\/09\/logocircless.png","contentUrl":"https:\/\/ucthpc.uct.ac.za\/wp-content\/uploads\/2015\/09\/logocircless.png","width":450,"height":423,"caption":"University of Cape Town High Performance Computing"},"image":{"@id":"https:\/\/ucthpc.uct.ac.za\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/ucthpc.uct.ac.za\/#\/schema\/person\/c183ad1c0a1063124a72d63963ae9c7e","name":"Andrew Lewis","image":{"@type":"ImageObject","inLanguage":"en-ZA","@id":"https:\/\/ucthpc.uct.ac.za\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/9652c9c73beeab594b8dc2383a880048?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/9652c9c73beeab594b8dc2383a880048?s=96&d=mm&r=g","caption":"Andrew Lewis"},"sameAs":["http:\/\/blogs.uct.ac.za\/blog\/big-bytes"]}]}},"_links":{"self":[{"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/posts\/837"}],"collection":[{"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/comments?post=837"}],"version-history":[{"count":2,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/posts\/837\/revisions"}],"predecessor-version":[{"id":2071,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/posts\/837\/revisions\/2071"}],"wp:attachment":[{"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/media?parent=837"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/categories?post=837"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/tags?post=837"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}