{"id":1122,"date":"2011-04-22T12:33:48","date_gmt":"2011-04-22T10:33:48","guid":{"rendered":"http:\/\/oldblogs.uct.ac.za\/blog\/big-bytes\/2011\/04\/22\/getting-gromacs"},"modified":"2022-09-26T20:04:47","modified_gmt":"2022-09-26T18:04:47","slug":"getting-gromacs","status":"publish","type":"post","link":"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/04\/22\/getting-gromacs\/","title":{"rendered":"Getting Gromacs"},"content":{"rendered":"<a href=\"http:\/\/www.gromacs.org\/\">Gromacs<\/a> is used to perform molecular dynamics simulations.\u00a0 It's a popular tool and we decided to install it on our cluster based on the needs of two researchers, one at UCT and the other in Kenya.\u00a0 Gromacs is designed to work on many platforms from desktops to high end clustered systems.\u00a0 It is optimized to run in multi threaded systems as well as over multiple nodes via MPI.\r\n\r\nThere are a number of different ways to install the software.\u00a0 As we are running Scientific Linux we decided to install via the <a href=\"http:\/\/fedoraproject.org\/wiki\/EPEL\">EPEL repository<\/a> rather than download an RPM or compile from scratch.\u00a0 While the instructions on the Gromacs site claim that installation is simple, in order to get the package to interact with FFT, as well as compile with double precision support, multi thread and MPI capability is not trivial.\u00a0 Gromacs' simple installation instructions can be <a href=\"http:\/\/www.gromacs.org\/Downloads\/Installation_Instructions\">found here<\/a> and we decided that while making for an interesting read we'd rather not grapple with them.\u00a0 So a quick \"yum install gromacs\" was all that was required.\u00a0 Right?\r\n\r\nNo, not quite so simple.\u00a0 The EPEL repo has a separate install for MPI, so another \"yum install gromacs-openmpi\" was required.\u00a0 Then it was a matter of discovering that under the new EPEL compile of Gromacs 4.5.3 the binaries have all been renamed and now start with \"g_\" so mdrun becomes g_mdrun. However once that hurdle was overcome running the software was fairly simple. Right?\r\n\r\nWell, not quite.\u00a0 While the program ran (and here one of our researchers at UCT was very helpful in providing some test cases), the software seemed to perform a bit, well, greedily.\u00a0 Running on a single worker node, or even multiple worker nodes it grabbed as many cores as were available, which is not great for the scheduling software as it looses track of how many cores are free for other applications.\u00a0 Some research indicated that mdrun no longer needed its own -np argument as Gromacs version 4 and above makes use of mpirun's -np argument.\u00a0 However it was only after we introduced the -nt argument for mdrun that we finally regained control over the number of cores used.\u00a0 Problem solved.\u00a0 Right?\r\n\r\nNot really.\u00a0 After examining the output our UCT user noticed that multiple output files were being created, almost as if independent instances of the software was being run on each node.\u00a0 After some investigation it turned out that this was exactly what was happening, and then it dawned on us that this was why the software had not behaved properly in the first instance and required the -nt argument.\u00a0 The problem was that we were running a non MPI aware compilation that had been configured only for multi-threading.\u00a0 After doing some searching through the file system we found what we were looking for, g_mdrun_openmpi, which strangely wasn't on the search path which was why we'd missed it initially.\u00a0 Running this with mpirun would surely solve the problem.\u00a0 Right?\r\n\r\nWell, almost.\u00a0 Turns out that EPEL's RPM was compiled without shared library support.\u00a0 Trying to run the software even directly on the worker node gave the following nasty message: <span style=\"font-size: xx-small; color: #ff0000;\">Error while loading shared libraries: libgmxpreprocess_openmpi.so.6: cannot open shared object file.\u00a0 <\/span>Manually adding the library to the LD_LIBRARY environment variable fixed the problem which should allow one to submit jobs to the scheduler.\u00a0 Right?\r\n\r\nWell technically yes.\u00a0 However the jobs still failed because of the shared library problem.\u00a0 Exporting the environment variable in the shell script or even placing it in the profile didn't seem to work.\u00a0 We're not quite sure why but it may have something to do with how mpirun connects to the worker nodes.\u00a0 Feeling a bit desperate now we pulled a sneaky trick and used mpirun's environment argument -x to set the LD_LIBRARY variable.\r\n\r\nFinally success.\u00a0 Jobs start on multiple cores, ignore the -nt argument, and the number of threads started equals the -np argument, correctly spread over the available cores in the cluster.\u00a0 There is only one log file, one topology file and one energy file, as expected.\u00a0 The log file indicates that there are a number of nodes being used equivalent to the -np argument.\u00a0 Running on more nodes also seems to introduce a significant speed improvement.\u00a0 The only fly in the ointment seems to be the visualization files with two of them being updated simultaneously.\u00a0 These are meant to be used with the Grace visualization package, are not compulsory and can even be disabled.\r\n\r\nRunning a job now shows how rapidly the threads are distributed over the cores in the following image, snapshots taken 10 seconds apart.\r\n\r\n<img src=\"https:\/\/ucthpc.uct.ac.za\/wp-content\/uploads\/2015\/07\/gromacs.png\" alt=\"gromacs jobs\" border=\"0\" \/>\r\n\r\nMore investigation is still to be done in looking at the efficiency of multiple threads, for instance is the relationship between threads and job speed inversely geometric as expected and if so what is the minimum number of cores that will achieve at least a 90% speed improvement?\u00a0 The communication latency between nodes will also be examined as multiple cores seem to consume considerable bandwidth.","protected":false},"excerpt":{"rendered":"<p><a href=\"http:\/\/www.gromacs.org\/\">Gromacs<\/a> is used to perform molecular dynamics simulations.&nbsp; It&#8217;s a popular tool and we decided to install it on our cluster based on the needs of two researchers, one at UCT and the other in Kenya.&nbsp; Gromacs is designed to work on many platforms from desktops to high end clustered systems.&nbsp; It is optimized to run in multi threaded systems as well as over multiple nodes via MPI.<\/p>\n<p>There are a number of different ways to install the software.&nbsp; As we are running Scientific Linux we decided to install via the <a href=\"http:\/\/fedoraproject.org\/wiki\/EPEL\">EPEL repository<\/a> rather than download an RPM or compile from scratch.&nbsp; While the instructions on the Gromacs site claim that installation is simple, in order to get the package to interact with FFT, as well as compile with double precision support, multi thread and MPI capability is not trivial.&nbsp; Gromacs&#8217; simple installation instructions can be <a href=\"http:\/\/www.gromacs.org\/Downloads\/Installation_Instructions\">found here<\/a> and we decided that while making for an interesting read we&#8217;d rather not grapple with them.&nbsp; So a quick &#8220;yum install gromacs&#8221; was all that was required.&nbsp; Right?<\/p>\n<p>No, not quite so simple.&nbsp; The EPEL repo has a separate install for MPI, so another &#8220;yum install gromacs-openmpi&#8221; was required.&nbsp; Then it was a matter of discovering that under the new EPEL compile of Gromacs 4.5.3 the binaries have all been renamed and now start with &#8220;g_&#8221; so mdrun becomes g_mdrun. However once that hurdle was overcome running the software was fairly simple. Right?<\/p>\n<p>Well, not quite.&nbsp; While the program ran (and here one of our researchers at UCT was very helpful in providing some test cases), the software seemed to perform a bit, well, greedily.&nbsp; Running on a single worker node, or even multiple worker nodes it grabbed as many cores as were available, which is not great for the scheduling software as it looses track of how many cores are free for other applications.&nbsp; Some research indicated that mdrun no longer needed its own -np argument as Gromacs version 4 and above makes use of mpirun&#8217;s -np argument.&nbsp; However it was only after we introduced the -nt argument for mdrun that we finally regained control over the number of cores used.&nbsp; Problem solved.&nbsp; Right?<\/p>\n<p>Not really.&nbsp; After examining the output our UCT user noticed that multiple output files were being created, almost as if independent instances of the software was being run on each node.&nbsp; After some investigation it turned out that this was exactly what was happening, and then it dawned on us that this was why the software had not behaved properly in the first instance and required the -nt argument.&nbsp; The problem was that we were running a non MPI aware compilation that had been configured only for multi-threading.&nbsp; After doing some searching through the file system we found what we were looking for, g_mdrun_openmpi, which strangely wasn&#8217;t on the search path which was why we&#8217;d missed it initially.&nbsp; Running this with mpirun would surely solve the problem.&nbsp; Right?<\/p>\n<p>Well, almost.&nbsp; Turns out that EPEL&#8217;s RPM was compiled without shared library support.&nbsp; Trying to run the software even directly on the worker node gave the following nasty message: <span>Error while loading shared libraries: libgmxpreprocess_openmpi.so.6: cannot open shared object file.&nbsp; <\/span>Manually adding the library to the LD_LIBRARY environment variable fixed the problem which should allow one to submit jobs to the scheduler.&nbsp; Right?<\/p>\n<p>Well technically yes.&nbsp; However the jobs still failed because of the shared library problem.&nbsp; Exporting the environment variable in the shell script or even placing it in the profile didn&#8217;t seem to work.&nbsp; We&#8217;re not quite sure why but it may have something to do with how mpirun connects to the worker nodes.&nbsp; Feeling a bit desperate now we pulled a sneaky trick and used mpirun&#8217;s environment argument -x to set the LD_LIBRARY variable.<\/p>\n<p>Finally success.&nbsp; Jobs start on multiple cores, ignore the -nt argument, and the number of threads started equals the -np argument, correctly spread over the available cores in the cluster.&nbsp; There is only one log file, one topology file and one energy file, as expected.&nbsp; The log file indicates that there are a number of nodes being used equivalent to the -np argument.&nbsp; Running on more nodes also seems to introduce a significant speed improvement.&nbsp; The only fly in the ointment seems to be the visualization files with two of them being updated simultaneously.&nbsp; These are meant to be used with the Grace visualization package, are not compulsory and can even be disabled.<\/p>\n<p>Running a job now shows how rapidly the threads are distributed over the cores in the following image, snapshots taken 10 seconds apart.<\/p>\n<p><img decoding=\"async\" src=\"http:\/\/blogs.uct.ac.za\/gallery\/1253\/gromacs.png\" border=\"0\" alt=\"gromacs jobs\"><\/p>\n<p>More investigation is still to be done in looking at the efficiency of multiple threads, for instance is the relationship between threads and job speed inversely geometric as expected and if so what is the minimum number of cores that will achieve at least a 90% speed improvement?&nbsp; The communication latency between nodes will also be examined as multiple cores seem to consume considerable bandwidth.<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[9,4,10],"tags":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v21.4 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Getting Gromacs - UCT HPC<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/04\/22\/getting-gromacs\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Getting Gromacs - UCT HPC\" \/>\n<meta property=\"og:description\" content=\"Gromacs is used to perform molecular dynamics simulations.&nbsp; It&#039;s a popular tool and we decided to install it on our cluster based on the needs of two researchers, one at UCT and the other in Kenya.&nbsp; Gromacs is designed to work on many platforms from desktops to high end clustered systems.&nbsp; It is optimized to run in multi threaded systems as well as over multiple nodes via MPI.There are a number of different ways to install the software.&nbsp; As we are running Scientific Linux we decided to install via the EPEL repository rather than download an RPM or compile from scratch.&nbsp; While the instructions on the Gromacs site claim that installation is simple, in order to get the package to interact with FFT, as well as compile with double precision support, multi thread and MPI capability is not trivial.&nbsp; Gromacs&#039; simple installation instructions can be found here and we decided that while making for an interesting read we&#039;d rather not grapple with them.&nbsp; So a quick &quot;yum install gromacs&quot; was all that was required.&nbsp; Right?No, not quite so simple.&nbsp; The EPEL repo has a separate install for MPI, so another &quot;yum install gromacs-openmpi&quot; was required.&nbsp; Then it was a matter of discovering that under the new EPEL compile of Gromacs 4.5.3 the binaries have all been renamed and now start with &quot;g_&quot; so mdrun becomes g_mdrun. However once that hurdle was overcome running the software was fairly simple. Right?Well, not quite.&nbsp; While the program ran (and here one of our researchers at UCT was very helpful in providing some test cases), the software seemed to perform a bit, well, greedily.&nbsp; Running on a single worker node, or even multiple worker nodes it grabbed as many cores as were available, which is not great for the scheduling software as it looses track of how many cores are free for other applications.&nbsp; Some research indicated that mdrun no longer needed its own -np argument as Gromacs version 4 and above makes use of mpirun&#039;s -np argument.&nbsp; However it was only after we introduced the -nt argument for mdrun that we finally regained control over the number of cores used.&nbsp; Problem solved.&nbsp; Right?Not really.&nbsp; After examining the output our UCT user noticed that multiple output files were being created, almost as if independent instances of the software was being run on each node.&nbsp; After some investigation it turned out that this was exactly what was happening, and then it dawned on us that this was why the software had not behaved properly in the first instance and required the -nt argument.&nbsp; The problem was that we were running a non MPI aware compilation that had been configured only for multi-threading.&nbsp; After doing some searching through the file system we found what we were looking for, g_mdrun_openmpi, which strangely wasn&#039;t on the search path which was why we&#039;d missed it initially.&nbsp; Running this with mpirun would surely solve the problem.&nbsp; Right?Well, almost.&nbsp; Turns out that EPEL&#039;s RPM was compiled without shared library support.&nbsp; Trying to run the software even directly on the worker node gave the following nasty message: Error while loading shared libraries: libgmxpreprocess_openmpi.so.6: cannot open shared object file.&nbsp; Manually adding the library to the LD_LIBRARY environment variable fixed the problem which should allow one to submit jobs to the scheduler.&nbsp; Right?Well technically yes.&nbsp; However the jobs still failed because of the shared library problem.&nbsp; Exporting the environment variable in the shell script or even placing it in the profile didn&#039;t seem to work.&nbsp; We&#039;re not quite sure why but it may have something to do with how mpirun connects to the worker nodes.&nbsp; Feeling a bit desperate now we pulled a sneaky trick and used mpirun&#039;s environment argument -x to set the LD_LIBRARY variable.Finally success.&nbsp; Jobs start on multiple cores, ignore the -nt argument, and the number of threads started equals the -np argument, correctly spread over the available cores in the cluster.&nbsp; There is only one log file, one topology file and one energy file, as expected.&nbsp; The log file indicates that there are a number of nodes being used equivalent to the -np argument.&nbsp; Running on more nodes also seems to introduce a significant speed improvement.&nbsp; The only fly in the ointment seems to be the visualization files with two of them being updated simultaneously.&nbsp; These are meant to be used with the Grace visualization package, are not compulsory and can even be disabled.Running a job now shows how rapidly the threads are distributed over the cores in the following image, snapshots taken 10 seconds apart.More investigation is still to be done in looking at the efficiency of multiple threads, for instance is the relationship between threads and job speed inversely geometric as expected and if so what is the minimum number of cores that will achieve at least a 90% speed improvement?&nbsp; The communication latency between nodes will also be examined as multiple cores seem to consume considerable bandwidth.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/04\/22\/getting-gromacs\/\" \/>\n<meta property=\"og:site_name\" content=\"UCT HPC\" \/>\n<meta property=\"article:published_time\" content=\"2011-04-22T10:33:48+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2022-09-26T18:04:47+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/ucthpc.uct.ac.za\/wp-content\/uploads\/2015\/07\/gromacs.png\" \/>\n<meta name=\"author\" content=\"Andrew Lewis\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Andrew Lewis\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/04\/22\/getting-gromacs\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/04\/22\/getting-gromacs\/\"},\"author\":{\"name\":\"Andrew Lewis\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#\/schema\/person\/c183ad1c0a1063124a72d63963ae9c7e\"},\"headline\":\"Getting Gromacs\",\"datePublished\":\"2011-04-22T10:33:48+00:00\",\"dateModified\":\"2022-09-26T18:04:47+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/04\/22\/getting-gromacs\/\"},\"wordCount\":834,\"publisher\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#organization\"},\"articleSection\":[\"application\",\"hpc\",\"MPI\"],\"inLanguage\":\"en-ZA\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/04\/22\/getting-gromacs\/\",\"url\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/04\/22\/getting-gromacs\/\",\"name\":\"Getting Gromacs - UCT HPC\",\"isPartOf\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#website\"},\"datePublished\":\"2011-04-22T10:33:48+00:00\",\"dateModified\":\"2022-09-26T18:04:47+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/04\/22\/getting-gromacs\/#breadcrumb\"},\"inLanguage\":\"en-ZA\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/04\/22\/getting-gromacs\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/04\/22\/getting-gromacs\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/ucthpc.uct.ac.za\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Getting Gromacs\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#website\",\"url\":\"https:\/\/ucthpc.uct.ac.za\/\",\"name\":\"UCT HPC\",\"description\":\"University of Cape Town High Performance Computing\",\"publisher\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/ucthpc.uct.ac.za\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-ZA\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#organization\",\"name\":\"University of Cape Town High Performance Computing\",\"url\":\"https:\/\/ucthpc.uct.ac.za\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-ZA\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/ucthpc.uct.ac.za\/wp-content\/uploads\/2015\/09\/logocircless.png\",\"contentUrl\":\"https:\/\/ucthpc.uct.ac.za\/wp-content\/uploads\/2015\/09\/logocircless.png\",\"width\":450,\"height\":423,\"caption\":\"University of Cape Town High Performance Computing\"},\"image\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#\/schema\/person\/c183ad1c0a1063124a72d63963ae9c7e\",\"name\":\"Andrew Lewis\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-ZA\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/9652c9c73beeab594b8dc2383a880048?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/9652c9c73beeab594b8dc2383a880048?s=96&d=mm&r=g\",\"caption\":\"Andrew Lewis\"},\"sameAs\":[\"http:\/\/blogs.uct.ac.za\/blog\/big-bytes\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Getting Gromacs - UCT HPC","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/04\/22\/getting-gromacs\/","og_locale":"en_US","og_type":"article","og_title":"Getting Gromacs - UCT HPC","og_description":"Gromacs is used to perform molecular dynamics simulations.&nbsp; It's a popular tool and we decided to install it on our cluster based on the needs of two researchers, one at UCT and the other in Kenya.&nbsp; Gromacs is designed to work on many platforms from desktops to high end clustered systems.&nbsp; It is optimized to run in multi threaded systems as well as over multiple nodes via MPI.There are a number of different ways to install the software.&nbsp; As we are running Scientific Linux we decided to install via the EPEL repository rather than download an RPM or compile from scratch.&nbsp; While the instructions on the Gromacs site claim that installation is simple, in order to get the package to interact with FFT, as well as compile with double precision support, multi thread and MPI capability is not trivial.&nbsp; Gromacs' simple installation instructions can be found here and we decided that while making for an interesting read we'd rather not grapple with them.&nbsp; So a quick \"yum install gromacs\" was all that was required.&nbsp; Right?No, not quite so simple.&nbsp; The EPEL repo has a separate install for MPI, so another \"yum install gromacs-openmpi\" was required.&nbsp; Then it was a matter of discovering that under the new EPEL compile of Gromacs 4.5.3 the binaries have all been renamed and now start with \"g_\" so mdrun becomes g_mdrun. However once that hurdle was overcome running the software was fairly simple. Right?Well, not quite.&nbsp; While the program ran (and here one of our researchers at UCT was very helpful in providing some test cases), the software seemed to perform a bit, well, greedily.&nbsp; Running on a single worker node, or even multiple worker nodes it grabbed as many cores as were available, which is not great for the scheduling software as it looses track of how many cores are free for other applications.&nbsp; Some research indicated that mdrun no longer needed its own -np argument as Gromacs version 4 and above makes use of mpirun's -np argument.&nbsp; However it was only after we introduced the -nt argument for mdrun that we finally regained control over the number of cores used.&nbsp; Problem solved.&nbsp; Right?Not really.&nbsp; After examining the output our UCT user noticed that multiple output files were being created, almost as if independent instances of the software was being run on each node.&nbsp; After some investigation it turned out that this was exactly what was happening, and then it dawned on us that this was why the software had not behaved properly in the first instance and required the -nt argument.&nbsp; The problem was that we were running a non MPI aware compilation that had been configured only for multi-threading.&nbsp; After doing some searching through the file system we found what we were looking for, g_mdrun_openmpi, which strangely wasn't on the search path which was why we'd missed it initially.&nbsp; Running this with mpirun would surely solve the problem.&nbsp; Right?Well, almost.&nbsp; Turns out that EPEL's RPM was compiled without shared library support.&nbsp; Trying to run the software even directly on the worker node gave the following nasty message: Error while loading shared libraries: libgmxpreprocess_openmpi.so.6: cannot open shared object file.&nbsp; Manually adding the library to the LD_LIBRARY environment variable fixed the problem which should allow one to submit jobs to the scheduler.&nbsp; Right?Well technically yes.&nbsp; However the jobs still failed because of the shared library problem.&nbsp; Exporting the environment variable in the shell script or even placing it in the profile didn't seem to work.&nbsp; We're not quite sure why but it may have something to do with how mpirun connects to the worker nodes.&nbsp; Feeling a bit desperate now we pulled a sneaky trick and used mpirun's environment argument -x to set the LD_LIBRARY variable.Finally success.&nbsp; Jobs start on multiple cores, ignore the -nt argument, and the number of threads started equals the -np argument, correctly spread over the available cores in the cluster.&nbsp; There is only one log file, one topology file and one energy file, as expected.&nbsp; The log file indicates that there are a number of nodes being used equivalent to the -np argument.&nbsp; Running on more nodes also seems to introduce a significant speed improvement.&nbsp; The only fly in the ointment seems to be the visualization files with two of them being updated simultaneously.&nbsp; These are meant to be used with the Grace visualization package, are not compulsory and can even be disabled.Running a job now shows how rapidly the threads are distributed over the cores in the following image, snapshots taken 10 seconds apart.More investigation is still to be done in looking at the efficiency of multiple threads, for instance is the relationship between threads and job speed inversely geometric as expected and if so what is the minimum number of cores that will achieve at least a 90% speed improvement?&nbsp; The communication latency between nodes will also be examined as multiple cores seem to consume considerable bandwidth.","og_url":"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/04\/22\/getting-gromacs\/","og_site_name":"UCT HPC","article_published_time":"2011-04-22T10:33:48+00:00","article_modified_time":"2022-09-26T18:04:47+00:00","og_image":[{"url":"https:\/\/ucthpc.uct.ac.za\/wp-content\/uploads\/2015\/07\/gromacs.png"}],"author":"Andrew Lewis","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Andrew Lewis","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/04\/22\/getting-gromacs\/#article","isPartOf":{"@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/04\/22\/getting-gromacs\/"},"author":{"name":"Andrew Lewis","@id":"https:\/\/ucthpc.uct.ac.za\/#\/schema\/person\/c183ad1c0a1063124a72d63963ae9c7e"},"headline":"Getting Gromacs","datePublished":"2011-04-22T10:33:48+00:00","dateModified":"2022-09-26T18:04:47+00:00","mainEntityOfPage":{"@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/04\/22\/getting-gromacs\/"},"wordCount":834,"publisher":{"@id":"https:\/\/ucthpc.uct.ac.za\/#organization"},"articleSection":["application","hpc","MPI"],"inLanguage":"en-ZA"},{"@type":"WebPage","@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/04\/22\/getting-gromacs\/","url":"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/04\/22\/getting-gromacs\/","name":"Getting Gromacs - UCT HPC","isPartOf":{"@id":"https:\/\/ucthpc.uct.ac.za\/#website"},"datePublished":"2011-04-22T10:33:48+00:00","dateModified":"2022-09-26T18:04:47+00:00","breadcrumb":{"@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/04\/22\/getting-gromacs\/#breadcrumb"},"inLanguage":"en-ZA","potentialAction":[{"@type":"ReadAction","target":["https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/04\/22\/getting-gromacs\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/04\/22\/getting-gromacs\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/ucthpc.uct.ac.za\/"},{"@type":"ListItem","position":2,"name":"Getting Gromacs"}]},{"@type":"WebSite","@id":"https:\/\/ucthpc.uct.ac.za\/#website","url":"https:\/\/ucthpc.uct.ac.za\/","name":"UCT HPC","description":"University of Cape Town High Performance Computing","publisher":{"@id":"https:\/\/ucthpc.uct.ac.za\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/ucthpc.uct.ac.za\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-ZA"},{"@type":"Organization","@id":"https:\/\/ucthpc.uct.ac.za\/#organization","name":"University of Cape Town High Performance Computing","url":"https:\/\/ucthpc.uct.ac.za\/","logo":{"@type":"ImageObject","inLanguage":"en-ZA","@id":"https:\/\/ucthpc.uct.ac.za\/#\/schema\/logo\/image\/","url":"https:\/\/ucthpc.uct.ac.za\/wp-content\/uploads\/2015\/09\/logocircless.png","contentUrl":"https:\/\/ucthpc.uct.ac.za\/wp-content\/uploads\/2015\/09\/logocircless.png","width":450,"height":423,"caption":"University of Cape Town High Performance Computing"},"image":{"@id":"https:\/\/ucthpc.uct.ac.za\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/ucthpc.uct.ac.za\/#\/schema\/person\/c183ad1c0a1063124a72d63963ae9c7e","name":"Andrew Lewis","image":{"@type":"ImageObject","inLanguage":"en-ZA","@id":"https:\/\/ucthpc.uct.ac.za\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/9652c9c73beeab594b8dc2383a880048?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/9652c9c73beeab594b8dc2383a880048?s=96&d=mm&r=g","caption":"Andrew Lewis"},"sameAs":["http:\/\/blogs.uct.ac.za\/blog\/big-bytes"]}]}},"_links":{"self":[{"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/posts\/1122"}],"collection":[{"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/comments?post=1122"}],"version-history":[{"count":5,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/posts\/1122\/revisions"}],"predecessor-version":[{"id":4289,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/posts\/1122\/revisions\/4289"}],"wp:attachment":[{"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/media?parent=1122"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/categories?post=1122"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/tags?post=1122"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}