{"id":811,"date":"2013-10-16T21:56:55","date_gmt":"2013-10-16T19:56:55","guid":{"rendered":"http:\/\/oldblogs.uct.ac.za\/blog\/big-bytes\/2013\/10\/16\/sam-really-really-wants-to-take-out-the-trash"},"modified":"2022-09-26T20:31:12","modified_gmt":"2022-09-26T18:31:12","slug":"sam-really-really-wants-to-take-out-the-trash","status":"publish","type":"post","link":"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/10\/16\/sam-really-really-wants-to-take-out-the-trash\/","title":{"rendered":"SAM really, really wants to take out the trash"},"content":{"rendered":"We encountered an issue where a user's jobs used far more cpus than allocated.\u00a0 This occured even though ppn was set to 1, the server load went up to 400+ and the individual processes were using up to 4000% CPU time.\"sam\"\r\n\r\nClearly a sub-optimal state.\r\n\r\nWe tracked this down to a java script, which was odd as it should run serial by default.\u00a0 Even stranger was the script,\u00a0SortSam<\/a>, a small, non parallel script for sorting bam files for GATK read-pair walkers.\u00a0 Googling against the script name provided the solution<\/a>.\r\n\r\nThe problem was when Garbage Collection ran java detected it was running on a 64 bit OS and the JVM then ran in server mode, grabbing all the cores, i.e. all 50 jobs ran GC on all 64 cores, and that doesn't even include all the MatLab, Gromacs and NAMD jobs that other users were trying to run. \u00a0We're pretty sure that we could have timed this condition by looking at the data centre temperature graphs.\r\n\r\nSetting -XX:ParallelGCThreads=1 solved this issue.","protected":false},"excerpt":{"rendered":"

We encountered an issue where a user’s jobs used far more cpus than allocated.  This occured even though ppn was set to 1, the server load went up to 400+ and the individual processes were using up to 4000% CPU time.\"sam\"<\/p>\n

Clearly a sub-optimal state.<\/p>\n

We tracked this down to a java script, which was odd as it should run serial by default.  Even stranger was the script, SortSam<\/a>, a small, non parallel script for sorting bam files for GATK read-pair walkers.  Googling against the script name provided the solution<\/a>.<\/p>\n

<\/a>The problem was when Garbage Collection ran java detected it was running on a 64 bit OS and the JVM then ran in server mode, grabbing all the cores, i.e. all 50 jobs ran GC on all 64 cores, and that doesn’t even include all the MatLab, Gromacs and NAMD jobs that other users were trying to run.  We’re pretty sure that we could have timed this condition by looking at the data centre temperature graphs.<\/p>\n

Setting -XX:ParallelGCThreads=1 solved this issue.<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[4],"tags":[],"yoast_head":"\nSAM really, really wants to take out the trash - UCT HPC<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/10\/16\/sam-really-really-wants-to-take-out-the-trash\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"SAM really, really wants to take out the trash - UCT HPC\" \/>\n<meta property=\"og:description\" content=\"We encountered an issue where a user's jobs used far more cpus than allocated.  This occured even though ppn was set to 1, the server load went up to 400+ and the individual processes were using up to 4000% CPU time.Clearly a sub-optimal state.We tracked this down to a java script, which was odd as it should run serial by default.  Even stranger was the script, SortSam, a small, non parallel script for sorting bam files for GATK read-pair walkers.  Googling against the script name provided the solution.The problem was when Garbage Collection ran java detected it was running on a 64 bit OS and the JVM then ran in server mode, grabbing all the cores, i.e. all 50 jobs ran GC on all 64 cores, and that doesn't even include all the MatLab, Gromacs and NAMD jobs that other users were trying to run.  We're pretty sure that we could have timed this condition by looking at the data centre temperature graphs.Setting -XX:ParallelGCThreads=1 solved this issue.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/10\/16\/sam-really-really-wants-to-take-out-the-trash\/\" \/>\n<meta property=\"og:site_name\" content=\"UCT HPC\" \/>\n<meta property=\"article:published_time\" content=\"2013-10-16T19:56:55+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2022-09-26T18:31:12+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/ucthpc.uct.ac.za\/wp-content\/uploads\/2015\/07\/SamSort.jpg\" \/>\n<meta name=\"author\" content=\"Andrew Lewis\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Andrew Lewis\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/10\/16\/sam-really-really-wants-to-take-out-the-trash\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/10\/16\/sam-really-really-wants-to-take-out-the-trash\/\"},\"author\":{\"name\":\"Andrew Lewis\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#\/schema\/person\/c183ad1c0a1063124a72d63963ae9c7e\"},\"headline\":\"SAM really, really wants to take out the trash\",\"datePublished\":\"2013-10-16T19:56:55+00:00\",\"dateModified\":\"2022-09-26T18:31:12+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/10\/16\/sam-really-really-wants-to-take-out-the-trash\/\"},\"wordCount\":176,\"publisher\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#organization\"},\"articleSection\":[\"hpc\"],\"inLanguage\":\"en-ZA\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/10\/16\/sam-really-really-wants-to-take-out-the-trash\/\",\"url\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/10\/16\/sam-really-really-wants-to-take-out-the-trash\/\",\"name\":\"SAM really, really wants to take out the trash - UCT HPC\",\"isPartOf\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#website\"},\"datePublished\":\"2013-10-16T19:56:55+00:00\",\"dateModified\":\"2022-09-26T18:31:12+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/10\/16\/sam-really-really-wants-to-take-out-the-trash\/#breadcrumb\"},\"inLanguage\":\"en-ZA\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/10\/16\/sam-really-really-wants-to-take-out-the-trash\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/10\/16\/sam-really-really-wants-to-take-out-the-trash\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/ucthpc.uct.ac.za\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"SAM really, really wants to take out the trash\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#website\",\"url\":\"https:\/\/ucthpc.uct.ac.za\/\",\"name\":\"UCT HPC\",\"description\":\"University of Cape Town High Performance Computing\",\"publisher\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/ucthpc.uct.ac.za\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-ZA\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#organization\",\"name\":\"University of Cape Town High Performance Computing\",\"url\":\"https:\/\/ucthpc.uct.ac.za\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-ZA\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/ucthpc.uct.ac.za\/wp-content\/uploads\/2015\/09\/logocircless.png\",\"contentUrl\":\"https:\/\/ucthpc.uct.ac.za\/wp-content\/uploads\/2015\/09\/logocircless.png\",\"width\":450,\"height\":423,\"caption\":\"University of Cape Town High Performance Computing\"},\"image\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#\/schema\/person\/c183ad1c0a1063124a72d63963ae9c7e\",\"name\":\"Andrew Lewis\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-ZA\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/9652c9c73beeab594b8dc2383a880048?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/9652c9c73beeab594b8dc2383a880048?s=96&d=mm&r=g\",\"caption\":\"Andrew Lewis\"},\"sameAs\":[\"http:\/\/blogs.uct.ac.za\/blog\/big-bytes\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"SAM really, really wants to take out the trash - UCT HPC","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/10\/16\/sam-really-really-wants-to-take-out-the-trash\/","og_locale":"en_US","og_type":"article","og_title":"SAM really, really wants to take out the trash - UCT HPC","og_description":"We encountered an issue where a user's jobs used far more cpus than allocated.  This occured even though ppn was set to 1, the server load went up to 400+ and the individual processes were using up to 4000% CPU time.Clearly a sub-optimal state.We tracked this down to a java script, which was odd as it should run serial by default.  Even stranger was the script, SortSam, a small, non parallel script for sorting bam files for GATK read-pair walkers.  Googling against the script name provided the solution.The problem was when Garbage Collection ran java detected it was running on a 64 bit OS and the JVM then ran in server mode, grabbing all the cores, i.e. all 50 jobs ran GC on all 64 cores, and that doesn't even include all the MatLab, Gromacs and NAMD jobs that other users were trying to run.  We're pretty sure that we could have timed this condition by looking at the data centre temperature graphs.Setting -XX:ParallelGCThreads=1 solved this issue.","og_url":"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/10\/16\/sam-really-really-wants-to-take-out-the-trash\/","og_site_name":"UCT HPC","article_published_time":"2013-10-16T19:56:55+00:00","article_modified_time":"2022-09-26T18:31:12+00:00","og_image":[{"url":"https:\/\/ucthpc.uct.ac.za\/wp-content\/uploads\/2015\/07\/SamSort.jpg"}],"author":"Andrew Lewis","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Andrew Lewis","Est. reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/10\/16\/sam-really-really-wants-to-take-out-the-trash\/#article","isPartOf":{"@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/10\/16\/sam-really-really-wants-to-take-out-the-trash\/"},"author":{"name":"Andrew Lewis","@id":"https:\/\/ucthpc.uct.ac.za\/#\/schema\/person\/c183ad1c0a1063124a72d63963ae9c7e"},"headline":"SAM really, really wants to take out the trash","datePublished":"2013-10-16T19:56:55+00:00","dateModified":"2022-09-26T18:31:12+00:00","mainEntityOfPage":{"@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/10\/16\/sam-really-really-wants-to-take-out-the-trash\/"},"wordCount":176,"publisher":{"@id":"https:\/\/ucthpc.uct.ac.za\/#organization"},"articleSection":["hpc"],"inLanguage":"en-ZA"},{"@type":"WebPage","@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/10\/16\/sam-really-really-wants-to-take-out-the-trash\/","url":"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/10\/16\/sam-really-really-wants-to-take-out-the-trash\/","name":"SAM really, really wants to take out the trash - UCT HPC","isPartOf":{"@id":"https:\/\/ucthpc.uct.ac.za\/#website"},"datePublished":"2013-10-16T19:56:55+00:00","dateModified":"2022-09-26T18:31:12+00:00","breadcrumb":{"@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/10\/16\/sam-really-really-wants-to-take-out-the-trash\/#breadcrumb"},"inLanguage":"en-ZA","potentialAction":[{"@type":"ReadAction","target":["https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/10\/16\/sam-really-really-wants-to-take-out-the-trash\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/2013\/10\/16\/sam-really-really-wants-to-take-out-the-trash\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/ucthpc.uct.ac.za\/"},{"@type":"ListItem","position":2,"name":"SAM really, really wants to take out the trash"}]},{"@type":"WebSite","@id":"https:\/\/ucthpc.uct.ac.za\/#website","url":"https:\/\/ucthpc.uct.ac.za\/","name":"UCT HPC","description":"University of Cape Town High Performance Computing","publisher":{"@id":"https:\/\/ucthpc.uct.ac.za\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/ucthpc.uct.ac.za\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-ZA"},{"@type":"Organization","@id":"https:\/\/ucthpc.uct.ac.za\/#organization","name":"University of Cape Town High Performance Computing","url":"https:\/\/ucthpc.uct.ac.za\/","logo":{"@type":"ImageObject","inLanguage":"en-ZA","@id":"https:\/\/ucthpc.uct.ac.za\/#\/schema\/logo\/image\/","url":"https:\/\/ucthpc.uct.ac.za\/wp-content\/uploads\/2015\/09\/logocircless.png","contentUrl":"https:\/\/ucthpc.uct.ac.za\/wp-content\/uploads\/2015\/09\/logocircless.png","width":450,"height":423,"caption":"University of Cape Town High Performance Computing"},"image":{"@id":"https:\/\/ucthpc.uct.ac.za\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/ucthpc.uct.ac.za\/#\/schema\/person\/c183ad1c0a1063124a72d63963ae9c7e","name":"Andrew Lewis","image":{"@type":"ImageObject","inLanguage":"en-ZA","@id":"https:\/\/ucthpc.uct.ac.za\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/9652c9c73beeab594b8dc2383a880048?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/9652c9c73beeab594b8dc2383a880048?s=96&d=mm&r=g","caption":"Andrew Lewis"},"sameAs":["http:\/\/blogs.uct.ac.za\/blog\/big-bytes"]}]}},"_links":{"self":[{"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/posts\/811"}],"collection":[{"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/comments?post=811"}],"version-history":[{"count":4,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/posts\/811\/revisions"}],"predecessor-version":[{"id":4346,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/posts\/811\/revisions\/4346"}],"wp:attachment":[{"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/media?parent=811"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/categories?post=811"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/tags?post=811"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}