{"id":1017,"date":"2011-11-03T16:01:23","date_gmt":"2011-11-03T14:01:23","guid":{"rendered":"http:\/\/oldblogs.uct.ac.za\/blog\/big-bytes\/2011\/11\/03\/gluster-wobble"},"modified":"2015-08-14T13:18:12","modified_gmt":"2015-08-14T11:18:12","slug":"gluster-wobble","status":"publish","type":"post","link":"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/11\/03\/gluster-wobble\/","title":{"rendered":"Gluster wobble"},"content":{"rendered":"So we've been making use of GlusterFS for a while now and generally it's been great.\u00a0 Gluster allows us to present unused space on a number of networked servers (our HPC worker nodes) as a single disk pool and make this available to researchers.\u00a0 We currently have two scratch areas, each one made up from a respective group of worker nodes; the 200 series join together to make scratch01 and the 400 series join together to make scratch02.\r\n\r\nWe learned a while back that it's improtant to get disk striping correct.\u00a0 When scratch02 was added the striping was set to 1, which meant that files were written to individual worker nodes causing the file systems to fill up consecutively (especially when large files were written) rather than all the worker node disks filling up simultaneously but far more slowly.\u00a0 Additionally in future iterations of HPC clusters we'll ensure that free space and OS areas are partitioned to avoid contention for critical file space.\r\n\r\nOur more recent issue was a bit more esoteric.\u00a0 Gluster is an abstratcion of disk space, but there is another layer of abstraction hidden from the users known as peering, where the gluster daemons on the worker nodes communicate amongst themselves to advertise availability and resources.\u00a0 The peering of our 200 and 400 series are intermingled, which means that the 200 and 400 series nodes are aware of each other at a peering level, even though they never interact.\u00a0 This\u00a0 theoretically is not a problem.\u00a0 However as in all things in life theory and reality can diverge and there are a number of learning points we'll be taking with us when we start working on the next iteration of our HPC cluster.\r\n\r\nThat being said, scratch02 is available again.\r\n\r\nThe error can be seen below and its location is \/var\/log\/glusterfs\/etc-glusterfs-glusterd.vol.log\r\n\r\nsocket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer<\/span>\r\n\r\nSince the 200 \/ 400 series peers are all considered as one gluster resource this makes it difficult to just restart volumes which are consumed from the 400 series bricks only. The solution was to unmount all volumes and restart the gluster daemon on all peers. After the restart the gluster volumes mounted successfully.\u00a0 We've since implemented daemon monitoring on all cluster nodes.","protected":false},"excerpt":{"rendered":"

So we’ve been making use of GlusterFS for a while now and generally it’s been great.  Gluster allows us to present unused space on a number of networked servers (our HPC worker nodes) as a single disk pool and make this available to researchers.  We currently have two scratch areas, each one made up from a respective group of worker nodes; the 200 series join together to make scratch01 and the 400 series join together to make scratch02.<\/p>\n

We learned a while back that it’s improtant to get disk striping correct.  When scratch02 was added the striping was set to 1, which meant that files were written to individual worker nodes causing the file systems to fill up consecutively (especially when large files were written) rather than all the worker node disks filling up simultaneously but far more slowly.  Additionally in future iterations of HPC clusters we’ll ensure that free space and OS areas are partitioned to avoid contention for critical file space.<\/p>\n

Our more recent issue was a bit more esoteric.  Gluster is an abstratcion of disk space, but there is another layer of abstraction hidden from the users known as peering, where the gluster daemons on the worker nodes communicate amongst themselves to advertise availability and resources.  The peering of our 200 and 400 series are intermingled, which means that the 200 and 400 series nodes are aware of each other at a peering level, even though they never interact.  This  theoretically is not a problem.  However as in all things in life theory and reality can diverge and there are a number of learning points we’ll be taking with us when we start working on the next iteration of our HPC cluster.<\/p>\n

That being said, scratch02 is available again.<\/p>\n

The error can be seen below and its location is \/var\/log\/glusterfs\/etc-glusterfs-glusterd.vol.log<\/p>\n

socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer<\/span><\/p>\n

Since the 200 \/ 400 series peers are all considered as one gluster resource this makes it difficult to just restart volumes which are consumed from the 400 series bricks only. The solution was to unmount all volumes and restart the gluster daemon on all peers. After the restart the gluster volumes mounted successfully.  We’ve since implemented daemon monitoring on all cluster nodes.<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[4],"tags":[],"yoast_head":"\nGluster wobble - UCT HPC<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/11\/03\/gluster-wobble\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Gluster wobble - UCT HPC\" \/>\n<meta property=\"og:description\" content=\"So we've been making use of GlusterFS for a while now and generally it's been great.  Gluster allows us to present unused space on a number of networked servers (our HPC worker nodes) as a single disk pool and make this available to researchers.  We currently have two scratch areas, each one made up from a respective group of worker nodes; the 200 series join together to make scratch01 and the 400 series join together to make scratch02.We learned a while back that it's improtant to get disk striping correct.  When scratch02 was added the striping was set to 1, which meant that files were written to individual worker nodes causing the file systems to fill up consecutively (especially when large files were written) rather than all the worker node disks filling up simultaneously but far more slowly.  Additionally in future iterations of HPC clusters we'll ensure that free space and OS areas are partitioned to avoid contention for critical file space.Our more recent issue was a bit more esoteric.  Gluster is an abstratcion of disk space, but there is another layer of abstraction hidden from the users known as peering, where the gluster daemons on the worker nodes communicate amongst themselves to advertise availability and resources.  The peering of our 200 and 400 series are intermingled, which means that the 200 and 400 series nodes are aware of each other at a peering level, even though they never interact.  This  theoretically is not a problem.  However as in all things in life theory and reality can diverge and there are a number of learning points we'll be taking with us when we start working on the next iteration of our HPC cluster.That being said, scratch02 is available again.The error can be seen below and its location is \/var\/log\/glusterfs\/etc-glusterfs-glusterd.vol.logsocket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peerSince the 200 \/ 400 series peers are all considered as one gluster resource this makes it difficult to just restart volumes which are consumed from the 400 series bricks only. The solution was to unmount all volumes and restart the gluster daemon on all peers. After the restart the gluster volumes mounted successfully.  We've since implemented daemon monitoring on all cluster nodes.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/11\/03\/gluster-wobble\/\" \/>\n<meta property=\"og:site_name\" content=\"UCT HPC\" \/>\n<meta property=\"article:published_time\" content=\"2011-11-03T14:01:23+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2015-08-14T11:18:12+00:00\" \/>\n<meta name=\"author\" content=\"Andrew Lewis\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Andrew Lewis\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/11\/03\/gluster-wobble\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/11\/03\/gluster-wobble\/\"},\"author\":{\"name\":\"Andrew Lewis\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#\/schema\/person\/c183ad1c0a1063124a72d63963ae9c7e\"},\"headline\":\"Gluster wobble\",\"datePublished\":\"2011-11-03T14:01:23+00:00\",\"dateModified\":\"2015-08-14T11:18:12+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/11\/03\/gluster-wobble\/\"},\"wordCount\":381,\"publisher\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#organization\"},\"articleSection\":[\"hpc\"],\"inLanguage\":\"en-ZA\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/11\/03\/gluster-wobble\/\",\"url\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/11\/03\/gluster-wobble\/\",\"name\":\"Gluster wobble - UCT HPC\",\"isPartOf\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#website\"},\"datePublished\":\"2011-11-03T14:01:23+00:00\",\"dateModified\":\"2015-08-14T11:18:12+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/11\/03\/gluster-wobble\/#breadcrumb\"},\"inLanguage\":\"en-ZA\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/11\/03\/gluster-wobble\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/11\/03\/gluster-wobble\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/ucthpc.uct.ac.za\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Gluster wobble\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#website\",\"url\":\"https:\/\/ucthpc.uct.ac.za\/\",\"name\":\"UCT HPC\",\"description\":\"University of Cape Town High Performance Computing\",\"publisher\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/ucthpc.uct.ac.za\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-ZA\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#organization\",\"name\":\"University of Cape Town High Performance Computing\",\"url\":\"https:\/\/ucthpc.uct.ac.za\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-ZA\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/ucthpc.uct.ac.za\/wp-content\/uploads\/2015\/09\/logocircless.png\",\"contentUrl\":\"https:\/\/ucthpc.uct.ac.za\/wp-content\/uploads\/2015\/09\/logocircless.png\",\"width\":450,\"height\":423,\"caption\":\"University of Cape Town High Performance Computing\"},\"image\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#\/schema\/person\/c183ad1c0a1063124a72d63963ae9c7e\",\"name\":\"Andrew Lewis\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-ZA\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/9652c9c73beeab594b8dc2383a880048?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/9652c9c73beeab594b8dc2383a880048?s=96&d=mm&r=g\",\"caption\":\"Andrew Lewis\"},\"sameAs\":[\"http:\/\/blogs.uct.ac.za\/blog\/big-bytes\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Gluster wobble - UCT HPC","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/11\/03\/gluster-wobble\/","og_locale":"en_US","og_type":"article","og_title":"Gluster wobble - UCT HPC","og_description":"So we've been making use of GlusterFS for a while now and generally it's been great.  Gluster allows us to present unused space on a number of networked servers (our HPC worker nodes) as a single disk pool and make this available to researchers.  We currently have two scratch areas, each one made up from a respective group of worker nodes; the 200 series join together to make scratch01 and the 400 series join together to make scratch02.We learned a while back that it's improtant to get disk striping correct.  When scratch02 was added the striping was set to 1, which meant that files were written to individual worker nodes causing the file systems to fill up consecutively (especially when large files were written) rather than all the worker node disks filling up simultaneously but far more slowly.  Additionally in future iterations of HPC clusters we'll ensure that free space and OS areas are partitioned to avoid contention for critical file space.Our more recent issue was a bit more esoteric.  Gluster is an abstratcion of disk space, but there is another layer of abstraction hidden from the users known as peering, where the gluster daemons on the worker nodes communicate amongst themselves to advertise availability and resources.  The peering of our 200 and 400 series are intermingled, which means that the 200 and 400 series nodes are aware of each other at a peering level, even though they never interact.  This  theoretically is not a problem.  However as in all things in life theory and reality can diverge and there are a number of learning points we'll be taking with us when we start working on the next iteration of our HPC cluster.That being said, scratch02 is available again.The error can be seen below and its location is \/var\/log\/glusterfs\/etc-glusterfs-glusterd.vol.logsocket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peerSince the 200 \/ 400 series peers are all considered as one gluster resource this makes it difficult to just restart volumes which are consumed from the 400 series bricks only. The solution was to unmount all volumes and restart the gluster daemon on all peers. After the restart the gluster volumes mounted successfully.  We've since implemented daemon monitoring on all cluster nodes.","og_url":"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/11\/03\/gluster-wobble\/","og_site_name":"UCT HPC","article_published_time":"2011-11-03T14:01:23+00:00","article_modified_time":"2015-08-14T11:18:12+00:00","author":"Andrew Lewis","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Andrew Lewis","Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/11\/03\/gluster-wobble\/#article","isPartOf":{"@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/11\/03\/gluster-wobble\/"},"author":{"name":"Andrew Lewis","@id":"https:\/\/ucthpc.uct.ac.za\/#\/schema\/person\/c183ad1c0a1063124a72d63963ae9c7e"},"headline":"Gluster wobble","datePublished":"2011-11-03T14:01:23+00:00","dateModified":"2015-08-14T11:18:12+00:00","mainEntityOfPage":{"@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/11\/03\/gluster-wobble\/"},"wordCount":381,"publisher":{"@id":"https:\/\/ucthpc.uct.ac.za\/#organization"},"articleSection":["hpc"],"inLanguage":"en-ZA"},{"@type":"WebPage","@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/11\/03\/gluster-wobble\/","url":"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/11\/03\/gluster-wobble\/","name":"Gluster wobble - UCT HPC","isPartOf":{"@id":"https:\/\/ucthpc.uct.ac.za\/#website"},"datePublished":"2011-11-03T14:01:23+00:00","dateModified":"2015-08-14T11:18:12+00:00","breadcrumb":{"@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/11\/03\/gluster-wobble\/#breadcrumb"},"inLanguage":"en-ZA","potentialAction":[{"@type":"ReadAction","target":["https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/11\/03\/gluster-wobble\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/11\/03\/gluster-wobble\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/ucthpc.uct.ac.za\/"},{"@type":"ListItem","position":2,"name":"Gluster wobble"}]},{"@type":"WebSite","@id":"https:\/\/ucthpc.uct.ac.za\/#website","url":"https:\/\/ucthpc.uct.ac.za\/","name":"UCT HPC","description":"University of Cape Town High Performance Computing","publisher":{"@id":"https:\/\/ucthpc.uct.ac.za\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/ucthpc.uct.ac.za\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-ZA"},{"@type":"Organization","@id":"https:\/\/ucthpc.uct.ac.za\/#organization","name":"University of Cape Town High Performance Computing","url":"https:\/\/ucthpc.uct.ac.za\/","logo":{"@type":"ImageObject","inLanguage":"en-ZA","@id":"https:\/\/ucthpc.uct.ac.za\/#\/schema\/logo\/image\/","url":"https:\/\/ucthpc.uct.ac.za\/wp-content\/uploads\/2015\/09\/logocircless.png","contentUrl":"https:\/\/ucthpc.uct.ac.za\/wp-content\/uploads\/2015\/09\/logocircless.png","width":450,"height":423,"caption":"University of Cape Town High Performance Computing"},"image":{"@id":"https:\/\/ucthpc.uct.ac.za\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/ucthpc.uct.ac.za\/#\/schema\/person\/c183ad1c0a1063124a72d63963ae9c7e","name":"Andrew Lewis","image":{"@type":"ImageObject","inLanguage":"en-ZA","@id":"https:\/\/ucthpc.uct.ac.za\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/9652c9c73beeab594b8dc2383a880048?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/9652c9c73beeab594b8dc2383a880048?s=96&d=mm&r=g","caption":"Andrew Lewis"},"sameAs":["http:\/\/blogs.uct.ac.za\/blog\/big-bytes"]}]}},"_links":{"self":[{"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/posts\/1017"}],"collection":[{"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/comments?post=1017"}],"version-history":[{"count":2,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/posts\/1017\/revisions"}],"predecessor-version":[{"id":2196,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/posts\/1017\/revisions\/2196"}],"wp:attachment":[{"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/media?parent=1017"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/categories?post=1017"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/tags?post=1017"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}