{"id":1071,"date":"2011-07-29T15:24:15","date_gmt":"2011-07-29T13:24:15","guid":{"rendered":"http:\/\/oldblogs.uct.ac.za\/blog\/big-bytes\/2011\/07\/29\/lusting-after-glusterfs"},"modified":"2015-08-14T13:23:18","modified_gmt":"2015-08-14T11:23:18","slug":"lusting-after-glusterfs","status":"publish","type":"post","link":"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/07\/29\/lusting-after-glusterfs\/","title":{"rendered":"Lusting after GlusterFS"},"content":{"rendered":"

GlusterFS is an open source,\r\ndistributed file system capable of scaling to several petabytes and\r\nhandling thousands of clients. The HPC team at UCT had some time to\r\nplay and implement Gluster. The idea behind Gluster is to scale the\r\navailable space of servers into a pool which can be used to store\r\ndata of (n)size. A general first impression of Gluster was the\r\nimpressive CLI with very simple execution commands. On a more\r\ntechnical angle, Gluster is the only distributed file system that I\r\nknow which has no requirement for metadata servers \/ chunk servers or\r\nlogger servers.<\/p>\r\n

The architecture aggregates its storage\r\ninto a global namespace. Gluster makes use of the FUSE (Filesystem in\r\nUserspace ) module which allows for non-privileged users to create\r\ntheir own file systems.<\/p>\r\n

Configuration commenced by creating a\r\ndistributed striped volume across 8 servers with a TCP, 1gigE network.\r\nThe internal disks which are mirrored. This is a good thing as the\r\ndistributed striped volume has no redundancy.<\/p>\r\n

<\/p>\r\n

Write speeds\r\n<\/strong><\/p>\r\n

dd if=\/dev\/zero\r\nof=\/scratch01\/dd-test.dd bs=10235\r\n<\/span><\/p>\r\n

47286+0 records in\r\n<\/span><\/p>\r\n

47286+0 records out\r\n<\/span><\/p>\r\n

483972210 bytes (484 MB) copied,\r\n4.90815 seconds, 98.6 MB\/s\r\n<\/span><\/p>\r\n

Read speeds<\/strong><\/p>\r\n

dd if=\/scratch01\/dd-test.dd of=\/dev\/null<\/span><\/p>\r\n

349409+0 records in\r\n<\/span><\/p>\r\n

349408+0 records out\r\n<\/span><\/p>\r\n

178896896 bytes (179 MB) copied,\r\n4.99041 seconds, 35.8 MB\/s\r\n<\/span><\/p>\r\n

<\/p>\r\n

One of the reasons why the read speed\r\nis less than the write speed was caused by a bug in Gluster which we\r\nhave identified and submitted to Gluster Dev support here<\/a> and they in return\r\nhave released a fix for it here<\/a>.<\/p>\r\n

The bug surfaced when\r\ncreating a stripe volume and data written to the FUSE mount was read\r\nback in binary format. Using a tool called \u201cod\u201d<\/strong> with the \"-x\"<\/strong> option we were able to\r\nidentify the inconsistencies and reported the bug to Gluster. The fix\r\nhas been incorporated into the next release Gluster 3.3.The workaround was to disable the\r\nperformance.quick-read setting under the volume being accessed.<\/p>\r\n

To\r\napply the workaround \u201c gluster volume set <vol_name>\r\nperformance.quick-read off \u201c<\/strong> . However this may be the reason for\r\nthe slower read speed we receive when reading data from the FUSE\r\nmount into \/dev\/null. We will update this blog with a post stripe.c\r\nfix update and enable the quick read option. This option is enabled\r\nby default when creating a volume.<\/p>","protected":false},"excerpt":{"rendered":"

GlusterFS is an open source,
\ndistributed file system capable of scaling to several petabytes and
\nhandling thousands of clients. The HPC team at UCT had some time to
\nplay and implement Gluster. The idea behind Gluster is to scale the
\navailable space of servers into a pool which can be used to store
\ndata of (n)size. A general first impression of Gluster was the
\nimpressive CLI with very simple execution commands. On a more
\ntechnical angle, Gluster is the only distributed file system that I
\nknow which has no requirement for metadata servers \/ chunk servers or
\nlogger servers.\n<\/p>\n

The architecture aggregates its storage
\ninto a global namespace. Gluster makes use of the FUSE (Filesystem in
\nUserspace ) module which allows for non-privileged users to create
\ntheir own file systems.\n<\/p>\n

Configuration commenced by creating a
\ndistributed striped volume across 8 servers with a TCP, 1gigE network.
\nThe internal disks which are mirrored. This is a good thing as the
\ndistributed striped volume has no redundancy.<\/p>\n

 <\/p>\n

Write speeds
\n<\/strong><\/p>\n

dd if=\/dev\/zero
\nof=\/scratch01\/dd-test.dd bs=10235
\n<\/span><\/p>\n

47286+0 records in
\n<\/span><\/p>\n

47286+0 records out
\n<\/span><\/p>\n

483972210 bytes (484 MB) copied,
\n4.90815 seconds, 98.6 MB\/s
\n<\/span><\/p>\n

Read speeds<\/strong><\/p>\n

dd if=\/scratch01\/dd-test.dd of=\/dev\/null<\/span><\/p>\n

349409+0 records in
\n<\/span><\/p>\n

349408+0 records out
\n<\/span><\/p>\n

178896896 bytes (179 MB) copied,
\n4.99041 seconds, 35.8 MB\/s
\n<\/span><\/p>\n

 <\/p>\n

One of the reasons why the read speed
\nis less than the write speed was caused by a bug in Gluster which we
\nhave identified and submitted to Gluster Dev support here<\/a> and they in return
\nhave released a fix for it here<\/a>. <\/p>\n

The bug surfaced when
\ncreating a stripe volume and data written to the FUSE mount was read
\nback in binary format. Using a tool called “od”<\/strong> with the “-x”<\/strong> option we were able to
\nidentify the inconsistencies and reported the bug to Gluster. The fix
\nhas been incorporated into the next release Gluster 3.3.The workaround was to disable the
\nperformance.quick-read setting under the volume being accessed. <\/p>\n

To
\napply the workaround “ gluster volume set <vol_name>
\nperformance.quick-read off “<\/strong> . However this may be the reason for
\nthe slower read speed we receive when reading data from the FUSE
\nmount into \/dev\/null. We will update this blog with a post stripe.c
\nfix update and enable the quick read option. This option is enabled
\nby default when creating a volume. <\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[4],"tags":[],"yoast_head":"\nLusting after GlusterFS - UCT HPC<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/07\/29\/lusting-after-glusterfs\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Lusting after GlusterFS - UCT HPC\" \/>\n<meta property=\"og:description\" content=\"GlusterFS is an open source, distributed file system capable of scaling to several petabytes and handling thousands of clients. The HPC team at UCT had some time to play and implement Gluster. The idea behind Gluster is to scale the available space of servers into a pool which can be used to store data of (n)size. A general first impression of Gluster was the impressive CLI with very simple execution commands. On a more technical angle, Gluster is the only distributed file system that I know which has no requirement for metadata servers \/ chunk servers or logger servers. The architecture aggregates its storage into a global namespace. Gluster makes use of the FUSE (Filesystem in Userspace ) module which allows for non-privileged users to create their own file systems. Configuration commenced by creating a distributed striped volume across 8 servers with a TCP, 1gigE network. The internal disks which are mirrored. This is a good thing as the distributed striped volume has no redundancy. Write speeds dd if=\/dev\/zero of=\/scratch01\/dd-test.dd bs=10235 47286+0 records in 47286+0 records out 483972210 bytes (484 MB) copied, 4.90815 seconds, 98.6 MB\/s Read speedsdd if=\/scratch01\/dd-test.dd of=\/dev\/null 349409+0 records in 349408+0 records out 178896896 bytes (179 MB) copied, 4.99041 seconds, 35.8 MB\/s   One of the reasons why the read speed is less than the write speed was caused by a bug in Gluster which we have identified and submitted to Gluster Dev support here and they in return have released a fix for it here. The bug surfaced when creating a stripe volume and data written to the FUSE mount was read back in binary format. Using a tool called “od” with the "-x" option we were able to identify the inconsistencies and reported the bug to Gluster. The fix has been incorporated into the next release Gluster 3.3.The workaround was to disable the performance.quick-read setting under the volume being accessed. To apply the workaround “ gluster volume set <vol_name> performance.quick-read off “ . However this may be the reason for the slower read speed we receive when reading data from the FUSE mount into \/dev\/null. We will update this blog with a post stripe.c fix update and enable the quick read option. This option is enabled by default when creating a volume.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/07\/29\/lusting-after-glusterfs\/\" \/>\n<meta property=\"og:site_name\" content=\"UCT HPC\" \/>\n<meta property=\"article:published_time\" content=\"2011-07-29T13:24:15+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2015-08-14T11:23:18+00:00\" \/>\n<meta name=\"author\" content=\"Timothy Carr\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Timothy Carr\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/07\/29\/lusting-after-glusterfs\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/07\/29\/lusting-after-glusterfs\/\"},\"author\":{\"name\":\"Timothy Carr\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#\/schema\/person\/41f6cd039836d7741f2b82a7b7cfe8d0\"},\"headline\":\"Lusting after GlusterFS\",\"datePublished\":\"2011-07-29T13:24:15+00:00\",\"dateModified\":\"2015-08-14T11:23:18+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/07\/29\/lusting-after-glusterfs\/\"},\"wordCount\":382,\"publisher\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#organization\"},\"articleSection\":[\"hpc\"],\"inLanguage\":\"en-ZA\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/07\/29\/lusting-after-glusterfs\/\",\"url\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/07\/29\/lusting-after-glusterfs\/\",\"name\":\"Lusting after GlusterFS - UCT HPC\",\"isPartOf\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#website\"},\"datePublished\":\"2011-07-29T13:24:15+00:00\",\"dateModified\":\"2015-08-14T11:23:18+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/07\/29\/lusting-after-glusterfs\/#breadcrumb\"},\"inLanguage\":\"en-ZA\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/07\/29\/lusting-after-glusterfs\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/07\/29\/lusting-after-glusterfs\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/ucthpc.uct.ac.za\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Lusting after GlusterFS\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#website\",\"url\":\"https:\/\/ucthpc.uct.ac.za\/\",\"name\":\"UCT HPC\",\"description\":\"University of Cape Town High Performance Computing\",\"publisher\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/ucthpc.uct.ac.za\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-ZA\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#organization\",\"name\":\"University of Cape Town High Performance Computing\",\"url\":\"https:\/\/ucthpc.uct.ac.za\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-ZA\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/ucthpc.uct.ac.za\/wp-content\/uploads\/2015\/09\/logocircless.png\",\"contentUrl\":\"https:\/\/ucthpc.uct.ac.za\/wp-content\/uploads\/2015\/09\/logocircless.png\",\"width\":450,\"height\":423,\"caption\":\"University of Cape Town High Performance Computing\"},\"image\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#\/schema\/person\/41f6cd039836d7741f2b82a7b7cfe8d0\",\"name\":\"Timothy Carr\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-ZA\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/7e94dcf3a408e6ada008042fc29d4b15?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/7e94dcf3a408e6ada008042fc29d4b15?s=96&d=mm&r=g\",\"caption\":\"Timothy Carr\"},\"sameAs\":[\"http:\/\/ucthpc.uct.ac.za\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Lusting after GlusterFS - UCT HPC","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/07\/29\/lusting-after-glusterfs\/","og_locale":"en_US","og_type":"article","og_title":"Lusting after GlusterFS - UCT HPC","og_description":"GlusterFS is an open source, distributed file system capable of scaling to several petabytes and handling thousands of clients. The HPC team at UCT had some time to play and implement Gluster. The idea behind Gluster is to scale the available space of servers into a pool which can be used to store data of (n)size. A general first impression of Gluster was the impressive CLI with very simple execution commands. On a more technical angle, Gluster is the only distributed file system that I know which has no requirement for metadata servers \/ chunk servers or logger servers. The architecture aggregates its storage into a global namespace. Gluster makes use of the FUSE (Filesystem in Userspace ) module which allows for non-privileged users to create their own file systems. Configuration commenced by creating a distributed striped volume across 8 servers with a TCP, 1gigE network. The internal disks which are mirrored. This is a good thing as the distributed striped volume has no redundancy. Write speeds dd if=\/dev\/zero of=\/scratch01\/dd-test.dd bs=10235 47286+0 records in 47286+0 records out 483972210 bytes (484 MB) copied, 4.90815 seconds, 98.6 MB\/s Read speedsdd if=\/scratch01\/dd-test.dd of=\/dev\/null 349409+0 records in 349408+0 records out 178896896 bytes (179 MB) copied, 4.99041 seconds, 35.8 MB\/s   One of the reasons why the read speed is less than the write speed was caused by a bug in Gluster which we have identified and submitted to Gluster Dev support here and they in return have released a fix for it here. The bug surfaced when creating a stripe volume and data written to the FUSE mount was read back in binary format. Using a tool called “od” with the \"-x\" option we were able to identify the inconsistencies and reported the bug to Gluster. The fix has been incorporated into the next release Gluster 3.3.The workaround was to disable the performance.quick-read setting under the volume being accessed. To apply the workaround “ gluster volume set <vol_name> performance.quick-read off “ . However this may be the reason for the slower read speed we receive when reading data from the FUSE mount into \/dev\/null. We will update this blog with a post stripe.c fix update and enable the quick read option. This option is enabled by default when creating a volume.","og_url":"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/07\/29\/lusting-after-glusterfs\/","og_site_name":"UCT HPC","article_published_time":"2011-07-29T13:24:15+00:00","article_modified_time":"2015-08-14T11:23:18+00:00","author":"Timothy Carr","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Timothy Carr","Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/07\/29\/lusting-after-glusterfs\/#article","isPartOf":{"@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/07\/29\/lusting-after-glusterfs\/"},"author":{"name":"Timothy Carr","@id":"https:\/\/ucthpc.uct.ac.za\/#\/schema\/person\/41f6cd039836d7741f2b82a7b7cfe8d0"},"headline":"Lusting after GlusterFS","datePublished":"2011-07-29T13:24:15+00:00","dateModified":"2015-08-14T11:23:18+00:00","mainEntityOfPage":{"@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/07\/29\/lusting-after-glusterfs\/"},"wordCount":382,"publisher":{"@id":"https:\/\/ucthpc.uct.ac.za\/#organization"},"articleSection":["hpc"],"inLanguage":"en-ZA"},{"@type":"WebPage","@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/07\/29\/lusting-after-glusterfs\/","url":"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/07\/29\/lusting-after-glusterfs\/","name":"Lusting after GlusterFS - UCT HPC","isPartOf":{"@id":"https:\/\/ucthpc.uct.ac.za\/#website"},"datePublished":"2011-07-29T13:24:15+00:00","dateModified":"2015-08-14T11:23:18+00:00","breadcrumb":{"@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/07\/29\/lusting-after-glusterfs\/#breadcrumb"},"inLanguage":"en-ZA","potentialAction":[{"@type":"ReadAction","target":["https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/07\/29\/lusting-after-glusterfs\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/2011\/07\/29\/lusting-after-glusterfs\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/ucthpc.uct.ac.za\/"},{"@type":"ListItem","position":2,"name":"Lusting after GlusterFS"}]},{"@type":"WebSite","@id":"https:\/\/ucthpc.uct.ac.za\/#website","url":"https:\/\/ucthpc.uct.ac.za\/","name":"UCT HPC","description":"University of Cape Town High Performance Computing","publisher":{"@id":"https:\/\/ucthpc.uct.ac.za\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/ucthpc.uct.ac.za\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-ZA"},{"@type":"Organization","@id":"https:\/\/ucthpc.uct.ac.za\/#organization","name":"University of Cape Town High Performance Computing","url":"https:\/\/ucthpc.uct.ac.za\/","logo":{"@type":"ImageObject","inLanguage":"en-ZA","@id":"https:\/\/ucthpc.uct.ac.za\/#\/schema\/logo\/image\/","url":"https:\/\/ucthpc.uct.ac.za\/wp-content\/uploads\/2015\/09\/logocircless.png","contentUrl":"https:\/\/ucthpc.uct.ac.za\/wp-content\/uploads\/2015\/09\/logocircless.png","width":450,"height":423,"caption":"University of Cape Town High Performance Computing"},"image":{"@id":"https:\/\/ucthpc.uct.ac.za\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/ucthpc.uct.ac.za\/#\/schema\/person\/41f6cd039836d7741f2b82a7b7cfe8d0","name":"Timothy Carr","image":{"@type":"ImageObject","inLanguage":"en-ZA","@id":"https:\/\/ucthpc.uct.ac.za\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/7e94dcf3a408e6ada008042fc29d4b15?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/7e94dcf3a408e6ada008042fc29d4b15?s=96&d=mm&r=g","caption":"Timothy Carr"},"sameAs":["http:\/\/ucthpc.uct.ac.za"]}]}},"_links":{"self":[{"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/posts\/1071"}],"collection":[{"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/comments?post=1071"}],"version-history":[{"count":2,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/posts\/1071\/revisions"}],"predecessor-version":[{"id":2221,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/posts\/1071\/revisions\/2221"}],"wp:attachment":[{"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/media?parent=1071"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/categories?post=1071"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/tags?post=1071"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}