{"version":"1.0","provider_name":"UCT HPC","provider_url":"https:\/\/ucthpc.uct.ac.za","author_name":"Andrew Lewis","author_url":"https:\/\/ucthpc.uct.ac.za\/index.php\/author\/andrew-lewis\/","title":"AWS and MIT's StarCluster - UCT HPC","type":"rich","width":600,"height":338,"html":"<blockquote class=\"wp-embedded-content\" data-secret=\"hJBLizQPIZ\"><a href=\"https:\/\/ucthpc.uct.ac.za\/index.php\/2012\/11\/02\/aws-and-mits-starcluster\/\">AWS and MIT&#8217;s StarCluster<\/a><\/blockquote><iframe sandbox=\"allow-scripts\" security=\"restricted\" src=\"https:\/\/ucthpc.uct.ac.za\/index.php\/2012\/11\/02\/aws-and-mits-starcluster\/embed\/#?secret=hJBLizQPIZ\" width=\"600\" height=\"338\" title=\"&#8220;AWS and MIT&#8217;s StarCluster&#8221; &#8212; UCT HPC\" data-secret=\"hJBLizQPIZ\" frameborder=\"0\" marginwidth=\"0\" marginheight=\"0\" scrolling=\"no\" class=\"wp-embedded-content\"><\/iframe><script type=\"text\/javascript\">\n\/*! This file is auto-generated *\/\n!function(c,d){\"use strict\";var e=!1,o=!1;if(d.querySelector)if(c.addEventListener)e=!0;if(c.wp=c.wp||{},c.wp.receiveEmbedMessage);else if(c.wp.receiveEmbedMessage=function(e){var t=e.data;if(!t);else if(!(t.secret||t.message||t.value));else if(\/[^a-zA-Z0-9]\/.test(t.secret));else{for(var r,s,a,i=d.querySelectorAll('iframe[data-secret=\"'+t.secret+'\"]'),n=d.querySelectorAll('blockquote[data-secret=\"'+t.secret+'\"]'),o=new RegExp(\"^https?:$\",\"i\"),l=0;l<n.length;l++)n[l].style.display=\"none\";for(l=0;l<i.length;l++)if(r=i[l],e.source!==r.contentWindow);else{if(r.removeAttribute(\"style\"),\"height\"===t.message){if(1e3<(s=parseInt(t.value,10)))s=1e3;else if(~~s<200)s=200;r.height=s}if(\"link\"===t.message)if(s=d.createElement(\"a\"),a=d.createElement(\"a\"),s.href=r.getAttribute(\"src\"),a.href=t.value,!o.test(a.protocol));else if(a.host===s.host)if(d.activeElement===r)c.top.location.href=t.value}}},e)c.addEventListener(\"message\",c.wp.receiveEmbedMessage,!1),d.addEventListener(\"DOMContentLoaded\",t,!1),c.addEventListener(\"load\",t,!1);function t(){if(o);else{o=!0;for(var e,t,r,s=-1!==navigator.appVersion.indexOf(\"MSIE 10\"),a=!!navigator.userAgent.match(\/Trident.*rv:11\\.\/),i=d.querySelectorAll(\"iframe.wp-embedded-content\"),n=0;n<i.length;n++){if(!(r=(t=i[n]).getAttribute(\"data-secret\")))r=Math.random().toString(36).substr(2,10),t.src+=\"#?secret=\"+r,t.setAttribute(\"data-secret\",r);if(s||a)(e=t.cloneNode(!0)).removeAttribute(\"security\"),t.parentNode.replaceChild(e,t);t.contentWindow.postMessage({message:\"ready\",secret:r},\"*\")}}}}(window,document);\n<\/script>\n","description":"We've been playing about with Amazon Web Service cloud based  infrastructure for the last few weeks and thought we'd share this.&nbsp;  Using&nbsp;StarCluster&nbsp;and following this&nbsp;very cool demo&nbsp;we launched a 3 node cluster  on Amazon&rsquo;s Elastic Compute Cloud infrastructure within the space of  about 10 minutes. You'll need an Amazon account (i.e. you give Amazon your credit card  details) and create a private key pair via their website.&nbsp; You then  launch a small machine instance of StarCluster to act as the coordination  server.&nbsp; Installation of StarCluster is a single command.&nbsp; To configure  the 'personality' of the cluster one modifies the configuration file  (about 2 lines).&nbsp; To set up permissions you need to copy your key to the  coordination server and reference this key in the config file. Starting the cluster is also one command, as is adding or deleting  nodes.&nbsp; In about 5 minutes we had a head node and a worker node. &nbsp;We decided to add a second worker node which took about 4 minutes.&nbsp; StarCluster uses Oracle  Grid Engine and works in a very standard manner.&nbsp; We were able to  compile and submit an OpenMPI job in the space of about 60 seconds and  it ran perfectly.However the coolest bit is the ability to&nbsp;'bid' for Amazon spot  instances.&nbsp; StarCluster  allows one to look at the spot instance price history over time to allow  you to make an informed bid in $\/hour.&nbsp; You can launch your cluster  based on this bid and as resources become available so your nodes are  provisioned.&nbsp; As long as your bid is (just) over the average price  you're pretty much guaranteed a reasonably priced compute cluster for as  long as you want.&nbsp;Above is the current hourly value of an 8 core Nehalem server with two 2050 NVIDIA GPU cards.We're now looking at ways to scale our current (and pending) infrastructure into this space.","thumbnail_url":"https:\/\/ucthpc.uct.ac.za\/wp-content\/uploads\/2015\/07\/starcluster.jpg"}