{"id":335,"date":"2015-06-19T11:53:40","date_gmt":"2015-06-19T11:53:40","guid":{"rendered":"http:\/\/blogs.uct.ac.za\/blog\/big-bytes\/2015\/06\/19\/slurm-job-preemption"},"modified":"2015-08-14T10:01:18","modified_gmt":"2015-08-14T08:01:18","slug":"slurm-job-preemption","status":"publish","type":"post","link":"https:\/\/ucthpc.uct.ac.za\/index.php\/2015\/06\/19\/slurm-job-preemption\/","title":{"rendered":"SLURM job preemption"},"content":{"rendered":"

SLURM provides a preemption mechanism to deal with situations where cluster become overloaded. This can be configured in several ways:<\/p>\n

FIFO:<\/b>
\nThis is the most simplistic method of queueing in which there is no preemption, jobs come in, queue and are dealt with in that order. Backfill scheduling is also enabled by default and this allows advanced scheduling of jobs as long as they won’t delay starting jobs ahead of them in the queue.<\/p>\n

Preemption via Thread Control:<\/b>
\nHere priority is based on the partition that the user submits jobs to. Core sharing is disabled and PreemptMode is set to SUSPEND. Below is a case where all cores used…<\/p>\n

JOBID PARTITION     NAME     USER ST    TIME  NODELIST\r\n 1807     uctlo MemTestB      bob  R    5:28  hpc402\r\n 1808     uctlo MemTestB      bob  R    5:25  hpc402\r\n 1809     uctlo MemTestB      bob  R    5:25  hpc402\r\n 1811     ucthi MemTestA     andy  R    4:53  hpc401\r\n 1812     ucthi MemTestA     andy  R    3:07  hpc401\r\n 1813     ucthi MemTestA     andy  R    1:30  hpc401\r\n<\/pre>\n
User andy submits a job to the ucthi partition…<\/p>\n
JOBID PARTITION     NAME     USER ST    TIME  NODELIST\r\n 1808     uctlo MemTestB      bob  R    5:34  hpc402\r\n 1809     uctlo MemTestB      bob  R    5:34  hpc402\r\n 1811     ucthi MemTestA     andy  R    5:02  hpc401\r\n 1812     ucthi MemTestA     andy  R    3:16  hpc401\r\n 1813     ucthi MemTestA     andy  R    1:39  hpc401\r\n 1814     ucthi MemTestA     andy  R    0:01  hpc402\r\n 1807     uctlo MemTestB      bob  S    5:36  hpc402\r\n<\/pre>\n
One of bob’s jobs is now suspended while user andy’s job runs. However here bob’s job 1807 is still consuming RAM which may be a problem if user Andy’s job also needs lots of RAM. In the case where RAM is an issue it is best to cancel and resubmit low priority jobs.<\/p>\n
Here core sharing disabled and PreemptMode is set to REQUEUE. User bob is running jobs on all available cores on uctlo partition…<\/p>\n
JOBID PARTITION     NAME     USER ST    TIME  NODELIST\r\n 1794     uctlo MemTestB      bob  R    0:33  hpc401\r\n 1795     uctlo MemTestB      bob  R    0:30  hpc401\r\n 1796     uctlo MemTestB      bob  R    0:30  hpc401\r\n 1797     uctlo MemTestB      bob  R    0:30  hpc402\r\n 1798     uctlo MemTestB      bob  R    0:30  hpc402\r\n 1799     uctlo MemTestB      bob  R    0:27  hpc402\r\n<\/pre>\n
User andy starts submitting jobs to the ucthi partition, bob’s jobs are cancelled…<\/p>\n
JOBID PARTITION     NAME     USER ST    TIME  NODELIST\r\n 1794     uctlo MemTestB      bob CG    0:00  hpc401\r\n 1795     uctlo MemTestB      bob  R    0:49  hpc401\r\n 1796     uctlo MemTestB      bob  R    0:49  hpc401\r\n 1797     uctlo MemTestB      bob  R    0:49  hpc402\r\n 1798     uctlo MemTestB      bob  R    0:49  hpc402\r\n 1799     uctlo MemTestB      bob  R    0:46  hpc402\r\n 1801     ucthi MemTestA     andy  R    0:01  hpc401\r\n<\/pre>\n
3 of bob’s jobs have been cancelled and resubmitted in pending state while andy’s 3 jobs are now running…<\/p>\n
 1794     uctlo MemTestB      bob PD    0:00  (BeginTime)\r\n 1795     uctlo MemTestB      bob PD    0:00  (BeginTime)\r\n 1796     uctlo MemTestB      bob PD    0:00  (BeginTime)\r\n 1797     uctlo MemTestB      bob  R    0:59  hpc402\r\n 1798     uctlo MemTestB      bob  R    0:59  hpc402\r\n 1799     uctlo MemTestB      bob  R    0:56  hpc402\r\n 1801     ucthi MemTestA     andy  R    0:11  hpc401\r\n 1802     ucthi MemTestA     andy  R    0:06  hpc401\r\n 1803     ucthi MemTestA     andy  R    0:02  hpc401\r\n<\/pre>\n
In this instance it is possible to oversubscribe the cores on a node and hence memory limits need to be protected by setting memory as a consumable resource. With this scheduling methods jobs may still not run if memory is set as a consumable resource and there is insufficient RAM per core. A default RAM\/core\/job is set in the configuration file, but users can set there own requirements within the bounds of their account settings.<\/p>\n
Preemption via Gang scheduling:<\/b>
\nHere users andy and bob have filled up all available cores and some of andy’s jobs are suspended.<\/p>\n
JOBID PARTITION     NAME     USER ST    TIME  NODELIST\r\n 1815  ucthimem MemTestB      bob  R    0:50  hpc406\r\n 1816  ucthimem MemTestB      bob  R    0:47  hpc406\r\n 1817  ucthimem MemTestB      bob  R    0:47  hpc407\r\n 1820  ucthimem MemTestA   alewis  R    0:19  hpc407\r\n 1821  ucthimem MemTestA   alewis  R    0:16  hpc408\r\n 1822  ucthimem MemTestA   alewis  R    0:16  hpc408\r\n 1823  ucthimem MemTestA   alewis  S    0:00  hpc406\r\n 1824  ucthimem MemTestA   alewis  S    0:00  hpc406\r\n<\/pre>\n
After a time slice has passed some of bob’s jobs are suspended and andy’s jobs run.<\/p>\n
JOBID PARTITION     NAME     USER ST    TIME  NODELIST\r\n 1817  ucthimem MemTestB      bob  R    0:47  hpc407\r\n 1820  ucthimem MemTestA     andy  R    0:19  hpc407\r\n 1821  ucthimem MemTestA     andy  R    0:16  hpc408\r\n 1822  ucthimem MemTestA     andy  R    0:16  hpc408\r\n 1823  ucthimem MemTestA     andy  R    0:00  hpc406\r\n 1824  ucthimem MemTestA     andy  R    0:00  hpc406\r\n 1815  ucthimem MemTestB      bob  S    0:50  hpc406\r\n 1816  ucthimem MemTestB      bob  S    0:47  hpc406\r\n<\/pre>\n
This is repeated until all jobs are completed. The time slice between job suspensions is 60 seconds by default. In the case of user andy submitting to a higher priority partition this scheduling scheme reverts to standard job preemption and bob’s jobs are suspended indefinitely until cores become free. Once again cores can be oversubscribed and memory needs to be protected.<\/p>\n
Multifactor preemption:<\/b>
\nThis is the most complex form of preemption. Here the job priority is based upon a complex factoring algorithm<\/p>\n
Job_priority =\r\n(PriorityWeightAge) * (age_factor) +\r\n(PriorityWeightFairshare) * (fair-share_factor) +\r\n(PriorityWeightJobSize) * (job_size_factor) +\r\n(PriorityWeightPartition) * (partition_factor) +\r\n(PriorityWeightQOS) * (QOS_factor)\r\n<\/pre>\n
You are encouraged to read the official documentation<\/a> on how this can be configured. Below is a simple example based only on user QOS for priority. User bob is consuming all available cores…<\/p>\n
JOBID PARTITION        NAME  USER  ST    TIME  NODELIST\r\n 1893     ucthi   CoreTestB   bob   R    0:09  hpc402\r\n 1892     ucthi   CoreTestB   bob   R    0:12  hpc402\r\n 1889     ucthi   CoreTestB   bob   R    0:15  hpc401\r\n 1890     ucthi   CoreTestB   bob   R    0:15  hpc401\r\n 1891     ucthi   CoreTestB   bob   R    0:15  hpc402\r\n 1888     ucthi   CoreTestB   bob   R    0:18  hpc401\r\n<\/pre>\n
User andy submits a job which won’t run as there are no resources free. As andy’s job has a higher priority sufficient jobs of bob’s are cancelled to allow it to run…<\/p>\n
JOBID PARTITION        NAME  USER  ST    TIME  NODELIST\r\n 1889     ucthi   CoreTestB   bob  CG    0:00  hpc401\r\n 1890     ucthi   CoreTestB   bob  CG    0:00  hpc401\r\n 1888     ucthi   CoreTestB   bob  CG    0:00  hpc401\r\n 1894     ucthi   CoreTestA  andy  PD    0:00  (Resources)\r\n 1893     ucthi   CoreTestB   bob   R    0:13  hpc402\r\n 1892     ucthi   CoreTestB   bob   R    0:16  hpc402\r\n 1891     ucthi   CoreTestB   bob   R    0:19  hpc402\r\n<\/pre>\n
Once andy’s job is running bob’s jobs are requeued…<\/p>\n
JOBID PARTITION        NAME  USER  ST    TIME  NODELIST\r\n 1888     ucthi   CoreTestB   bob  PD    0:00  (BeginTime)\r\n 1889     ucthi   CoreTestB   bob  PD    0:00  (BeginTime)\r\n 1890     ucthi   CoreTestB   bob  PD    0:00  (BeginTime)\r\n 1894     ucthi   CoreTestA  andy   R    0:03  hpc401\r\n 1893     ucthi   CoreTestB   bob   R    0:17  hpc402\r\n 1892     ucthi   CoreTestB   bob   R    0:20  hpc402\r\n 1891     ucthi   CoreTestB   bob   R    0:23  hpc402\r\n<\/pre>\n
After a short time the backfill scheduler allows one of bob’s restarted jobs to run. This is because andy’s job needs 2 nodes but bob’s jobs only need one node.<\/p>\n
JOBID PARTITION        NAME  USER  ST    TIME  NODELIST\r\n 1890     ucthi   CoreTestB   bob  PD    0:00  (Priority)\r\n 1889     ucthi   CoreTestB   bob  PD    0:00  (Resources)\r\n 1894     ucthi   CoreTestA  andy   R    2:41  hpc401\r\n 1888     ucthi   CoreTestB   bob   R    2:16  hpc401\r\n 1893     ucthi   CoreTestB   bob   R    2:55  hpc402\r\n 1892     ucthi   CoreTestB   bob   R    2:58  hpc402\r\n 1891     ucthi   CoreTestB   bob   R    3:01  hpc402\r\n<\/pre>\n
Unfortunately this preemption mode is not compatible with thread suspension.<\/p>\n","protected":false},"excerpt":{"rendered":"
SLURM provides a preemption mechanism to deal with situations where cluster become overloaded. This can be configured in several ways:<\/div>\n
FIFO:<\/strong><\/div>\n
This is the most simplistic method of queueing in which there is no preemption, jobs come in, queue and are dealt with in that order. Backfill scheduling is also enabled by default and this allows advanced scheduling of jobs as long as they won’t delay starting jobs ahead of them in the queue.<\/div>\n
Preemption via Thread Control:<\/strong><\/div>\n
Here priority is based on the partition that the user submits jobs to. Core sharing is disabled and PreemptMode is set to SUSPEND. Below is a case where all cores used…<\/div>\n
JOBID PARTITION     NAME     USER ST    TIME  NODELIST<\/span><\/div>\n
 1807     uctlo MemTestB      bob  R    5:28  hpc402<\/span><\/div>\n
 1808     uctlo MemTestB      bob  R    5:25  hpc402<\/span><\/div>\n
 1809     uctlo MemTestB      bob  R    5:25  hpc402<\/span><\/div>\n
 1811     ucthi MemTestA     andy  R    4:53  hpc401<\/span><\/div>\n
 1812     ucthi MemTestA     andy  R    3:07  hpc401<\/span><\/div>\n
 1813     ucthi MemTestA     andy  R    1:30  hpc401<\/span><\/div>\n
User andy submits a job to the ucthi partition…<\/div>\n
JOBID PARTITION     NAME     USER ST    TIME  NODELIST<\/span><\/div>\n
 1808     uctlo MemTestB      bob  R    5:34  hpc402<\/span><\/div>\n
 1809     uctlo MemTestB      bob  R    5:34  hpc402<\/span><\/div>\n
 1811     ucthi MemTestA     andy  R    5:02  hpc401<\/span><\/div>\n
 1812     ucthi MemTestA     andy  R    3:16  hpc401<\/span><\/div>\n
 1813     ucthi MemTestA     andy  R    1:39  hpc401<\/span><\/div>\n
 1814     ucthi MemTestA     andy  R    0:01  hpc402<\/span><\/div>\n
 1807     uctlo MemTestB      bob  S    5:36  hpc402<\/span><\/div>\n
One of bob’s jobs is now suspended while user andy’s job runs. However here bob’s job 1807 is still consuming RAM which may be a problem if user Andy’s job also needs lots of RAM. In the case where RAM is an issue it is best to cancel and resubmit low priority jobs.<\/div>\n
 <\/div>\n
Here core sharing disabled and PreemptMode is set to REQUEUE. User bob is running jobs on all available cores on uctlo partition…<\/div>\n
JOBID PARTITION     NAME     USER ST    TIME  NODELIST<\/span><\/div>\n
 1794     uctlo MemTestB      bob  R    0:33  hpc401<\/span><\/div>\n
 1795     uctlo MemTestB      bob  R    0:30  hpc401<\/span><\/div>\n
 1796     uctlo MemTestB      bob  R    0:30  hpc401<\/span><\/div>\n
 1797     uctlo MemTestB      bob  R    0:30  hpc402<\/span><\/div>\n
 1798     uctlo MemTestB      bob  R    0:30  hpc402<\/span><\/div>\n
 1799     uctlo MemTestB      bob  R    0:27  hpc402<\/span><\/div>\n
User andy starts submitting jobs to the ucthi partition, bob’s jobs are cancelled…<\/div>\n
JOBID PARTITION     NAME     USER ST    TIME  NODELIST<\/span><\/div>\n
 1794     uctlo MemTestB      bob CG    0:00  hpc401<\/span><\/div>\n
 1795     uctlo MemTestB      bob  R    0:49  hpc401<\/span><\/div>\n
 1796     uctlo MemTestB      bob  R    0:49  hpc401<\/span><\/div>\n
 1797     uctlo MemTestB      bob  R    0:49  hpc402<\/span><\/div>\n
 1798     uctlo MemTestB      bob  R    0:49  hpc402<\/span><\/div>\n
 1799     uctlo MemTestB      bob  R    0:46  hpc402<\/span><\/div>\n
 1801     ucthi MemTestA     andy  R    0:01  hpc401<\/span><\/div>\n
3 of bob’s jobs have been cancelled and resubmitted in pending state while andy’s 3 jobs are now running…<\/div>\n
 1794     uctlo MemTestB      bob PD    0:00  (BeginTime)<\/span><\/div>\n
 1795     uctlo MemTestB      bob PD    0:00  (BeginTime)<\/span><\/div>\n
 1796     uctlo MemTestB      bob PD    0:00  (BeginTime)<\/span><\/div>\n
 1797     uctlo MemTestB      bob  R    0:59  hpc402<\/span><\/div>\n
 1798     uctlo MemTestB      bob  R    0:59  hpc402<\/span><\/div>\n
 1799     uctlo MemTestB      bob  R    0:56  hpc402<\/span><\/div>\n
 1801     ucthi MemTestA     andy  R    0:11  hpc401<\/span><\/div>\n
 1802     ucthi MemTestA     andy  R    0:06  hpc401<\/span><\/div>\n
 1803     ucthi MemTestA     andy  R    0:02  hpc401<\/span><\/div>\n
In this instance it is possible to oversubscribe the cores on a node and hence memory limits need to be protected by setting memory as a consumable resource. With this scheduling methods jobs may still not run if memory is set as a consumable resource and there is insufficient RAM per core. A default RAM\/core\/job is set in the configuration file, but users can set there own requirements within the bounds of their account settings.<\/div>\n
<\/div>\n
Preemption via Gang scheduling:<\/strong><\/div>\n
Here users andy and bob have filled up all available cores and some of andy’s jobs are suspended.<\/div>\n
JOBID PARTITION     NAME     USER ST    TIME  NODELIST<\/span><\/div>\n
 1815  ucthimem MemTestB      bob  R    0:50  hpc406<\/span><\/div>\n
 1816  ucthimem MemTestB      bob  R    0:47  hpc406<\/span><\/div>\n
 1817  ucthimem MemTestB      bob  R    0:47  hpc407<\/span><\/div>\n
 1820  ucthimem MemTestA   alewis  R    0:19  hpc407<\/span><\/div>\n
 1821  ucthimem MemTestA   alewis  R    0:16  hpc408<\/span><\/div>\n
 1822  ucthimem MemTestA   alewis  R    0:16  hpc408<\/span><\/div>\n
 1823  ucthimem MemTestA   alewis  S    0:00  hpc406<\/span><\/div>\n
 1824  ucthimem MemTestA   alewis  S    0:00  hpc406<\/span><\/div>\n
After a time slice has passed some of bob’s jobs are suspended and andy’s jobs run.<\/div>\n
JOBID PARTITION     NAME     USER ST    TIME  NODELIST<\/span><\/div>\n
 1817  ucthimem MemTestB      bob  R    0:47  hpc407<\/span><\/div>\n
 1820  ucthimem MemTestA     andy  R    0:19  hpc407<\/span><\/div>\n
 1821  ucthimem MemTestA     andy  R    0:16  hpc408<\/span><\/div>\n
 1822  ucthimem MemTestA     andy  R    0:16  hpc408<\/span><\/div>\n
 1823  ucthimem MemTestA     andy  R    0:00  hpc406<\/span><\/div>\n
 1824  ucthimem MemTestA     andy  R    0:00  hpc406<\/span><\/div>\n
 1815  ucthimem MemTestB      bob  S    0:50  hpc406<\/span><\/div>\n
 1816  ucthimem MemTestB      bob  S    0:47  hpc406<\/span><\/div>\n
This is repeated until all jobs are completed. The time slice between job suspensions is 60 seconds by default. In the case of user andy submitting to a higher priority partition this scheduling scheme reverts to standard job preemption and bob’s jobs are suspended indefinitely until cores become free. Once again cores can be oversubscribed and memory needs to be protected.<\/div>\n
<\/div>\n
Multifactor preemption:<\/strong><\/div>\n
This is the most complex form of preemption. Here the job priority is based upon a complex factoring algorithm<\/div>\n
Job_priority =<\/span><\/div>\n
(PriorityWeightAge) * (age_factor) +<\/span><\/div>\n
(PriorityWeightFairshare) * (fair-share_factor) +<\/span><\/div>\n
(PriorityWeightJobSize) * (job_size_factor) +<\/span><\/div>\n
(PriorityWeightPartition) * (partition_factor) +<\/span><\/div>\n
(PriorityWeightQOS) * (QOS_factor)<\/span><\/div>\n
<\/div>\n
You are encouraged to read the official documentation on how this can be configured<\/a>. Below is a simple example based only on user QOS for priority. User bob is consuming all available cores…<\/div>\n
JOBID PARTITION        NAME  USER  ST    TIME  NODELIST<\/span><\/div>\n
 1893     ucthi   CoreTestB   bob   R    0:09  hpc402<\/span><\/div>\n
 1892     ucthi   CoreTestB   bob   R    0:12  hpc402<\/span><\/div>\n
 1889     ucthi   CoreTestB   bob   R    0:15  hpc401<\/span><\/div>\n
 1890     ucthi   CoreTestB   bob   R    0:15  hpc401<\/span><\/div>\n
 1891     ucthi   CoreTestB   bob   R    0:15  hpc402<\/span><\/div>\n
 1888     ucthi   CoreTestB   bob   R    0:18  hpc401<\/span><\/div>\n
User andy submits a job which won’t run as there are no resources free. As andy’s job has a higher priority sufficient jobs of bob’s are cancelled to allow it to run…<\/div>\n
JOBID PARTITION        NAME  USER  ST    TIME  NODELIST<\/span><\/div>\n
 1889     ucthi   CoreTestB   bob  CG    0:00  hpc401<\/span><\/div>\n
 1890     ucthi   CoreTestB   bob  CG    0:00  hpc401<\/span><\/div>\n
 1888     ucthi   CoreTestB   bob  CG    0:00  hpc401<\/span><\/div>\n
 1894     ucthi   CoreTestA  andy  PD    0:00  (Resources)<\/span><\/div>\n
 1893     ucthi   CoreTestB   bob   R    0:13  hpc402<\/span><\/div>\n
 1892     ucthi   CoreTestB   bob   R    0:16  hpc402<\/span><\/div>\n
 1891     ucthi   CoreTestB   bob   R    0:19  hpc402<\/span><\/div>\n
Once andy’s job is running bob’s jobs are requeued…<\/div>\n
JOBID PARTITION        NAME  USER  ST    TIME  NODELIST<\/span><\/div>\n
 1888     ucthi   CoreTestB   bob  PD    0:00  (BeginTime)<\/span><\/div>\n
 1889     ucthi   CoreTestB   bob  PD    0:00  (BeginTime)<\/span><\/div>\n
 1890     ucthi   CoreTestB   bob  PD    0:00  (BeginTime)<\/span><\/div>\n
 1894     ucthi   CoreTestA  andy   R    0:03  hpc401<\/span><\/div>\n
 1893     ucthi   CoreTestB   bob   R    0:17  hpc402<\/span><\/div>\n
 1892     ucthi   CoreTestB   bob   R    0:20  hpc402<\/span><\/div>\n
 1891     ucthi   CoreTestB   bob   R    0:23  hpc402<\/span><\/div>\n
After a short time the backfill scheduler allows one of bob’s restarted jobs to run. This is because andy’s job needs 2 nodes but bob’s jobs only need one node.<\/div>\n
JOBID PARTITION        NAME  USER  ST    TIME  NODELIST<\/span><\/div>\n
 1890     ucthi   CoreTestB   bob  PD    0:00  (Priority)<\/span><\/div>\n
 1889     ucthi   CoreTestB   bob  PD    0:00  (Resources)<\/span><\/div>\n
 1894     ucthi   CoreTestA  andy   R    2:41  hpc401<\/span><\/div>\n
 1888     ucthi   CoreTestB   bob   R    2:16  hpc401<\/span><\/div>\n
 1893     ucthi   CoreTestB   bob   R    2:55  hpc402<\/span><\/div>\n
 1892     ucthi   CoreTestB   bob   R    2:58  hpc402<\/span><\/div>\n
 1891     ucthi   CoreTestB   bob   R    3:01  hpc402<\/span><\/div>\n
Unfortunately this preemption mode is not compatible with thread suspension.<\/div>\n
<\/div>\n
<\/div>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[4,5],"tags":[],"yoast_head":"\nSLURM job preemption - UCT HPC<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/ucthpc.uct.ac.za\/index.php\/2015\/06\/19\/slurm-job-preemption\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"SLURM job preemption - UCT HPC\" \/>\n<meta property=\"og:description\" content=\"SLURM provides a preemption mechanism to deal with situations where cluster become overloaded. This can be configured in several ways:FIFO:This is the most simplistic method of queueing in which there is no preemption, jobs come in, queue and are dealt with in that order. Backfill scheduling is also enabled by default and this allows advanced scheduling of jobs as long as they won't delay starting jobs ahead of them in the queue.Preemption via Thread Control:Here priority is based on the partition that the user submits jobs to. Core sharing is disabled and PreemptMode is set to SUSPEND. Below is a case where all cores used...JOBID PARTITION     NAME     USER ST    TIME  NODELIST 1807     uctlo MemTestB      bob  R    5:28  hpc402 1808     uctlo MemTestB      bob  R    5:25  hpc402 1809     uctlo MemTestB      bob  R    5:25  hpc402 1811     ucthi MemTestA     andy  R    4:53  hpc401 1812     ucthi MemTestA     andy  R    3:07  hpc401 1813     ucthi MemTestA     andy  R    1:30  hpc401User andy submits a job to the ucthi partition...JOBID PARTITION     NAME     USER ST    TIME  NODELIST 1808     uctlo MemTestB      bob  R    5:34  hpc402 1809     uctlo MemTestB      bob  R    5:34  hpc402 1811     ucthi MemTestA     andy  R    5:02  hpc401 1812     ucthi MemTestA     andy  R    3:16  hpc401 1813     ucthi MemTestA     andy  R    1:39  hpc401 1814     ucthi MemTestA     andy  R    0:01  hpc402 1807     uctlo MemTestB      bob  S    5:36  hpc402One of bob's jobs is now suspended while user andy's job runs. However here bob's job 1807 is still consuming RAM which may be a problem if user Andy's job also needs lots of RAM. In the case where RAM is an issue it is best to cancel and resubmit low priority jobs. Here core sharing disabled and PreemptMode is set to REQUEUE. User bob is running jobs on all available cores on uctlo partition...JOBID PARTITION     NAME     USER ST    TIME  NODELIST 1794     uctlo MemTestB      bob  R    0:33  hpc401 1795     uctlo MemTestB      bob  R    0:30  hpc401 1796     uctlo MemTestB      bob  R    0:30  hpc401 1797     uctlo MemTestB      bob  R    0:30  hpc402 1798     uctlo MemTestB      bob  R    0:30  hpc402 1799     uctlo MemTestB      bob  R    0:27  hpc402User andy starts submitting jobs to the ucthi partition, bob's jobs are cancelled...JOBID PARTITION     NAME     USER ST    TIME  NODELIST 1794     uctlo MemTestB      bob CG    0:00  hpc401 1795     uctlo MemTestB      bob  R    0:49  hpc401 1796     uctlo MemTestB      bob  R    0:49  hpc401 1797     uctlo MemTestB      bob  R    0:49  hpc402 1798     uctlo MemTestB      bob  R    0:49  hpc402 1799     uctlo MemTestB      bob  R    0:46  hpc402 1801     ucthi MemTestA     andy  R    0:01  hpc4013 of bob's jobs have been cancelled and resubmitted in pending state while andy's 3 jobs are now running... 1794     uctlo MemTestB      bob PD    0:00  (BeginTime) 1795     uctlo MemTestB      bob PD    0:00  (BeginTime) 1796     uctlo MemTestB      bob PD    0:00  (BeginTime) 1797     uctlo MemTestB      bob  R    0:59  hpc402 1798     uctlo MemTestB      bob  R    0:59  hpc402 1799     uctlo MemTestB      bob  R    0:56  hpc402 1801     ucthi MemTestA     andy  R    0:11  hpc401 1802     ucthi MemTestA     andy  R    0:06  hpc401 1803     ucthi MemTestA     andy  R    0:02  hpc401In this instance it is possible to oversubscribe the cores on a node and hence memory limits need to be protected by setting memory as a consumable resource. With this scheduling methods jobs may still not run if memory is set as a consumable resource and there is insufficient RAM per core. A default RAM\/core\/job is set in the configuration file, but users can set there own requirements within the bounds of their account settings.Preemption via Gang scheduling:Here users andy and bob have filled up all available cores and some of andy's jobs are suspended.JOBID PARTITION     NAME     USER ST    TIME  NODELIST 1815  ucthimem MemTestB      bob  R    0:50  hpc406 1816  ucthimem MemTestB      bob  R    0:47  hpc406 1817  ucthimem MemTestB      bob  R    0:47  hpc407 1820  ucthimem MemTestA   alewis  R    0:19  hpc407 1821  ucthimem MemTestA   alewis  R    0:16  hpc408 1822  ucthimem MemTestA   alewis  R    0:16  hpc408 1823  ucthimem MemTestA   alewis  S    0:00  hpc406 1824  ucthimem MemTestA   alewis  S    0:00  hpc406After a time slice has passed some of bob's jobs are suspended and andy's jobs run.JOBID PARTITION     NAME     USER ST    TIME  NODELIST 1817  ucthimem MemTestB      bob  R    0:47  hpc407 1820  ucthimem MemTestA     andy  R    0:19  hpc407 1821  ucthimem MemTestA     andy  R    0:16  hpc408 1822  ucthimem MemTestA     andy  R    0:16  hpc408 1823  ucthimem MemTestA     andy  R    0:00  hpc406 1824  ucthimem MemTestA     andy  R    0:00  hpc406 1815  ucthimem MemTestB      bob  S    0:50  hpc406 1816  ucthimem MemTestB      bob  S    0:47  hpc406This is repeated until all jobs are completed. The time slice between job suspensions is 60 seconds by default. In the case of user andy submitting to a higher priority partition this scheduling scheme reverts to standard job preemption and bob's jobs are suspended indefinitely until cores become free. Once again cores can be oversubscribed and memory needs to be protected.Multifactor preemption:This is the most complex form of preemption. Here the job priority is based upon a complex factoring algorithmJob_priority =(PriorityWeightAge) * (age_factor) +(PriorityWeightFairshare) * (fair-share_factor) +(PriorityWeightJobSize) * (job_size_factor) +(PriorityWeightPartition) * (partition_factor) +(PriorityWeightQOS) * (QOS_factor)You are encouraged to read the official documentation on how this can be configured. Below is a simple example based only on user QOS for priority. User bob is consuming all available cores...JOBID PARTITION        NAME  USER  ST    TIME  NODELIST 1893     ucthi   CoreTestB   bob   R    0:09  hpc402 1892     ucthi   CoreTestB   bob   R    0:12  hpc402 1889     ucthi   CoreTestB   bob   R    0:15  hpc401 1890     ucthi   CoreTestB   bob   R    0:15  hpc401 1891     ucthi   CoreTestB   bob   R    0:15  hpc402 1888     ucthi   CoreTestB   bob   R    0:18  hpc401User andy submits a job which won't run as there are no resources free. As andy's job has a higher priority sufficient jobs of bob's are cancelled to allow it to run...JOBID PARTITION        NAME  USER  ST    TIME  NODELIST 1889     ucthi   CoreTestB   bob  CG    0:00  hpc401 1890     ucthi   CoreTestB   bob  CG    0:00  hpc401 1888     ucthi   CoreTestB   bob  CG    0:00  hpc401 1894     ucthi   CoreTestA  andy  PD    0:00  (Resources) 1893     ucthi   CoreTestB   bob   R    0:13  hpc402 1892     ucthi   CoreTestB   bob   R    0:16  hpc402 1891     ucthi   CoreTestB   bob   R    0:19  hpc402Once andy's job is running bob's jobs are requeued...JOBID PARTITION        NAME  USER  ST    TIME  NODELIST 1888     ucthi   CoreTestB   bob  PD    0:00  (BeginTime) 1889     ucthi   CoreTestB   bob  PD    0:00  (BeginTime) 1890     ucthi   CoreTestB   bob  PD    0:00  (BeginTime) 1894     ucthi   CoreTestA  andy   R    0:03  hpc401 1893     ucthi   CoreTestB   bob   R    0:17  hpc402 1892     ucthi   CoreTestB   bob   R    0:20  hpc402 1891     ucthi   CoreTestB   bob   R    0:23  hpc402After a short time the backfill scheduler allows one of bob's restarted jobs to run. This is because andy's job needs 2 nodes but bob's jobs only need one node.JOBID PARTITION        NAME  USER  ST    TIME  NODELIST 1890     ucthi   CoreTestB   bob  PD    0:00  (Priority) 1889     ucthi   CoreTestB   bob  PD    0:00  (Resources) 1894     ucthi   CoreTestA  andy   R    2:41  hpc401 1888     ucthi   CoreTestB   bob   R    2:16  hpc401 1893     ucthi   CoreTestB   bob   R    2:55  hpc402 1892     ucthi   CoreTestB   bob   R    2:58  hpc402 1891     ucthi   CoreTestB   bob   R    3:01  hpc402Unfortunately this preemption mode is not compatible with thread suspension.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/ucthpc.uct.ac.za\/index.php\/2015\/06\/19\/slurm-job-preemption\/\" \/>\n<meta property=\"og:site_name\" content=\"UCT HPC\" \/>\n<meta property=\"article:published_time\" content=\"2015-06-19T11:53:40+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2015-08-14T08:01:18+00:00\" \/>\n<meta name=\"author\" content=\"Andrew Lewis\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Andrew Lewis\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2015\/06\/19\/slurm-job-preemption\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2015\/06\/19\/slurm-job-preemption\/\"},\"author\":{\"name\":\"Andrew Lewis\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#\/schema\/person\/c183ad1c0a1063124a72d63963ae9c7e\"},\"headline\":\"SLURM job preemption\",\"datePublished\":\"2015-06-19T11:53:40+00:00\",\"dateModified\":\"2015-08-14T08:01:18+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2015\/06\/19\/slurm-job-preemption\/\"},\"wordCount\":534,\"publisher\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#organization\"},\"articleSection\":[\"hpc\",\"SLURM\"],\"inLanguage\":\"en-ZA\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2015\/06\/19\/slurm-job-preemption\/\",\"url\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2015\/06\/19\/slurm-job-preemption\/\",\"name\":\"SLURM job preemption - UCT HPC\",\"isPartOf\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#website\"},\"datePublished\":\"2015-06-19T11:53:40+00:00\",\"dateModified\":\"2015-08-14T08:01:18+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2015\/06\/19\/slurm-job-preemption\/#breadcrumb\"},\"inLanguage\":\"en-ZA\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/ucthpc.uct.ac.za\/index.php\/2015\/06\/19\/slurm-job-preemption\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/2015\/06\/19\/slurm-job-preemption\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/ucthpc.uct.ac.za\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"SLURM job preemption\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#website\",\"url\":\"https:\/\/ucthpc.uct.ac.za\/\",\"name\":\"UCT HPC\",\"description\":\"University of Cape Town High Performance Computing\",\"publisher\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/ucthpc.uct.ac.za\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-ZA\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#organization\",\"name\":\"University of Cape Town High Performance Computing\",\"url\":\"https:\/\/ucthpc.uct.ac.za\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-ZA\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/ucthpc.uct.ac.za\/wp-content\/uploads\/2015\/09\/logocircless.png\",\"contentUrl\":\"https:\/\/ucthpc.uct.ac.za\/wp-content\/uploads\/2015\/09\/logocircless.png\",\"width\":450,\"height\":423,\"caption\":\"University of Cape Town High Performance Computing\"},\"image\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#\/schema\/person\/c183ad1c0a1063124a72d63963ae9c7e\",\"name\":\"Andrew Lewis\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-ZA\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/9652c9c73beeab594b8dc2383a880048?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/9652c9c73beeab594b8dc2383a880048?s=96&d=mm&r=g\",\"caption\":\"Andrew Lewis\"},\"sameAs\":[\"http:\/\/blogs.uct.ac.za\/blog\/big-bytes\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"SLURM job preemption - UCT HPC","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/ucthpc.uct.ac.za\/index.php\/2015\/06\/19\/slurm-job-preemption\/","og_locale":"en_US","og_type":"article","og_title":"SLURM job preemption - UCT HPC","og_description":"SLURM provides a preemption mechanism to deal with situations where cluster become overloaded. This can be configured in several ways:FIFO:This is the most simplistic method of queueing in which there is no preemption, jobs come in, queue and are dealt with in that order. Backfill scheduling is also enabled by default and this allows advanced scheduling of jobs as long as they won't delay starting jobs ahead of them in the queue.Preemption via Thread Control:Here priority is based on the partition that the user submits jobs to. Core sharing is disabled and PreemptMode is set to SUSPEND. Below is a case where all cores used...JOBID PARTITION     NAME     USER ST    TIME  NODELIST 1807     uctlo MemTestB      bob  R    5:28  hpc402 1808     uctlo MemTestB      bob  R    5:25  hpc402 1809     uctlo MemTestB      bob  R    5:25  hpc402 1811     ucthi MemTestA     andy  R    4:53  hpc401 1812     ucthi MemTestA     andy  R    3:07  hpc401 1813     ucthi MemTestA     andy  R    1:30  hpc401User andy submits a job to the ucthi partition...JOBID PARTITION     NAME     USER ST    TIME  NODELIST 1808     uctlo MemTestB      bob  R    5:34  hpc402 1809     uctlo MemTestB      bob  R    5:34  hpc402 1811     ucthi MemTestA     andy  R    5:02  hpc401 1812     ucthi MemTestA     andy  R    3:16  hpc401 1813     ucthi MemTestA     andy  R    1:39  hpc401 1814     ucthi MemTestA     andy  R    0:01  hpc402 1807     uctlo MemTestB      bob  S    5:36  hpc402One of bob's jobs is now suspended while user andy's job runs. However here bob's job 1807 is still consuming RAM which may be a problem if user Andy's job also needs lots of RAM. In the case where RAM is an issue it is best to cancel and resubmit low priority jobs. Here core sharing disabled and PreemptMode is set to REQUEUE. User bob is running jobs on all available cores on uctlo partition...JOBID PARTITION     NAME     USER ST    TIME  NODELIST 1794     uctlo MemTestB      bob  R    0:33  hpc401 1795     uctlo MemTestB      bob  R    0:30  hpc401 1796     uctlo MemTestB      bob  R    0:30  hpc401 1797     uctlo MemTestB      bob  R    0:30  hpc402 1798     uctlo MemTestB      bob  R    0:30  hpc402 1799     uctlo MemTestB      bob  R    0:27  hpc402User andy starts submitting jobs to the ucthi partition, bob's jobs are cancelled...JOBID PARTITION     NAME     USER ST    TIME  NODELIST 1794     uctlo MemTestB      bob CG    0:00  hpc401 1795     uctlo MemTestB      bob  R    0:49  hpc401 1796     uctlo MemTestB      bob  R    0:49  hpc401 1797     uctlo MemTestB      bob  R    0:49  hpc402 1798     uctlo MemTestB      bob  R    0:49  hpc402 1799     uctlo MemTestB      bob  R    0:46  hpc402 1801     ucthi MemTestA     andy  R    0:01  hpc4013 of bob's jobs have been cancelled and resubmitted in pending state while andy's 3 jobs are now running... 1794     uctlo MemTestB      bob PD    0:00  (BeginTime) 1795     uctlo MemTestB      bob PD    0:00  (BeginTime) 1796     uctlo MemTestB      bob PD    0:00  (BeginTime) 1797     uctlo MemTestB      bob  R    0:59  hpc402 1798     uctlo MemTestB      bob  R    0:59  hpc402 1799     uctlo MemTestB      bob  R    0:56  hpc402 1801     ucthi MemTestA     andy  R    0:11  hpc401 1802     ucthi MemTestA     andy  R    0:06  hpc401 1803     ucthi MemTestA     andy  R    0:02  hpc401In this instance it is possible to oversubscribe the cores on a node and hence memory limits need to be protected by setting memory as a consumable resource. With this scheduling methods jobs may still not run if memory is set as a consumable resource and there is insufficient RAM per core. A default RAM\/core\/job is set in the configuration file, but users can set there own requirements within the bounds of their account settings.Preemption via Gang scheduling:Here users andy and bob have filled up all available cores and some of andy's jobs are suspended.JOBID PARTITION     NAME     USER ST    TIME  NODELIST 1815  ucthimem MemTestB      bob  R    0:50  hpc406 1816  ucthimem MemTestB      bob  R    0:47  hpc406 1817  ucthimem MemTestB      bob  R    0:47  hpc407 1820  ucthimem MemTestA   alewis  R    0:19  hpc407 1821  ucthimem MemTestA   alewis  R    0:16  hpc408 1822  ucthimem MemTestA   alewis  R    0:16  hpc408 1823  ucthimem MemTestA   alewis  S    0:00  hpc406 1824  ucthimem MemTestA   alewis  S    0:00  hpc406After a time slice has passed some of bob's jobs are suspended and andy's jobs run.JOBID PARTITION     NAME     USER ST    TIME  NODELIST 1817  ucthimem MemTestB      bob  R    0:47  hpc407 1820  ucthimem MemTestA     andy  R    0:19  hpc407 1821  ucthimem MemTestA     andy  R    0:16  hpc408 1822  ucthimem MemTestA     andy  R    0:16  hpc408 1823  ucthimem MemTestA     andy  R    0:00  hpc406 1824  ucthimem MemTestA     andy  R    0:00  hpc406 1815  ucthimem MemTestB      bob  S    0:50  hpc406 1816  ucthimem MemTestB      bob  S    0:47  hpc406This is repeated until all jobs are completed. The time slice between job suspensions is 60 seconds by default. In the case of user andy submitting to a higher priority partition this scheduling scheme reverts to standard job preemption and bob's jobs are suspended indefinitely until cores become free. Once again cores can be oversubscribed and memory needs to be protected.Multifactor preemption:This is the most complex form of preemption. Here the job priority is based upon a complex factoring algorithmJob_priority =(PriorityWeightAge) * (age_factor) +(PriorityWeightFairshare) * (fair-share_factor) +(PriorityWeightJobSize) * (job_size_factor) +(PriorityWeightPartition) * (partition_factor) +(PriorityWeightQOS) * (QOS_factor)You are encouraged to read the official documentation on how this can be configured. Below is a simple example based only on user QOS for priority. User bob is consuming all available cores...JOBID PARTITION        NAME  USER  ST    TIME  NODELIST 1893     ucthi   CoreTestB   bob   R    0:09  hpc402 1892     ucthi   CoreTestB   bob   R    0:12  hpc402 1889     ucthi   CoreTestB   bob   R    0:15  hpc401 1890     ucthi   CoreTestB   bob   R    0:15  hpc401 1891     ucthi   CoreTestB   bob   R    0:15  hpc402 1888     ucthi   CoreTestB   bob   R    0:18  hpc401User andy submits a job which won't run as there are no resources free. As andy's job has a higher priority sufficient jobs of bob's are cancelled to allow it to run...JOBID PARTITION        NAME  USER  ST    TIME  NODELIST 1889     ucthi   CoreTestB   bob  CG    0:00  hpc401 1890     ucthi   CoreTestB   bob  CG    0:00  hpc401 1888     ucthi   CoreTestB   bob  CG    0:00  hpc401 1894     ucthi   CoreTestA  andy  PD    0:00  (Resources) 1893     ucthi   CoreTestB   bob   R    0:13  hpc402 1892     ucthi   CoreTestB   bob   R    0:16  hpc402 1891     ucthi   CoreTestB   bob   R    0:19  hpc402Once andy's job is running bob's jobs are requeued...JOBID PARTITION        NAME  USER  ST    TIME  NODELIST 1888     ucthi   CoreTestB   bob  PD    0:00  (BeginTime) 1889     ucthi   CoreTestB   bob  PD    0:00  (BeginTime) 1890     ucthi   CoreTestB   bob  PD    0:00  (BeginTime) 1894     ucthi   CoreTestA  andy   R    0:03  hpc401 1893     ucthi   CoreTestB   bob   R    0:17  hpc402 1892     ucthi   CoreTestB   bob   R    0:20  hpc402 1891     ucthi   CoreTestB   bob   R    0:23  hpc402After a short time the backfill scheduler allows one of bob's restarted jobs to run. This is because andy's job needs 2 nodes but bob's jobs only need one node.JOBID PARTITION        NAME  USER  ST    TIME  NODELIST 1890     ucthi   CoreTestB   bob  PD    0:00  (Priority) 1889     ucthi   CoreTestB   bob  PD    0:00  (Resources) 1894     ucthi   CoreTestA  andy   R    2:41  hpc401 1888     ucthi   CoreTestB   bob   R    2:16  hpc401 1893     ucthi   CoreTestB   bob   R    2:55  hpc402 1892     ucthi   CoreTestB   bob   R    2:58  hpc402 1891     ucthi   CoreTestB   bob   R    3:01  hpc402Unfortunately this preemption mode is not compatible with thread suspension.","og_url":"https:\/\/ucthpc.uct.ac.za\/index.php\/2015\/06\/19\/slurm-job-preemption\/","og_site_name":"UCT HPC","article_published_time":"2015-06-19T11:53:40+00:00","article_modified_time":"2015-08-14T08:01:18+00:00","author":"Andrew Lewis","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Andrew Lewis","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/2015\/06\/19\/slurm-job-preemption\/#article","isPartOf":{"@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/2015\/06\/19\/slurm-job-preemption\/"},"author":{"name":"Andrew Lewis","@id":"https:\/\/ucthpc.uct.ac.za\/#\/schema\/person\/c183ad1c0a1063124a72d63963ae9c7e"},"headline":"SLURM job preemption","datePublished":"2015-06-19T11:53:40+00:00","dateModified":"2015-08-14T08:01:18+00:00","mainEntityOfPage":{"@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/2015\/06\/19\/slurm-job-preemption\/"},"wordCount":534,"publisher":{"@id":"https:\/\/ucthpc.uct.ac.za\/#organization"},"articleSection":["hpc","SLURM"],"inLanguage":"en-ZA"},{"@type":"WebPage","@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/2015\/06\/19\/slurm-job-preemption\/","url":"https:\/\/ucthpc.uct.ac.za\/index.php\/2015\/06\/19\/slurm-job-preemption\/","name":"SLURM job preemption - UCT HPC","isPartOf":{"@id":"https:\/\/ucthpc.uct.ac.za\/#website"},"datePublished":"2015-06-19T11:53:40+00:00","dateModified":"2015-08-14T08:01:18+00:00","breadcrumb":{"@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/2015\/06\/19\/slurm-job-preemption\/#breadcrumb"},"inLanguage":"en-ZA","potentialAction":[{"@type":"ReadAction","target":["https:\/\/ucthpc.uct.ac.za\/index.php\/2015\/06\/19\/slurm-job-preemption\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/2015\/06\/19\/slurm-job-preemption\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/ucthpc.uct.ac.za\/"},{"@type":"ListItem","position":2,"name":"SLURM job preemption"}]},{"@type":"WebSite","@id":"https:\/\/ucthpc.uct.ac.za\/#website","url":"https:\/\/ucthpc.uct.ac.za\/","name":"UCT HPC","description":"University of Cape Town High Performance Computing","publisher":{"@id":"https:\/\/ucthpc.uct.ac.za\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/ucthpc.uct.ac.za\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-ZA"},{"@type":"Organization","@id":"https:\/\/ucthpc.uct.ac.za\/#organization","name":"University of Cape Town High Performance Computing","url":"https:\/\/ucthpc.uct.ac.za\/","logo":{"@type":"ImageObject","inLanguage":"en-ZA","@id":"https:\/\/ucthpc.uct.ac.za\/#\/schema\/logo\/image\/","url":"https:\/\/ucthpc.uct.ac.za\/wp-content\/uploads\/2015\/09\/logocircless.png","contentUrl":"https:\/\/ucthpc.uct.ac.za\/wp-content\/uploads\/2015\/09\/logocircless.png","width":450,"height":423,"caption":"University of Cape Town High Performance Computing"},"image":{"@id":"https:\/\/ucthpc.uct.ac.za\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/ucthpc.uct.ac.za\/#\/schema\/person\/c183ad1c0a1063124a72d63963ae9c7e","name":"Andrew Lewis","image":{"@type":"ImageObject","inLanguage":"en-ZA","@id":"https:\/\/ucthpc.uct.ac.za\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/9652c9c73beeab594b8dc2383a880048?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/9652c9c73beeab594b8dc2383a880048?s=96&d=mm&r=g","caption":"Andrew Lewis"},"sameAs":["http:\/\/blogs.uct.ac.za\/blog\/big-bytes"]}]}},"_links":{"self":[{"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/posts\/335"}],"collection":[{"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/comments?post=335"}],"version-history":[{"count":4,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/posts\/335\/revisions"}],"predecessor-version":[{"id":2019,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/posts\/335\/revisions\/2019"}],"wp:attachment":[{"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/media?parent=335"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/categories?post=335"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/tags?post=335"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}