Using the IRIDIA Cluster - Revision history

Manubrambi: Replacing page with 'See http://majorana.ulb.ac.be/wordpress/ http://majorana.ulb.ac.be/wordpress/'

2012-08-08T08:21:06Z

Replacing page with 'See http://majorana.ulb.ac.be/wordpress/ http://majorana.ulb.ac.be/wordpress/'

Arne: /* Cluster status */

2008-02-29T17:16:17Z

Cluster status

Arne: /* Cluster status */

2008-02-29T17:06:40Z

Cluster status

Arne: /* Cluster status */

2008-02-20T17:49:59Z

Cluster status

Arne: /* Cluster status */

2008-02-20T17:32:47Z

Cluster status

Arne: /* How to submit a job */

2008-02-20T17:31:21Z

How to submit a job

Arne: /* Cluster status */

2008-02-20T17:30:59Z

Cluster status

Arne: /* Cluster status */

2008-02-18T16:31:43Z

Cluster status

Arne at 14:49, 5 February 2008

2008-02-05T14:49:12Z

Mmanfrin at 13:57, 5 February 2008

2008-02-05T13:57:49Z

← Older revision		Revision as of 08:21, 8 August 2012
Line 1:		Line 1:
		+	See [[http://majorana.ulb.ac.be/wordpress/ http://majorana.ulb.ac.be/wordpress/]]
−	== Cluster composition ==
−
−	Currently the IRIDIA cluster is composed by two racks. The first one contains 1 server (majorana) + 32 computational nodes (from c0-0 to c0-31), while the second contains 16 computational nodes (from c1-0 to c1-15). In total the cluster is composed of 96 CPUs (64 single-core + 32 dual-core) dedicated to computations and 2 CPUs for administrative purposes.
−
−
−	Each of the units in the first rack (from c0-0 to c0-31) features 2 CPUs AMD Opteron 244 (each with 1MB L2 cache) working at 1,75GHz and 2GB of RAM. Nodes from c0-0 to c0-15 have 4 modules of 512MB 400MHz DDR ECC REG DIMM. Nodes from c0-16 to c0-31 have 8 modules of 256MB 400MHz DDR ECC REG DIMM.
−
−	Each of the units in the second rack (from c1-0 to c1-15) features 2 Dual-Core AMD Opteron Processors 2216 HE (each with 2x1MB L2 cache) working at 2,4GHz and 4GB of RAM.
−
−
−
−	COMPLEX_NAME: '''opteron244'''
−
−	- 2 Single-Core AMD Opteron244 @ 1,75GHz
−
−	nodes: c0-0, c0-1, c0-2, c0-3, c0-4, c0-5, c0-6, c0-7, c0-8, c0-9, c0-10, c0-11, c0-12, c0-13, c0-14, c0-15, c0-16, c0-17, c0-18, c0-19, c0-20, c0-21, c0-22, c0-23, c0-24, c0-25, c0-26, c0-27, c0-28, c0-29, c0-30, c0-31
−
−
−	COMPLEX_NAME: '''opteron2216'''
−
−	- 2 Dual-Core AMD Opteron2216 HE @ 2,4GHz
−
−	nodes: c1-0, c1-1, c1-2, c1-3, c1-4, c1-5, c1-6, c1-7, c1-8, c1-9, c1-10, c1-11, c1-12, c1-13, c1-14, c1-15
−
−
−	=== Cluster status ===
−
−	{\| border=1 cellspacing=0 cellpadding=2
−	! Node !! Effect !! Date !! Note
−	\|-
−	\| c0-0 \|\| \|\| 20 Feb 2008 \|\|
−	\|-
−	\| c0-1 \|\| reserved to ESA project \|\| 20 Feb 2008 \|\| to submit jobs here add -l esa (HW: first NIC broken, uses spare NIC now)
−	\|-
−	\| c0-2 \|\| reserved to ESA project \|\| 30 Nov 2007 \|\| to submit jobs here add -l esa
−	\|-
−	\| c0-3 \|\| reserved to ESA project \|\| 3 Dec 2007 \|\| to submit jobs here add -l esa
−	\|-
−	\| c0-4 \|\| reserved to ESA project \|\| 30 Nov 2007 \|\| to submit jobs here add -l esa
−	\|-
−	\| c0-5 \|\| reserved to ESA project \|\| 3 Dec 2007 \|\| to submit jobs here add -l esa
−	\|-
−	\| c0-6 \|\| reserved to ESA project \|\| 4 Dec 2007 \|\| to submit jobs here add -l esa
−	\|-
−	\| c0-7 \|\| reserved to ESA project \|\| 4 Dec 2007 \|\| to submit jobs here add -l esa
−	\|-
−	\| c0-8 \|\| reserved to ESA project \|\| 4 Dec 2007 \|\| to submit jobs here add -l esa
−	\|-
−	\| c0-9 \|\| \|\| 20 Feb 2008 \|\| (HW: first NIC broken, uses spare NIC now)
−	\|-
−	\| c0-10 \|\| \|\| \|\|
−	\|-
−	\| c0-11 \|\| \|\| \|\|
−	\|-
−	\| c0-12 \|\| \|\| \|\|
−	\|-
−	\| c0-13 \|\| \|\| \|\|
−	\|-
−	\| c0-14 \|\| \|\| \|\|
−	\|-
−	\| c0-15 \|\| \|\| \|\|
−	\|-
−	\| c0-16 \|\| reserved to ESA project \|\| 23 Jan 2008 \|\| to submit jobs here add -l esa
−	\|-
−	\| c0-17 \|\| \|\| \|\|
−	\|-
−	\| c0-18 \|\| \|\| \|\|
−	\|-
−	\| c0-19 \|\| \|\| \|\|
−	\|-
−	\| c0-20 \|\| \|\| \|\|
−	\|-
−	\| c0-21 \|\| \|\| \|\|
−	\|-
−	\| c0-22 \|\| \|\| \|\|
−	\|-
−	\| c0-23 \|\| \|\| \|\|
−	\|-
−	\| c0-24 \|\| \|\| 20 Feb 2008 \|\| (HW: first NIC broken, uses spare NIC now)
−	\|-
−	\| c0-25 \|\| offline \|\| 20 Feb 2008 \|\| everything broken
−	\|-
−	\| c0-26 \|\| \|\| \|\|
−	\|-
−	\| c0-27 \|\| \|\| \|\|
−	\|-
−	\| c0-28 \|\| reserved to ESA project \|\| 23 Jan 2008 \|\| to submit jobs here add -l esa
−	\|-
−	\| c0-29 \|\| \|\| 5 Feb 2008 \|\| (HW: first NIC broken, uses spare NIC now)
−	\|-
−	\| c0-30 \|\| \|\| \|\|
−	\|-
−	\| c0-31 \|\| reserved to ESA project \|\| 23 Jan 2008 \|\| to submit jobs here add -l esa
−	\|-
−	\| c1-0 \|\| \|\| \|\|
−	\|-
−	\| c1-1 \|\| \|\| \|\|
−	\|-
−	\| c1-2 \|\| \|\| \|\|
−	\|-
−	\| c1-3 \|\| \|\| \|\|
−	\|-
−	\| c1-4 \|\| \|\| 5 Feb 2008 \|\| (HW: problem with CMOS battery, may not boot up automatically)
−	\|-
−	\| c1-5 \|\| \|\| \|\|
−	\|-
−	\| c1-6 \|\| \|\| \|\|
−	\|-
−	\| c1-7 \|\| \|\| \|\|
−	\|-
−	\| c1-8 \|\| \|\| \|\|
−	\|-
−	\| c1-9 \|\| \|\| \|\|
−	\|-
−	\| c1-10 \|\| \|\| \|\|
−	\|-
−	\| c1-11 \|\| \|\| \|\|
−	\|-
−	\| c1-12 \|\| \|\| \|\|
−	\|-
−	\| c1-13 \|\| \|\| \|\|
−	\|-
−	\| c1-14 \|\| \|\| \|\|
−	\|-
−	\| c1-15 \|\| \|\| \|\|
−	\|-
−	\|}
−
−	== Queues ==
−
−	Each computational node has the following queues; each queue has as many slot as the number of cores in that node:
−
−
−	*'''<node>.short''': jobs run in this queue at nice-level 2 for '''maximum 24h of CPU time''' (real execution of the program, without counting the time needed by the system for multitasking, etc). If a job still runs after the 24th hour, it will receive a signal SIGUSR1 and after some more time a SIGKILL that will terminate it.
−
−	*'''<node>.medium''': jobs run in this queue at nice-level 3 (lower priority than the short ones) for '''maximum 72h of CPU time''' (real execution of the program, without counting the time needed by the system for multitasking, etc). If a job still runs after the 72nd hour, it will receive a signal SIGUSR1 and after some more time a SIGKILL that will terminate it.
−
−	*'''<node>.long''': jobs run in this queue at nice-level 3 (lower priority than the short ones) for '''maximum 168h of CPU time''' (real execution of the program, without counting the time needed by the system for multitasking, etc). If a job still runs after the 168th hour, it will receive a signal SIGUSR1 and after some more time a SIGKILL that will terminate it.
−
−
−	Summarizing: on a node up to 6 jobs (Opteron 244 nodes) or 12 jobs (Opteron 2216 nodes) can run concurrently with an average space in RAM of 341MB per job.
−
−
−
−	'''YOU HAVE TO DESIGN YOUR COMPUTATIONS IN SUCH A WAY THAT EACH SINGLE JOB DOESN'T RUN FOR MORE THAN 7 DAYS (of CPU time)'''.
−
−	== How to submit a job ==
−
−
−	To submit a job you have to create a script (that we indicate with SCRIPT_NAME.sh) with the commands you want to execute. Once you have the script, from majorana execute the command qsub SCRIPT_NAME.sh
−
−
−	The script should begin like this:
−
−	#!/bin/bash
−	#$ -N NAME_OF_JOB
−	#$ -cwd
−
−	In this case the job will be scheduled for execution in one node chosen from any node, and in one queue chosen from any queue of that node.
−
−
−	If you want, you can impose some restrictions on the kind of node or queue to choose from.
−
−	To select a node with Opteron 244 CPUs, you must add to your script the line '''#$ -l opteron244''', while to select a node with Opteron 2216 CPUs, you must add to your script the line '''#$ -l opteron2216'''.
−	To select only among the short queues, you must add to your script the line '''#$ -l short'''; to select only among the medium queues, you must add the line '''#$ -l medium'''; to select only among the long queues, you must add to your script the line '''#$ -l long'''.
−
−
−	'''THE SCHEDULER ADDS TO THE QUEUEING SYSTEM A JOB ONLY IF THERE ARE LESS THAN 10000 JOBS IN THE QUEUE; A SAME USER CAN ADD A JOB TO THE QUEUEING SYSYEM ONLY IF HE HAS LESS THAN 1500 JOBS IN THE QUEUE'''.
−
−
−	'''IF THERE IS AN EXECUTION SLOT AVAILABLE, THE SCHEDULER ASSIGN IT TO A USER JOB ONLY IF THAT USER HAS LESS THAN 128 JOBS ALREADY RUNNING'''.
−
−	== Programming tips for the cluster ==
−
−	If the jobs needs to read/write quite much and often, in the submission script it is better to copy the input files to the /tmp directory (which is in the local harddrive of the node) and to write the output files also there, moving them in the /home/user_name directory only when the computation is over. In this way your job does not have to use NFS for each read/write operation relieving majorana of some weight (the /home partition is exported from there to all the nodes), making it more fast.
−
−	'''REMEMBER TO REMOVE YOUR FILES FROM THE /TMP DIRECTORY ONCE THE COMPUTATION IS OVER'''

@@ Line 49: / Line 49: @@
 | c0-9 || || 20 Feb 2008  ||   (HW: first NIC broken, uses spare NIC now)
 |-
-| c0-10 || converted node to new partition layout  ||  ||
+| c0-10 || ||  ||
 |-
 | c0-11 ||  ||  ||

@@ Line 49: / Line 49: @@
 | c0-9 || || 20 Feb 2008  ||   (HW: first NIC broken, uses spare NIC now)
 |-
-| c0-10 ||  ||  ||
+| c0-10 || converted node to new partition layout  ||  ||
 |-
 | c0-11 ||  ||  ||

@@ Line 77: / Line 77: @@
 | c0-23 ||  ||  ||
 |-
-| c0-24 ||  ||  ||
+| c0-24 ||  || 20 Feb 2008 ||  (HW: first NIC broken, uses spare NIC now)
 |-
 | c0-25 || offline || 20 Feb 2008 || everything broken

@@ Line 69: / Line 69: @@
 | c0-19 ||  ||  ||
 |-
-| c0-20 ||  || 20 Feb 2008 ||
+| c0-20 ||  ||  ||
 |-
 | c0-21 ||  ||  ||
 |-
-| c0-22 ||  || 20 Feb 2008 ||
+| c0-22 ||  ||  ||
 |-
 | c0-23 ||  ||  ||
 |-
-| c0-24 || offline || 20 Feb 2008 || everything broken
+| c0-24 ||  ||  ||
 |-
-| c0-25 ||  || 20 Feb 2008 ||
+| c0-25 || offline || 20 Feb 2008 || everything broken
 |-
 | c0-26 ||  ||  ||

← Older revision		Revision as of 17:31, 20 February 2008
Line 169:		Line 169:


−	'''IF THERE IS AN EXECUTION SLOT AVAILABLE, THE SCHEDULER ASSIGN IT TO A USER JOB ONLY IF THAT USER HAS LESS THAN 64 JOBS ALREADY RUNNING'''.	+	'''IF THERE IS AN EXECUTION SLOT AVAILABLE, THE SCHEDULER ASSIGN IT TO A USER JOB ONLY IF THAT USER HAS LESS THAN 128 JOBS ALREADY RUNNING'''.

	== Programming tips for the cluster ==		== Programming tips for the cluster ==

@@ Line 29: / Line 29: @@
 ! Node !! Effect !! Date !! Note
 |-
-| c0-0 || offline  || 13 Jul 07  || cpu-fan broken
+| c0-0 ||  || 20 Feb 2008  ||
 |-
-| c0-1 || offline (reserved to ESA project) || 23 Jan 2008 || unknown cause, needs to be checked
+| c0-1 || reserved to ESA project || 20 Feb 2008 || to submit jobs here add -l esa (HW: first NIC broken, uses spare NIC now)
 |-
-| c0-2 || reserved to ESA project || 30 Nov 2007  ||  to submit jobs here add -l esa
+| c0-2 || reserved to ESA project || 30 Nov 2007 || to submit jobs here add -l esa
 |-
-| c0-3 || reserved to ESA project || 3 Dec 2007 || to submit jobs here add -l esa
+| c0-3 || reserved to ESA project ||  3 Dec 2007 || to submit jobs here add -l esa
 |-
-| c0-4 ||  reserved to ESA project || 30 Nov 2007 ||  to submit jobs here add -l esa
+| c0-4 || reserved to ESA project || 30 Nov 2007 || to submit jobs here add -l esa
 |-
 | c0-5 || reserved to ESA project || 3 Dec 2007  || to submit jobs here add -l esa
 |-
-| c0-6 || reserved to ESA project || 4 Dec 2007 || to submit jobs here add -l esa
+| c0-6 || reserved to ESA project ||  4 Dec 2007 || to submit jobs here add -l esa
 |-
-| c0-7 || reserved to ESA project || 4 Dec 2007 || to submit jobs here add -l esa
+| c0-7 || reserved to ESA project ||  4 Dec 2007 || to submit jobs here add -l esa
 |-
-| c0-8 || reserved to ESA project || 4 Dec 2007  ||to submit jobs here add -l esa
+| c0-8 || reserved to ESA project ||  4 Dec 2007 || to submit jobs here add -l esa
 |-
-| c0-9 || offline || 24 Aug 07  || ethernet card not recognized
+| c0-9 || || 20 Feb 2008  ||   (HW: first NIC broken, uses spare NIC now)
 |-
 | c0-10 ||  ||  ||
@@ Line 69: / Line 69: @@
 | c0-19 ||  ||  ||
 |-
-| c0-20 || offline || 18 Feb 2008 || unknown cause, needs to be checked
+| c0-20 ||  || 20 Feb 2008 ||
 |-
 | c0-21 ||  ||  ||
 |-
-| c0-22 || offline || 18 Feb 2008 || unknown cause, needs to be checked
+| c0-22 ||  || 20 Feb 2008 ||
 |-
 | c0-23 ||  ||  ||
 |-
-| c0-24 || offline || 13 Jul 2007 || power supply broken
+| c0-24 || offline || 20 Feb 2008 || everything broken
 |-
-| c0-25 || offline || 23 Jan 2008 || unknown cause, needs to be checked
+| c0-25 ||  || 20 Feb 2008 ||
 |-
 | c0-26 ||  ||  ||
 |-
-| c0-27 ||  ||   ||
+| c0-27 ||  ||  ||
 |-
 | c0-28 || reserved to ESA project || 23 Jan 2008  || to submit jobs here add -l esa
 |-
-| c0-29 || first NIC broken, switched of in BIOS, uses spare NIC now  || 5 Feb 2008 ||
+| c0-29 ||  || 5 Feb 2008 || (HW: first NIC broken, uses spare NIC now)
 |-
 | c0-30 ||  ||   ||
@@ Line 101: / Line 101: @@
 | c1-3 ||  ||  ||
 |-
-| c1-4 || problem with CMOS battery, may not boot up automatically || 5 Feb 2008 ||
+| c1-4 ||  || 5 Feb 2008 || (HW: problem with CMOS battery, may not boot up automatically)
 |-
 | c1-5 ||  ||  ||