Dalton Users Documentation

luckyluke :

• C'est une VM lente avec peu de CPU et mémoire. Ne pas faire de calcul ou compiler dessus.
• Accessible sous le nom dalton.bordeaux.inria.fr uniquement depuis les passerelles 'acces'
  • Documentation
  • Ticket pour demander un accès

Les autres daltons ne sont accessibles que depuis ce frontal.

Pour aller directement sur le frontal depuis l'extérieur en tapant `ssh dalton`, mettre dans le `.ssh/config` de la machine source :

Host dalton
User <login_dalton>
ProxyCommand ssh <login_inria>@acces.bordeaux.inria.fr nc dalton.bordeaux.inria.fr 22

Comme l'authentification se fait par clef ssh, il faut probablement propager votre agent ssh en ajoutant tout en haut du `.ssh/config` :

ForwardAgent yes

• betty
  • Dual-processeur AMD EPYC Icosikaitetra-core (2*24 = 48 coeurs, 48 threads) Zen2 Rome EPYC 7352 2.3GHz (x86_64)
  • 2*16Go NUMA
  • 2* GPU NVIDIA Tesla T4 16Go
  • 1* GPU NVIDIA Quatro RTX 8000 48Go
• peabody
  • Dual-processeur Intel Xeon Icosikaihexa-core (2*26 = 52 coeurs, 104 threads) Cascade Lake Gold 6230R 2.1GHz (x86_64)
  • 2*16Go NUMA
• billy0, billy1
  • Dual-processeur AMD EPYC Dodriaconta-core (2*32 = 64 cœurs, ~128 threads~) Zen2 Rome EPYC 7502 2.5GHz (x86_64)
  • 8*16Go NUMA
  • InfiniBand ConnectX-6 HDR
  • Atos BXI v1.3 (il faut booter avec le vieux noyau 4.19)
• leonide (le grand duc)
  • Dual-processeur Intel Xeon Icosa-core Hyperthreadé (2*20 = 40 cœurs, 80 threads) Cascade Lake Gold 6230 v2 2.1GHz (x86_64)
  • 2*192Go DDR NUMA + 2*768Go NVDIMM NUMA
• henri0, henri1
  • Dual-processeur Intel Xeon Octakaideca-core ~Hyperthreadé~ (2*18 = 36 cœurs, ~72 threads~) Skylake Gold 6140 2.3GHz (x86_64)
  • 2*48Go NUMA
  • Infiniband ConnectX-4 EDR
  • Atos/Bull BXI 1.3
• joe0, joe1
  • Dual-processeur Intel Xeon Tetrakaideca-core Hyperthreadé (2*14 = 28 cœurs, 56 threads) Broadwell E5-2650L v4 1.7GHz (x86_64)
  • 2*32Go RAM NUMA
  • 2*GPU AMD Radeon R9 Fiji XT Nano sur averell1
  • Infiniband ConnectX-4 EDR
• william0, william1 :
  • Dual-processeur Intel Xeon Octo-core Hyperthreadé (2*8 = 16 cœurs, 32 threads) Sandy Bridge E5-2650 2GHz (x86_64)
  • 2*16Go RAM NUMA
  • 1* NVIDIA Tesla M2075
  • Infiniband ConnectX-3 FDR
• ma (ancien frontal)
  • Mono-processeur Intel Xeon Hexa-core (1*6 = 6 cœurs) Haswell E5-2603v3 1.6GHz (x86_64)
  • 32*Go RAM

Les machines finissant par un numéro disposent de réseaux rapides (InfiniBand et/ou BXI) mais elles ne sont connectées que par paire, sans switch.

dalton32 est une VM 32bits fournie par la DSI avec les comptes/homes habituels des daltons.

If a machine doesn't answer to ssh, it might be off to save baby seals. To check this out, use

luckyluke $ ipmi status william0
Chassis power is off

If it's off, to switch it on you just need to run

luckyluke $ ipmi on william0
Chassis Power Control: Up/On

And wait for a few minutes so it can start.

To save baby seals, machines stop automatically when they are not used for some time. To know if a node is being used, we check whether

• somebody logged in recently
• it was switched on recently (and thus somebody will surely use it soon)
• a SCREEN session is opened
• the machine is reserved
• buildbot is running some tests
• a file starting with /tmp/autoshutdown-disabled exists (to prevent manually the machine shutdown)

If all these conditions are false, the machine stops. This test is done every hour.

The recently word depends on the time. On the week-end it is since 4 hours. During the week it is since 12 hours.

The ipmi script is an alias over a horrible command line which allows to control all nodes (except ma).

To see the available parameters, run the script without parameters.

To know if jolly is on

$ ipmi status jolly
Chassis Power is on

To switch averell0 on

$ ipmi on averell0

To shut a machine down properly (without having to connect and start sudo halt)

$ ipmi off averell1

The shutdown is software (just like sudo halt), so it can take time, or even fait. Thus, wait a bit to know whether it worked

$ ping averell1
...
<wait for not pinging any more>
...
$ ipmi status averell1
Chassis power is off

If a machine is hung and one can't do sudo reboot, one can force a reboot. But always prefer sudo reboot whenever possible.

$ ipmi reset william0

What's more, if sudo halt is impossible, and if proper shutdown doesn't work, one can force a shutdwon

$ ipmi forceoff averell1
Chassis Power Control: Down/Off
<attendre>
$ ipmi status averell1
Chassis power is off

Before running several commands in a row, check that the previous one has been taken into account. For instance a forceoff will only have been completely when the status is really switched to off. In the meanwhile it might stay on or ipmi might not answer for a few seconds.

To see the status of the the jack1 sensors, even if off

$ ipmi sensors jack1
...
Temp             | 31 degrees C      | ok
Temp             | disabled          | ns
Ambient Temp     | 20 degrees C      | ok
Planar Temp      | 35 degrees C      | ok
...

To check the hardware logs

$ ipmi sel william0
 1 | 05/15/2012 | 16:42:28 | Event Logging Disabled #0x72 | Log area reset/cleared | Asserted
 2 | 05/25/2012 | 13:35:10 | Power Supply #0x63 | Power Supply AC lost | Asserted
 3 | 05/25/2012 | 13:35:11 | Power Supply #0x74 | Redundancy Lost
 4 | 05/31/2012 | 10:29:08 | Physical Security #0x73 | General Chassis intrusion | Asserted
 5 | 05/31/2012 | 10:29:14 | Physical Security #0x73 | General Chassis intrusion | Deasserted

To access the serial console over IPMI:

$ ipmi sol jolly
Info: SOL payload already de-activated
[SOL Session operational.  Use [? for help]

Type enter, and a login prompt should appear

[SOL Session operational.  Use [? for help]
Debian GNU/Linux lenny/sid jolly ttyS1
jolly login: bgoglin
Password: 
Linux jolly 2.6.26-1-amd64 #1 SMP Thu Aug 28 11:13:42 UTC 2008 x86_64
bgoglin@jolly:~$ 

One can then log in and do usual stuff, in particular reboot with sudo reboot. One can also use this to see the kernel logs (and thus save them remotely).

To quit it, type <enter> then '[' then '.'

jolly login:   
Debian GNU/Linux lenny/sid jolly ttyS1
jolly login: [. [terminated ipmitool]
$

This technique is really useful if the kernel is failing because it allows to see the console event if the network driver or board don't work!