dell servers
I have adopted 4 dell poweredge servers from RINIS. This set me back 350 euro so far because it had to be completed with more memory.
However this chapter is more about silencing the noisy fans inside the chassis. There are 12 ! in total ! 6 blocks with two fans on either side.
I will try to describe the best way to keep noise down for these monsters.
tools
Some useful tools will be described next
- ipmitool for setting and reading fan speeds
- stress to stress test cpu's
- sensors to read core temperatures
idrac
Mentioned a lot: it means : integrated dell remote access controller linux
ipmitool
(IPMI) is a standardized message-based hardware management interface. At the core of the IPMI is a hardware chip that is known as the Baseboard Management Controller (BMC), or Management Controller (MC).24 Dec 2019
ipmitool sensor reading "temp" "Fan1A" "Fan1B" "Fan2A" "Fan2B" "Fan3A" "Fan4B" "Fan4A" "Fan4B" "Fan5A" "Fan5B" "Fan6A" "Fan6B"
This gives the rpms for each fan, this is with the lowest we can get these via the bios :
ipmitool sensor reading "temp" "Fan1A" "Fan1B" "Fan2A" "Fan2B" "Fan3A" "Fan4B" "Fan4A" "Fan4B" "Fan5A" "Fan5B" "Fan6A" "Fan6B"
IANA PEN registry open failed: No such file or directory
Sensor "temp" not found!
Fan1A | 2640
Fan1B | 2520
Fan2A | 7560
Fan2B | 7200
Fan3A | 6600
Fan4B | 6360
Fan4A | 6600
Fan4B | 6360
Fan5A | 6600
Fan5B | 6360
ipmitool sdr get "Fan1A" "Fan1B" "Fan2A" "Fan2B" "Fan3A" "Fan4B" "Fan4A" "Fan4B" "Fan5A" "Fan5B" "Fan6A" "Fan6B"
For just fan 6B you get this information
Sensor ID : Fan6B (0x3b)
Entity ID : 7.1 (System Board)
Sensor Type (Threshold) : Fan (0x04)
Sensor Reading : Disabled
Status : Not Available
Nominal Reading : 6720.000
Normal Minimum : 16680.000
Normal Maximum : 23640.000
Lower critical : 720.000
Lower non-critical : 840.000
Positive Hysteresis : 120.000
Negative Hysteresis : 120.000
Minimum sensor range : Unspecified
Maximum sensor range : Unspecified
Event Message Control : Per-threshold
Readable Thresholds : lcr lnc
Settable Thresholds :
Threshold Read Mask : lcr lnc
Assertion Events :
Assertions Enabled : lnc- lcr-
Deassertions Enabled : lnc- lcr-
First we need to enable the manual control to alter fan speeds.
ipmitool raw 0x30 0x30 0x01 0x00 # enable manual
ipmitool raw 0x30 0x30 0x01 0x01 # disable manual again
Note that these commands survive a reboot ! sauce : https://docs.oracle.com/cd/E19469-01/820-6413-13/IPMI_Overview.html
Then we can set the speeds with the next commands. You can exactly hear if it works or not. The last parameter sets the percentage in hex format. So i only use multiples of 16 for convenience. But for now let's set it to 20.
The last number means the percentage in hex (0x14 means 20 %) so you could also try lower and even off !!
So here are some extremes : 0, 50% and 100 %
ipmitool raw 0x30 0x30 0x02 0xff 0x00
ipmitool raw 0x30 0x30 0x02 0xff 0x32
ipmitool raw 0x30 0x30 0x02 0xff 0x64
Never go above 40, it is terribly noisy and it never gets that hot !!
sauce : https://gist.github.com/slykar/f90ad596b18d5ab1eb1c66b2ccf51c54#set-fan-speed-to-30-
100%
This is really like all hell breaks loose, it is also totally unnecessary as we will see later. So we will never go here again. Actually 50% is too high for any purpose. To set some boundaries : if we hog all cpus to 100% we get these results.
Note that compiling klopt code and generating networks is about as intensive as this stress test !
coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +92.0°C (high = +82.0°C, crit = +92.0°C)
Core 0: +89.0°C (high = +82.0°C, crit = +92.0°C)
Core 1: +89.0°C (high = +82.0°C, crit = +92.0°C)
Core 2: +88.0°C (high = +82.0°C, crit = +92.0°C)
Core 3: +89.0°C (high = +82.0°C, crit = +92.0°C)
Core 4: +88.0°C (high = +82.0°C, crit = +92.0°C)
Core 8: +88.0°C (high = +82.0°C, crit = +92.0°C)
Core 9: +88.0°C (high = +82.0°C, crit = +92.0°C)
Core 10: +88.0°C (high = +82.0°C, crit = +92.0°C)
Core 11: +88.0°C (high = +82.0°C, crit = +92.0°C)
Core 12: +87.0°C (high = +82.0°C, crit = +92.0°C)
Note that we see only half of the 20 cores, which has something to do with :
Maybe it is a different i5 CPU (Some are manufactured with 2 physical cores, some with 4, and some have HT enabled, which double the number of cores which the os sees. Check wiki Core i5 and run this cat /proc/cpuinfo
measurements
If we don't do anything, we can almost disable the fans. However they don't go lower than 0% but the fans are still turning. At about these speeds
Fan1A | 4800
Fan1B | 4440
Fan2A | 4800
Fan2B | 4440
Fan3A | 4920
Fan4B | 4440
Fan4A | 4920
Fan4B | 4440
Fan5A | 4800
Fan5B | 4440
Dell states that cores handle excess heat by either shutdown or throttling. Of course we want the last one and this also seems the default when testing.
0% work
With no work at all the fan speeds deliver these temperatures
- fanspeed 0% low use : 38 degrees
- fanspeed 16% low use : 28 degrees
- fanspeed 32% low use : 25 degrees
38 is perfectly fine so in rest just set speed to 0%, but also with higher stress we can manage :
100% work
With the stress command on all cores
Also we can monitor the (a) processor speed with :
- fanspeed 0% seems to throttle at 92 exactly (see below)
- fanspeed 10% goes around 90 degrees : freq : 2397409 (light throttling)
- fanspeed 20% goes down to 70 degrees freq around : 2597202
- fanspeed 30% about 57 (54-59)
- fanspeed 40% goes down to 50 degrees freq stays : 2597196
The throttling seems to work ! with fanspeed 0 the temperature rises to about 91 degrees and the speed keeps going down : below 200000 -> 1998230 -> 1694683 ...
conclusion
Use fan speed 0 for normal operation, compilation, data generation. Use fan speed 20 % for hard work. 10 goes up to 90 degrees and starts throttling.
Best is just to keep the server out of the office and use 20-30% for any use.
Setting 0% is not dangerous because of throttling, but not wise either : the processor will last longer when not becoming too hot and some speed is needed anyway for running.