ATCIntel Skylake ProcessorsIntel Cascade Lake ProcessorsHPE DL380Dell EMC PowerEdge R740IntelATC InsightServer InfrastructureDell TechHPEData Center
ATC Insight • • 10 minute read

To Turbo, or not to Turbo? That is the question.

A customer came to the ATC in search of an answer to a question "What is the effect on power consumption when enabling the Intel Turbo Boost feature on Dell and HPE servers of various hardware configurations?" The answer to this question is something that we at the ATC have been wondering as well. We expected the turbo would increase the power consumption but how much and would the results change by vendor? To answer these questions we utilized a tool called BurnInTest to put a consistent load on the servers. We recorded the results using Emerson PDUs reporting in SolarWinds at 1 minute intervals. We also compared the data collected to the onboard power utilities in Dell's iDRAC and HPE's iLO as a second data point. Curiosity drove another question, "Do the onboard consoles of these platforms report power accurately?" The answers to these questions may surprise you, they certainly surprised us.

In this ATC Insight

 ATC Insight

Read more about WWT's relationship with Intel.Partner

 Intel Turbo Boost On vs Turbo Boost Disabled

When we started down this journey of testing the impact on power consumption by Intel's Turbo feature on HPE and Dell servers, we thought that we would see an increase of about 5-10 percent in power usage (watts). We were flat wrong! The increase in power usage rose as high as 20 percent! Our thought here is if you don't need the turbo feature then do your pocket book and carbon footprint a favor and turn it off. But, don't stop here, let's explore this further below...

Here is a bit of info on how turbo affects the operation of a processor and why this is important to the testing performed. Turbo changes the maximum speeds at which a processor can operate based on an algorithm that considers various sensor statistics. This data allows the system to fluctuate the CPU operating frequencies effectively increasing and decreasing speeds while maintaining stability. This is important to note because any change in the testing environment, whether it be temperature or operations, can skew the results. To combat this, the ATC Lab Services team made every attempt to maintain a consistent test for all devices. It is also important to note that the amount of actual processing was not measured during this test. What we measured was power consumption when the system was pushed to it's max. Visit Intel's page for a more detailed description of Turbo and check out their short video explaining how it works. 

All the tested servers used as similar a configuration as possible when working with Dell and HPE. They had identical memory configurations, CPUs and disks during each of the tests. The left side of the charts shows the PDU results; the right side shows the results as reported to iLO or iDRAC respectability.

In order to shed some light on the results, we compared the averaged results over 15 minutes of load with turbo on and off. We then figured out the percentage of change to try to figure out what is really going on and how it affected power consumption. 

 Cascade Lake Testing

There were a few surprises that came to light during this test. One was the clear difference in the "% Decreased" power draw between the Dell and HPE products when turbo was off vs on. Dell seemed to hang around the upper teen percentages and even breaching the 20% savings when turbo was off. Conversely, HPE's "% Decreased" rate hovered in the low percentage teens. 

This hints towards a difference in the vendor's devices ability to ramp up the processing frequency. If both systems ramped up in similar ways then we would expect to see a more similar rate of change. Further testing is needed if we wanted to verify any notable processing improvements but this does beg the question. If turning off the turbo feature doesn't lead to large savings in power then is the feature being exploited to the fullest potential? 

Cascade Lake at 100% CPU

Another nugget of information that came out of these tests is that when the servers were not pushed to 100% CPU the vendors behaved differently. Power in relation to "% Decreased" on the HPE servers were mostly consistent or even trending towards an increase in savings.  The Dell power percentage savings generally went down. 

We already know the turbo feature adapts to increased workload. As the workload decreases it just makes sense that there would be less savings. The results on the HPE test do not follow this theory, and that is very interesting. The reason for this could be that the turbo feature is not exploited as much with HPE as it is with Dell, but this is just a guess. More testing would be required to prove this theory. 

Cascade Lake at 80% CPU

Although we did not test at lower CPU utilization levels, the savings are expected to continue to dwindle as utilization dwindles. 

 Skylake Testing

Skylake processors were the foundation for the construction of the Cascade Lake processors. Because of this we expected the test results to be very similar between the two, and we were right. Below is the results of the Skylake testing for both the Dell and HPE Platforms. Note that the HPE servers performed almost identical during both processor tests. The Dell servers on the other hand generally used less power with Cascade lake processors in the previous test (results above graph). 

Skylake at 100% CPU

 Console vs PDU

The console of the HPE and Dell servers provide all kinds of information around performance. We typically just take this information at face value because we have nothing to verify it's authenticity. One of the outcomes of this test is to verify the power readings of the onboard sensors respective to the PDU readings. 

The PDU's typically reported only a small difference on Dell's readings for the tests. This tells us iDRAC (Dell's console) is pretty close to recording the same results from the PDU's. 

HPE was on average 5-7% lower than what the PDU's recorded. This is still close to the PDU's readings but it does point to our theory that each vendor calculates power consumption in their own way. 

That being said, we are not saying Dell is more accurate than HPE in power readings. This simply indicates that the PDUs and Dell have a more similar method of calculating power consumption than HPE in this case (and based on our specific tests).  

Does this mean you can trust the console readings to give accurate results? We think so. But if you need more accurate recordings across a variety of vendors, then using a recording device that is vendor agnostic will provide consistent results. 

 Conclusion

Intel Turbo Boost is available on both HPE and Dell servers with Intel based processors. While both OEMs did operate somewhat differently they generally operated in the same fashion. Both OEMs showed an increase in power consumption when Turbo was enabled, especially at high CPU load. Intel Turbo Boost is a neat feature that allows those heavily tasked servers to potentially operate at a higher speed. This feature, however, comes with a cost, and that cost is power consumption and presumably cooling. So if power and cooling are major concerns in your data centers, then you need to be concerned with the state of the Intel Turbo Boost feature in your BIOS. 

One of the most important lessons we learned is that BIOS features can make a vast difference in power consumption and presumably the speed at which your server operates. Pay attention to that often overlooked BIOS configuration menu because it could be costing you in resources.

 Test Plan/Test Case

 High-Level Test Plan

  1. Set BIOS settings to customer specifics
  2. PXE boot the servers into Windows operating environment
  3. Setup Passmark BurnInTest to 80% or 100% CPU (always setting memory utilization to 100%)
  4. Run Passmark BurnInTest software test
  5. Record power draw (average and max) for PDU1, PDU2, and iDRAC/iLO

 Baseline BIOS Settings

All BIOS settings are recorded in the documentation section of the ATC Insight via an excel spreadsheet. If you want to go into the weeds and see each bios setting please feel free to review the Excel doc.

Skylake Testing

100% CPU Load (Turbo Boost Enabled)

  • Dell Compute VDI (R640, 768GB RAM, 2x 240GB SSD)
  • Dell Compute VSI (R640, 1.5TB RAM, 2x 240GB SSD)
  • Dell Performance (R640, 96GB RAM, 32GB NVDIMM, 8x 1.92TB SSD, 2x 240GB SSD)
  • Dell Capacity (R740xd, 96GB RAM, 96GB NVDIMM, 24x 3.84TB SSD, 2x 240GB SSD)
  • HPE Compute VDI (DL360, 768GB RAM, 2x 240GB SSD)
  • HPE Compute VSI (DL360, 1.5TB RAM, 2x 240GB SSD)
  • HPE Performance (DL360, 96GB RAM, 32GB NVDIMM, 8x 1.92TB SSD, 2x 240GB SSD)
  • HPE Capacity (DL380, 96GB RAM, 96GB NVDIMM, 24x 3.84TB SSD, 2x 240GB SSD)

100% CPU Load (Turbo Boost Disabled)

  • Dell Compute VDI (R640, 768GB RAM, 2x 240GB SSD)
  • Dell Compute VSI (R640, 1.5TB RAM, 2x 240GB SSD)
  • Dell Performance (R640, 96GB RAM, 32GB NVDIMM, 8x 1.92TB SSD, 2x 240GB SSD)
  • Dell Capacity (R740xd, 96GB RAM, 96GB NVDIMM, 24x 3.84TB SSD, 2x 240GB SSD)
  • HPE Compute VDI (DL360, 768GB RAM, 2x 240GB SSD)
  • HPE Compute VSI (DL360, 1.5TB RAM, 2x 240GB SSD)
  • HPE Performance (DL360, 96GB RAM, 32GB NVDIMM, 8x 1.92TB SSD, 2x 240GB SSD)
  • HPE Capacity (DL380, 96GB RAM, 96GB NVDIMM, 24x 3.84TB SSD, 2x 240GB SSD)

Cascade Lake Testing

80% CPU Load (Turbo Boost Enabled)

  • Dell Compute VDI (R640, 768GB RAM, 2x 240GB SSD)
  • Dell Compute VSI (R640, 1.5TB RAM, 2x 240GB SSD)
  • Dell Performance (R640, 96GB RAM, 32GB NVDIMM, 8x 1.92TB SSD, 2x 240GB SSD)
  • Dell Capacity (R740xd, 96GB RAM, 96GB NVDIMM, 24x 3.84TB SSD, 2x 240GB SSD)
  • HPE Compute VDI (DL360, 768GB RAM, 2x 240GB SSD)
  • HPE Compute VSI (DL360, 1.5TB RAM, 2x 240GB SSD)
  • HPE Performance (DL360, 96GB RAM, 32GB NVDIMM, 8x 1.92TB SSD, 2x 240GB SSD)
  • HPE Capacity (DL380, 96GB RAM, 96GB NVDIMM, 24x 3.84TB SSD, 2x 240GB SSD)

80% CPU Load (Turbo Boost Disabled)

  • Dell Compute VDI (R640, 768GB RAM, 2x 240GB SSD)
  • Dell Compute VSI (R640, 1.5TB RAM, 2x 240GB SSD)
  • Dell Performance (R640, 96GB RAM, 32GB NVDIMM, 8x 1.92TB SSD, 2x 240GB SSD)
  • Dell Capacity (R740xd, 96GB RAM, 96GB NVDIMM, 24x 3.84TB SSD, 2x 240GB SSD)
  • HPE Compute VDI (DL360, 768GB RAM, 2x 240GB SSD)
  • HPE Compute VSI (DL360, 1.5TB RAM, 2x 240GB SSD)
  • HPE Performance (DL360, 96GB RAM, 32GB NVDIMM, 8x 1.92TB SSD, 2x 240GB SSD)
  • HPE Capacity (DL380, 96GB RAM, 96GB NVDIMM, 24x 3.84TB SSD, 2x 240GB SSD)

100% CPU Load (Turbo Boost Enabled)

  • Dell Compute VDI (R640, 768GB RAM, 2x 240GB SSD)
  • Dell Compute VSI (R640, 1.5TB RAM, 2x 240GB SSD)
  • Dell Performance (R640, 96GB RAM, 32GB NVDIMM, 8x 1.92TB SSD, 2x 240GB SSD)
  • Dell Capacity (R740xd, 96GB RAM, 96GB NVDIMM, 24x 3.84TB SSD, 2x 240GB SSD)
  • HPE Compute VDI (DL360, 768GB RAM, 2x 240GB SSD)
  • HPE Compute VSI (DL360, 1.5TB RAM, 2x 240GB SSD)
  • HPE Performance (DL360, 96GB RAM, 32GB NVDIMM, 8x 1.92TB SSD, 2x 240GB SSD)
  • HPE Capacity (DL380, 96GB RAM, 96GB NVDIMM, 24x 3.84TB SSD, 2x 240GB SSD)

100% CPU Load (Turbo Boost Disabled)

  • Dell Compute VDI (R640, 768GB RAM, 2x 240GB SSD)
  • Dell Compute VSI (R640, 1.5TB RAM, 2x 240GB SSD)
  • Dell Performance (R640, 96GB RAM, 32GB NVDIMM, 8x 1.92TB SSD, 2x 240GB SSD)
  • Dell Capacity (R740xd, 96GB RAM, 96GB NVDIMM, 24x 3.84TB SSD, 2x 240GB SSD)
  • HPE Compute VDI (DL360, 768GB RAM, 2x 240GB SSD)
  • HPE Compute VSI (DL360, 1.5TB RAM, 2x 240GB SSD)
  • HPE Performance (DL360, 96GB RAM, 32GB NVDIMM, 8x 1.92TB SSD, 2x 240GB SSD)
  • HPE Capacity (DL380, 96GB RAM, 96GB NVDIMM, 24x 3.84TB SSD, 2x 240GB SSD)

Detailed Hardware Table

The table below goes through all the hardware that was installed and tested. During these tests there were four different models tested as defined by the customers standard for Compute VSI, Compute VDI, Performance and Capacity nodes.

 Test Tools

BurnInTEST Pro Software - Testing tool used to generate workload of RAM, CPU, and Disk.

Solarwinds - Used to record results from Emerson PDU's 

iDRAC - Dell Lights out Management Utility that allows kvm access to server, as well as some power and performance charts from the server. (Exert of power chart from iDRAC)

iLO - HPE Lights out Management Utility that allows kvm access to server, as well as some power and performance charts from the server. (Exert of power chart from iLO below)

 

Technologies