I have the pleasure of configuring some Dell kit without the use of a pre-execution environment. This presents quite a challenge as many of the management tools are designed to run within such an environment or an installed operating system which means that my options for configuring these serves is somewhat limited. Thankfully for most of the critical stuff Dell’s RACADM tool is more than capable of managing the server remotely however it unfortunately doesn’t have any access to the system BIOS where some critical changes need to be made. Thus I was in need of finding a solution to this problem and it seems that my saviour comes in the form of a protocol called Web Services Management (WSMAN).
WSMAN is an open protocol for server management which provides a rather feature rich interface to your hardware for getting, setting and enumerating the various features and settings on your hardware. Of course since its so powerful it’s also rather complex in nature and you won’t really be able to stumble your way through it without the help of a vendor specific guide. For Dell servers the appropriate guide is the Lifecycle Controller Web Services Interface Guide (there’s an equivalent available for Linux) which gives you a breakdown of the commands that are available and what they can accomplish.
They’re not fully documented however so I thought I’d show you a couple commands I’ve used in order to configure some BIOS settings on one of the M910 blades I’m currently working on. The first requirement was to disable all the on board NICs as we want to use the Qlogic QME8262-k 10GB NICs instead. In order to do this however we first need to get some information out of the WSMAN interface in order to know which variables to change. The first command you’ll want to run is the following:
winrm e http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/root/dcim/DCIM_BIOSEnumeration -u:root -p:calvin -r:https://[iDRACIP]/wsman -SkipCNcheck -SkipCAcheck -encoding:utf-8 -a:basic
Which will give you a whole bunch of output along these lines:
DCIM_BIOSEnumeration AttributeName = EmbNic1Nic2 Caption CurrentValue = Enabled DefaultValue Description ElementName FQDD = BIOS.Setup.1-1 InstanceID = BIOS.Setup.1-1:EmbNic1Nic2 IsOrderedList IsReadOnly = FALSE PendingValue PossibleValues = Disabled, Enabled DCIM_BIOSEnumeration AttributeName = EmbNic1 Caption CurrentValue = EnabledPxe DefaultValue Description ElementName FQDD = BIOS.Setup.1-1 InstanceID = BIOS.Setup.1-1:EmbNic1 IsOrderedList IsReadOnly = FALSE PendingValue PossibleValues = Disabled, EnablediScsi, EnabledPxe, Enabled
Of note in the output are the AttributeName and PossibleValues variables. In essence these represent the current and possible states of the BIOS variables and all of them can be modified through the appropriate WSMAN command. The Dell guide I referenced earlier though doesn’t exactly tell you how to do this and the only example that appears to be close is one for modifying the BIOS boot mode setting. However as it turns out this same command can be used to modify any variable that is output by the previous command so long as you create the appropriate XML file. Shown below is the command and XML file to disable the first 2 embedded NICs:
Code: winrm i SetAttribute http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/root/dcim/DCIM_BIOSService?SystemCreationClassName=DCIM_ComputerSystem +CreationClassName=DCIM_BIOSService +SystemName=DCIM:ComputerSystem+Name=DCIM:BIOSService -u:root -p:calvin -r:https://[iDRACIP]/wsman -SkipCNcheck -SkipCAcheck -encoding:utf-8 -a:basic -file:SetAttribute_BIOS.xml SetAttribute_BIOS.xml: <p:SetAttribute_INPUT xmlns:p="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/root/dcim/DCIM_BIOSService"> <p:Target>BIOS.Setup.1-1</p:Target> <p:AttributeName>EmbNic3Nic4</p:AttributeName> <p:AttributeValue>Disabled</p:AttributeValue> </p:SetAttribute_INPUT>
This appears to work quite well for individual attributes but I’ve encountered errors when trying to set more than one BIOS variable at a time. This could easily be due to me fat fingering the input file (I didn’t really check it before troubleshooting it further) but it could also be a limitation of the WSMAN implementation on the Dell servers. Either way once you’ve run that command you’ll notice the response from the server states that the values are pending and the server requires a reboot. Now I’m not 100% sure if you can get away with just rebooting it through the iDRAC or physically rebooting it but there is a WSMAN command which I can guarantee will apply the setting whilst also rebooting the server for you. Again this one relies on an XML file for it to succeed:
Code: winrm i CreateTargetedConfigJob http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/root/dcim/DCIM_BIOSService?SystemCreationClassName=DCIM_ComputerSystem +CreationClassName=DCIM_BIOSService +SystemName=DCIM:ComputerSystem +Name=DCIM:BIOSService -u:root -p:calvin -r:https://[iDRACIP]/wsman -SkipCNcheck -SkipCAcheck -encoding:utf-8 -a:basic -file:CreateTargetedConfigJob_BIOS.xml CreateTargetedConfigJob_BIOS.xml: <p:CreateTargetedConfigJob_INPUT xmlns:p="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/root/dcim/DCIM_BIOSService"> <p:Target>BIOS.Setup.1-1</p:Target> <p:RebootJobType>2</p:RebootJobType> <p:ScheduledStartTime>TIME_NOW</p:ScheduledStartTime> <p:UntilTime>20131111111111</p:UntilTime> </p:CreateTargetedConfigJob_INPUT>
Upon executing this command the server will reboot and then load into the Lifecycle Controller where it will apply the desired settings. After which it will reboot again and you’ll be able to view the settings inside the BIOS proper. It appears that this command can be used for any variable that appears within the initial BIOS enumeration so using this it is quite possible to fully configure the BIOS remotely. You can also access quite a lot of things within the iDRAC itself however I’ve found that RACADM is a much easier way to go about this, especially if you simply dump the entire config, edit it, then reupload it. Still the option is there if you want to use the single tool but unless you’re something of a masochist I wouldn’t recommend doing everything through WSMAN.
All that being said however the WSMAN API appears to cover pretty much everything in the server so if you need to do something remotely to it (hardware wise) and you don’t have the luxury of a PXE or installed operating system than its definitely something to look into. Hopefully the above commands will get you started and then the rest of the Dell integration guide will make a little more sense. If you’ve got any questions about a particular command hit me up in the comments, on Twitter or on my Facebook fan page and I’ll help you out as much as I can.
Back when virtualization was just starting to make headway into the corporate IT market the main aim of the game was consolidation. Vast quantities of CPU, memory and disk resources were being squandered as servers sat idle for the vast majority of their lives, barely ever using the capacity that was assigned to them. Virtualization allowed IT shops the ability to run many low resource servers on the one box, significantly reducing the hardware requirement cost whilst providing a whole host of other features. It followed then that administrators looked towards over-provisioning their hosts, I.E. creating more virtual machines than the host was technically capable of handling.
The reason this works is because of a feature of virtualization platforms called scheduling. In essence when you put a virtual machine on an over-provisioned host it will not be guaranteed to get resources when it needs them, instead it’s scheduled on and in order to keep it and all the other virtual machines running properly. Surprisingly this works quite well as for the most part virtual machines spend a good part of their life idle and the virtualization platform uses this information to schedule busy machines ahead of idle ones. Recently I was approached to find out what the limits were of a new piece of hardware that we had procured and I’ve discovered some rather interesting results.
The piece of kit in question is a Dell M610x blade server with the accompanying chassis and interconnects. The specifications we got were pretty good being a dual processor arrangement (2 x Intel Xeon X5660) with 96GB of memory. What we were trying to find out was what kind of guidelines should we have around how many virtual machines could comfortably run on such hardware before performance started to degrade. There was no such testing done with previous hardware so I was working in the dark on this one, so I’ve devised my own test methodology in order to figure out the upper limits of over-provisioning in a virtual world.
The primary performance bottleneck for any virtual environment is the disk subsystem. You can have the fastest CPUs and oodles of RAM and still get torn down by slow disk. However most virtual hosts will use some form of shared storage so testing that is out of the equation. The two primary resources we’re left with then are CPU and memory and the latter is already a well known problem space. However I wasn’t able to find any good articles on CPU over-provisioning so I devised some simple tests to see how the systems would perform when under a load that was well above its capabilities.
The first test was a simple baseline, since the server has 12 available physical cores (HyperThreading might say you get another core, but that’s a pipe dream) I created 12 virtual machines each with a single core. I then fully loaded the CPUs to max capacity. Shown below is a stacked graph of each virtual machine’s ready time which is a representation of how long the virtual machine was ready¹ to execute some instruction but was not able to get scheduled onto the CPU.
The initial part of this graph shows the machines all at idle. Now you’d think at that stage that their ready times would be zero since there’s no load on the server. However since VMware’s hypervisor knows when a virtual machine is idle it won’t schedule it on as often as the idle loops are simply wasted CPU cycles. The jumpy period after that is when I was starting up a couple virtual machines at a time and as you can see those virtual machine’s ready times drop to 0. The very last part of the graph shows the ready time rocketing down to nothing for all the virtual machines with the top grey part of the graph being the ready time of the hypervisor itself.
This test doesn’t show anything revolutionary as this is pretty much the expected behaviour of a virtualized system. It does however provide us with a solid baseline from which we can draw some conclusions from further tests. The next test I performed was to see what would happen when I doubled the work load on the server, increasing the virtual core count from 12 to a whopping 24.
For comparison’s sake the first graph’s peak is equivalent to the first peak of the second graph. What this shows is that when the CPU is oversubscribed by 100% the CPU wait times rocket through the roof with the virtual machines waiting up to 10 seconds in some cases to get scheduled back onto the CPU. The average was somewhere around half a second which for most applications is an unacceptable amount of time. Just imagine trying to use your desktop and having it freeze for half a second every 20 seconds or so, you’d say it was unusable. Taking this into consideration we now know that there must be some level of happy medium in the centre. The next test then aimed right bang in the middle of these two extremes, putting 18 CPUs on a 12 core host.
Here’s where it gets interesting. The graph depicts the same test running over the entire time but as you can see there are very distinct sections depicting what I call different modes of operation. The lower end of the graph shows a time when the scheduler is hitting bang on its scheduling and the wait times are overall quite low. The second is when the scheduler gives much more priority to the virtual machines that are thrashing their cores and the machines that aren’t doing anything get pushed to the side. However in both instances the 18 cores running are able to get the serviced in a maximum of 20 milliseconds or so, well within the acceptable range of most programs and user experience guidelines.
Taking this all into consideration it’s then reasonable to say that the maximum you can oversubscribe a virtual host in regards to CPU is 1.5 times the number of physical cores. You can extrapolate that further by taking into consideration the average load and if it’s below 100% constantly then you can divide the number of CPUs by that percentage. For example if the average load of these virtual machines was 50% then theoretically you could support 36 single core virtual machines on this particular host. Of course once you get into the very high CPU count things like overhead start to come into consideration, but as a hard and fast rule it works quite well.
If I’m honest I was quite surprised with these results as I thought once I put a single extra thrashing virtual machine on the server it’d fall over in a screaming heap with the additional load. It seems though that VMware’s scheduler is smart enough to be able to service a load much higher than what the server should be capable of without affecting the other virtual machines that adversely. This is especially good news for virtual desktop deployments as typically the limiting factor there was the number of CPU cores available. If you’re an administrator of a virtual deployment I hope you found this informative and it will help you when planning future virtual deployments.
¹CPU ready time was chosen as the metric as it most aptly showcases a server’s ability to serve a virtual machine’s request of the CPU when in a heavy scheduling scenario. Usage wouldn’t be an accurate metric to use since for all these tests the blade was 100% utilized no matter the number of virtual machines running.