High density virtualization using the IBM System x3850 X5

384 virtual machines on a single IBM® System x3850 X5 server with 4 Intel® Xeon® 8-core processors, 512 GB memory, and Windows® Server 2008 R2 Hyper-V

The IBM System x3850 X5 server is the fifth generation of the Enterprise X-Architecture, delivering innovation with enhanced reliability and availability features to enable optimal performance for databases, enterprise applications, and virtualized environments. Benefits of IBM eX5 servers include:

• Increased performance with the next generation Intel Xeon processors
• Memory reliability and availability
• Higher performing databases and faster time to value for database workloads
• Up to eight sockets and 128 DIMMs in a two-chassis configuration for larger databases, virtualization, enterprise, and mission-critical workloads

The IBM System x3850 X5 is shown in Figure 1.


Figure 1. The IBM System x3850 X5

Hyper-V in Microsoft® Windows Server 2008 R2 can now support 64 logical processors, a significant increase from the 24 logical processors in the first release of Hyper-V. Each virtual machine can be configured with up to 4 virtual processors and 64 GB of memory per virtual machine. Live migration and Clustered Shared Volumes (CSV) features are now supported with this new release of Hyper-V.

More processors and memory configurable on the x3850 X5 opens the door for achieving unparalleled VM density on one host server. You can create and run 384 virtual machines on a single x3850 X5 server with 4 Intel Xeon 8-core processors and 512 GB memory made up of 64 8GB DIMMs.

Through virtualizing workloads with Microsoft Hyper-V technology on IBM System x3850 X5, businesses can make the best use of server hardware investments by consolidating multiple server roles as separate virtual machines running on a single physical machine. Running the same hypervisor with fewer physical servers allows your IT administrators to manage more servers than without virtualization. Managing applications within the virtual machines results in less downtime per physical server and increases IT flexibility due to streamlining deployment and upgrade processes in both physical and virtual machines.

In addition, businesses can enjoy a lower total cost of ownership (TCO) from reduced hardware requirements achieved by maximizing the resource utilization of each server purchased. Reducing the number of physical servers required to support operations lowers capital acquisition, power consumption, and cooling costs. Infrastructure optimization also simplifies and standardizes IT administration, helping to control labor costs. Reduced downtime costs are realized by the ability to easily move and balance workloads across various resources, improve operational agility, and provide flexibility in managing maintenance schedules and responding to today’s dynamic IT demands.

Hyper-V runs in 64-bit mode only as a role in Windows Server 2008 R2, or as a stand-alone, console-based hypervisor in the Microsoft Hyper-V Server 2008 R2 product (available for download from http://www.microsoft.com/hvs). Virtual machines can support up to four virtual processors and 64 GB of memory, depending on the operating system loaded. Each virtual machine will have its own operating system instance and will be completely isolated from the host operating system as well as from other virtual machines. High availability is an option when running the Enterprise or Datacenter Edition of Windows Server 2008 R2 or Microsoft Hyper-V Server 2008 R2 with Microsoft Clustering installed. Hyper-V is not supported on Intel Itanium processor-based servers.

Complementing the solution, IBM System Storage provides the high performing, scalable, reliable, available, and flexible midrange storage systems needed to support Microsoft Hyper-V information infrastructure. With the IBM System Storage DS5000, you can enhance the improved efficiency and reduced total cost of ownership in your server infrastructure.

As IBM’s most powerful midrange storage system, the DS5000 is well-suited for a virtualized environment that can keep pace with your business growth. Organizations can buy only the capacity needed initially, and can then dynamically upgrade and reconfigure additional capacity and features later to meet changing business requirements, all without any system downtime. The DS5000 provides flexibility intermixing both Fibre Channel and SATA disks for midrange storage systems. The DS5000 delivers class-leading performance and is well suited for consolidation and virtualization.

 

The advanced features incorporated throughout the IBM X-Architecture portfolio help IBM System x servers and IBM BladeCenter chassis and blades deliver the reliability and high availability necessary to virtualize a wide range of applications on Hyper-V:

• IBM System x3850 X5 enterprise servers offer up to 64 logical processors, up to 1 TB of memory with 64 16GB DIMMs, 7 Gen2 PCI-E slots, and up to 8 internal HDDs.
• Powered by next generation Intel Xeon processors with up to eight cores.
• Highly available memory with IBM Memory ProteXion, Chipkill, and memory mirroring.
• Superior memory per processor capacity supports the needs of virtualization servers in 2010 and beyond.
• Superior memory technology helps to lower cost, lower latency, increase bandwidth, and lower power consumption with MAX5 and the Advanced Buffer eXecution chip.

To test the maximum supported limit of 384 VMs by Microsoft Hyper-V, we set up an IBM System x3850 X5 server with the following components:

• Four Intel Xeon 8-core processors
• 64 8GB Samsung DIMMs for a total of 512GB memory
• Two 146GB HDDs in RAID-1 for the OS
• One Emulex LP12002 8Gb dual-port HBA
• One Broadcom 5709 Dual-Port NIC
• One Intel PRO/1000 PT Dual-port NIC
• One DS5300 dual storage controller with 4GB battery-backed cache
• Four EXP810s with 16 300GB HDDs each for a total of 64 disk drives to host the VMs

Virtual machine hardware and software profiles were created using System Center Virtual Machine Manager (SCVMM) 2008 R2. This tool allows the administrator to set a predefined memory size, network assignments, processors, and hard disk in the hardware profile, and Active Directory membership, default password, and license key in the software profile. These profiles were paired with a sysprep’d virtual machine stored in the library to create the 384 virtual machines.

Tip: If you are planning to create more than 255 VMs on a server, the default MAC address range must be increased for the source server, using the Virtual Network Manager settings in Hyper-V.

The virtual machines were placed on the attached fault-tolerant DS5000 arrays. Sizings should take into account not only the raw size of the virtual machines, but also the aggregate I/O requirements expected from the sum of the virtual machines. The greater the disk I/O requirements the more disk spindles need to be committed to the arrays. In addition, ensure sufficient network I/O with additional NIC(s) for your host server as needed.

384 virtual machines were configured, each with the following:

• One Virtual Processor
• 1 GB of memory
• 12 GB Operating System virtual hard disk
• One Virtual Network Port

For this technology demonstration, there was no specific mission-critical workload being run on the virtual machines and therefore memory, processor, and I/O utilization were fairly low. The memory and I/O capacity of the x3850 X5 server was not maximized and could be increased if needed. This proof of concept configuration represents a consolidation with virtualization deployment scenario. This tested configuration relied on the redundancy and High Availability (HA) infrastructure built into the server and storage. The test confirmed the supported limit of 384 Hyper-V virtual machines on a single host server and demonstrated the capabilities of IBM X5 technology and the scalability of Microsoft virtualization with Windows Server 2008 R2 Hyper-V.

Figure 2 below shows a snapshot of the single x3850 X5 host running 384 virtual machines.


Figure 2. VMM overview with 384 virtual machines running

The IBM System x3850 X5 can easily support even the maximum supported configuration using Microsoft Windows Server 2008 R2 with Hyper-V virtualization and is well-suited to consolidate underutilized servers in an easy-to-manage configuration. By providing an environment with a reduced number of physical machines to manage, the solution greatly improvesTCO with reduced space, power, and cooling needs.

The IBM System x3850 X5 server is a high performing, reliable server that continues to set the pace in the x86 market. This server drives the industry to new levels of virtualization densities, availability, flexible scalability, and transaction performance. IBM System x3850 X5 servers maximize memory, minimize cost, and simplify development. Fifth-generation Enterprise X-Architecture technology with the latest Intel Xeon Processors delivers affordable pricing and leadership performance without compromising high availability or rack density. Coupled with high-performing and flexible IBM System Storage DS5000 systems, the x3850 X5 server and Microsoft Hyper-V solution can help optimize your IT environment.

To learn more about virtualization using Microsoft Hyper-V on the IBM System x platform, please visit:

http://ibm.com/systems/x/solutions/infrastructure/virtualization

To learn more about IBM storage solutions, see the paper IBM System Storage Solutions for Microsoft Hyper-V, available from:

ftp://ftp.software.ibm.com/common/ssi/pm/sp/n/tss03028usen/TSS03028USEN.PDF

For more information about IBM eX5 and the IBM System x3850 X5, please visit:

http://ibm.com/systems/ex5

Don Clemmer is an IBM System x systems engineer. For over 12 years, Don has worked at the IBM Center for Microsoft Technologies with IBM customers and partners in designing Windows- and virtualization-based solutions to meet their business needs. He is currently based at Microsoft's Redmond campus working on optimizing the performance of IBM's System x servers running Microsoft Windows Server operating systems environments.

Scott Smith is an IBM System x systems engineer working at the IBM Center for Microsoft Technology. Over the past 15 years, Scott has worked to optimize the performance of IBM x86-based servers running the Microsoft Windows Server operating system and Microsoft application software. He has extensive experience in helping IBM customers to understand the issues they are facing and developing solutions that address them.

Vinay Kulkarni is an IBM System x performance engineer working on-site at the Microsoft Redmond campus. He has been working with Microsoft over the past 8 years to optimize performance of System x servers running Microsoft Windows and SQL Server software. He also works with IBM clients to tune performance of System x and storage environments and supports publication of industry-leading benchmarks alongside System x benchmark teams.