WWT Datacenter Services Team Blog

Our Team's Ramblings about Server, Desktop, and Application Virtualization, Cloud Computing, and other fun stuff.

XenDesktop and a Web Proxy.

1 Comment Posted by on August 3, 2010

A FYI to anyone out there that ever comes across this combo…

As part of the XenDesktop Desktop Pool creation process you have to tell it where/what the hosting infrastructure is.  In my customers case it was VMware.  I entered the URL of the vCenter host, and it failed.  Checked the proxy.xml file, still failed.

Turns out if there is an internal web proxy on the customers network that requires authentication (that you probably won’t know about yet) it will not work.  In order to track down what is going on, enable logging for the Pool Management Service.

Edit the appSettings section of the CdsPoolMgr.exe.config file.  It is located in Program Files\Citrix\Vm Management on a 32bit server or in Program Files(x86)\Citrix\Vm Management on a 64bit server.

<appSettings>
    <add key=”LogToCDF” value =”1″/>
    <add key=”LogFileName” value =”c:\<choose a folder that exists>\<choose a name>.log”/>
    <add key=”OverwriteLogFile” value =”1″/>
</appSettings>

Bounce the pool management service now, and try the pool creation wizard again.  Now digging through the logs you will see:

“Error: (407) Proxy Authentication Required”

To resolve this edit \Program Files\Citrix\VMManagement\CdsPoolMgr.exe.config on a 32bit server or in Program Files(x86)\Citrix\Vm Management on a 64bit server and put this in the middle of the configuration section:

 <system.net>
    <defaultProxy>
         <proxy usesystemdefault=”False” bypassonlocal=”True” autoDetect=”False”/>
    </defaultProxy>
</system.net>

I put it between the <runtime> and <appSettings> section.  And now, that will stop this .net app from using the auto-detected proxy servers.  (It is case-sensitive.) After you have confirmed it will successfully work, go back and disable logging.

This will also affect the XenDesktop Setup Wizard.  You can find the file to modify on the server that you installed the XD Setup Wizard on.  (Typically the PVS host is where I put it.)

Edit \program files\citrix\xendesktop setup wizard\setuptoolapplication.exe.config

Throw the same blurb from above in the middle of the config section again.  Restart the app. 

Hope this helps you save some time!

~Jason

Citrix, vCenter, WWT, XenDesktop , ,

UCS WWN and MAC Pools

10 Comments Posted by on July 22, 2010

This past week, I’ve been working deploying a vBlock1.  In case you weren’t aware, the vBlock solutions are basically reference architecture’s consisting of EMC storage, Cisco UCS blade servers, and VMware.  They come in three different flavors (vBlock0, vBlock1, and vBlock2) each with different models of SAN’s and quantity of UCS chassis/blades.  Specifically, this vBlock1 solution being deployed consisted of an EMC CX4-480 array, two UCS blade chassis with eight full width blades, and obviously VMware, which would be installed on to each blade server.

During design discussions with the client, one thing discussed quite a bit consisted of how to ensure uniqueness of systems and resources residing in this particular vBlock.  Specifically here, I’m referring to WWPN/WWNN and MAC Addresses.

The UCS system allows the creation of Pools that will automatically assign WWPN/WWNN and MAC Addresses to servers as service profiles are applied to them.  Just as with any server, we need to ensure these values are unique within the environment.  Because portions of each of these values are adjustable by UCS Administrators, it’s important to put in a standard practice for determining what these values are to be.  Further, with a standard in place these values can be easily used by Network and Storage Administrators to determine what type of system is accessing what within their world.

Digging some for best practices surrounding this and through personal experience, the below is how we determined what these pools would consist of.

First – WWPN/WWNN:

The IEEE standard for WWPN/WWNN’s dictates a value of eight octets.  By default if we look at what UCS Manager automatically uses we’ll see – 20:00:00:25:B5:00:00:00. When creating a pool you would change portions of this value (in order to make it unique) and then determine how many WWPN/WWNN’s you require for your environment.  This ultimately creates a block of WWPN/WWNN’s.  As service profiles are associated to blade servers, WWPN/WWNN’s are derived from this block and assigned to servers.  So how can we ensure UCS clusters are unique within a given environment housing multiple clusters?  Looking at the below helps break out that eight octet value to something we can work with and better understand.

Breaking this all down we can say:

  • A – Denotes this is an IEEE standard format.  This means we must leave the value of 2 intact and unchanged.
  • B:CD – Are vendor specific so we can put what we like here.  For now, my suggestion is leave B unchanged.  More on the CD values in a minute
  • EF:GH:IJ – These 24 bits are the OUI.  This should be the hardware vendor assigned OUI.  Because of this, best practice is to not touch these values.
  • KL:MN:OP – The last 24 bits are fair game to change as needed.

The default UCS entry put in the above picture would look like this:

So how do we come up with a standard unique pattern?  As mentioned already, there are certain bits that must remain untouched so lets look at those we can.  The second octet – CD, can be utilized to denote the UCS domain (or cluster).  If this is the first UCS cluster in the environment, I suggest making this 01 then increasing the count as new clusters are deployed (i.e. the second cluster would be 02, third 03, and so on).  For the last 24 bits, there are no limits as to what we can use here so wouldn’t it be useful to have something that helped Storage Administrators know what it was checking into their environment.  Some proposed values may look like:

  • KL = Identifies this as a WWPN or WWNN.
    • FF = Denotes this is a WWNN
    • 0A = Denotes an WWPN on Fabric A
    • 0B = Denotes an WWPN on Fabric B
    • MN:OP = Sequential numbering handled automatically by UCS Manager

Putting this to practice I created the following for the first UCS cluster:

WWNN Pool = 20:01:00:25:B5:FF:00:01

WWPN Pool Fabric A = 20:01:00:25:B5:0A:00:01

WWPN Pool Fabric B = 20:01:00:25:B5:0B:00:01

Now let’s look at the MAC Address Pools.  This pool is not as detailed or cryptic as the WWPN/WWNN’s so we’ll get through this quickly.

MAC Addresses are derived from a total of six octets.  Like WWPN/WWNN’s certain values cannot be changed.  Let’s break each octet out again to better understand.

Breaking this down we can say:

  • AB:CD:EF – Like the WWPN/WWNN these first 24 bits are specific to the hardware vendor OUI.  We should leave these set and fortunately, UCS Manager will not allow you to change these.
  • GH:IJ:KL – The last 24 bits can be changed as needed.

The default UCS entry put in the above picture would look like the below:

***UPDATE BEGIN***

I had some great input from a few people (@stevie_chambers, @jonisick, and @jeremywaldrop) on the original post on how best to handle these pools.  Me being an old server guy, I thought being able to tell what OS was running on the blade from it’s MAC would be helpful.  In addition, the networking gents made a suggestion that including the fabric ID would also be helpful.  Something I very much agree with here as should you start to see MAC’s from an A fabric running over the B fabric you would know something is wrong.

With the above in mind, to make MAC addresses unique, I’d propose the following breakdown for the last 24 bits:

  • GH = UCS Domain
  • IJ = Denote what host OS is to be installed onto the blade along with the specific fabric ID.
    • 0A = VMware Fabric A
    • 1A = Linux Fabric A
    • 2A = Windows Fabric A
    • 3A = Solaris Fabric A
    • 0B = VMware Fabric B
    • 1B = Linux Fabric B
    • 2B = Windows Fabric B
    • 3B = Solaris Fabric B
  • KL = Sequential number handled automatically by UCS Manager.

Putting this to practice, I created the following pools:

ESX MAC Address Pool Fabric A = 00:25:B5:01:0A:01

ESX MAC Address Pool Fabric B = 00:25:B5:01:0B:01

Windows MAC Address Pool Fabric A = 00:25:B5:01:2A:01

Windows MAC Address Pool Fabric B = 00:25:B5:01:2B:01

***UPDATE ENDS***

Some final thoughts and proposed best practices pertaining to both WWNN/WWPN and MAC Addresses pools would be:

  • Create WWN pools in block sizes of 16.  This is a best practice as you can easily determine which derived WWPN/WWNN will get assigned to a specific blade.
  • Match WWPN with WWNN blocks.  This ensures when both are assigned to a server profile, they line up with one another and can be easily distinguished.
  • Only create block sizes large enough for your environment.  Additional blocks can be created as required.
  • Remember if you’re utilizing the Palo card, you’ll need to figure out how many dynamic NIC’s are to be allocated to each server.  Each vNIC will have an assigned MAC Address.

Like most things, I don’t think there is one way to do this.  The above is just something that made sense to me and will ensure uniqueness across any given environment (big or small).

If you have a scheme you have used in the past, I’d love to hear how you handled things…

Cisco, Cloud, EMC, ESX, FCoE, Networking, Storage, VMware, vSphere , , , , , , , ,

Collapsing Server Management Points with UCS

1 Comment Posted by on June 22, 2010

Management of the data center server architectures is a major pain point for many data centers and becomes exponentially worse the larger the infrastructure scales. While virtualization can be used to solve several infrastructure pain points it can also further exacerbate the management issues by adding another layer of server and network management. The network management can be an even greater challenge because instead of being owned solely by the network team as it is in a physical environment it becomes shared by the network and server/virtualization teams.

The challenge of managing both physical and virtual server environments is one of the things Cisco’s Unified Computing System looks to address. Within the UCS architecture all hardware management of the server architecture and its access layer network connections are handled by one management tool called the UCS Manager. This management tool allows for seamless repeatable deployment processes for deploying new applications on blades and rack-mount servers. While this greatly reduces the management of the physical infrastructure it leaves a separate point of management for the virtual environment and network.

image

Cisco’s solution to this is Pass-Through Switching (PTS) which is currently available at no additional cost within the UCS system. Conceptually PTS pulls the management of the virtual network up into the UCS Manager allowing the network infrastructure to be designed and managed in the same fashion for both virtual and physical networks. Additionally it consistently enforces policy for both virtual and physical machine traffic in UCS switching hardware.

PTS is accomplished using the Cisco Virtual Interface Card (VIC) in the UCS blade. The VIC is a Dual 10GE/FCoE PCIe device that supports the creation of up to 128 virtual interfaces which are then presented to the Operating System (OS) as physical cards. Each of these interfaces can then be used in the same fashion as a physical Ethernet or Fibre Channel port.

image

When this card is utilized along with PTS each Virtual Machine (VM) is given one or more Virtual Interfaces (VIF) which are then mapped across a VMware Virtual Distributed Switch (vDS) on a 1-to-1 basis. Each vNIC in each VM is then recognized as an individual switch port within the UCS switching system and individually managed/switched in the same way as physical severs.

image

Summary:

Pass-Through Switching provides an additional reduction in management points and reduces misconfiguration through consistent policy based management for the physical and virtual network within UCS.

WWT

Data Center Summit

2 Comments Posted by on June 18, 2010

I recently spent two days in San Jose at Cisco’s Data Center Summit.  Filled with lots of Cloud discussions, the point of the conference was to receive feedback from select partners around the US on future product roadmaps and where we saw technologies going from a data center perspective.

One thing that was talked plenty about was getting to the Cloud, what it might look like, necessary security measures, and more.  Hardware components like EMC’s vBlock and NetApp’s SMT (Secure Multiple Tenancy) was obviously part of these discussions.  The best way to describe both of these solutions at a very high level are, packaged reference architectures for purchase.  They each consist of Cisco UCS, VMware, and some sort of storage (obviously EMC for vBlock and NetApp for SMT).

Our last speaker centered in on SMT explaining upcoming features and the architecture (sorry NDA’s prevent me from going into details here).  One thing that stood out to me in this session was the following…In this session our speaker asked how many of us had sold vBlock.  Lots of hands raised.  Then he asked how many had sold SMT.  Zero hands went up.  How many talk to your customers about SMT, he asked?  One hand went up.  I have to admit that where I knew SMT was NetApp’s competing product to EMC’s vBlock I really haven’t heard a whole lot about it.  You don’t see much information from Cisco or NetApp or public blogs about it.  Further through the presentation I started wondering why this might be the case.

As we move towards a Cloud like or enabling architecture, we have to look at the key components in which everything will run.  Both solutions from EMC and NetApp are solid products however I think a key differential to both is the elasticity SMT provides.  vBlock is sold in three different options at this time, each with very limited customization.  If you want to build onto it at a later date you must purchase another vBlock.  This is the “pod” architecture that so many discuss these days.  SMT is not this at all and allows quite a bit of customization.  Now there are obviously technical differences between both offerings but I wanted to focus at the physical layer here.  With Cloud, I think it’s imperative to have a flexible infrastructure – one that can be easily scaled when needed.  Having “islands” or “pods” of hardware won’t allow us to scale very efficiently.

Getting back to the interesting questions the presenter asked, I started inquiring with various attendees as to why might this be – the fact that there isn’t a whole lot of talk or sales of SMT.  The common theme seemed to center around marketing.  Where some of this responsibility lies in Cisco’s hands, I think majority of it lies with in NetApp’s.  It’s very clear that they are not marketing the product offering and I find this amazing given the offering they have.  Clearly without much effort, EMC will own this market.

Having options and choices when purchasing product is a good thing.  Not knowing about other competing options is a bad thing.  In addition, this competitive nature builds feature sets into each solution strengthening it as each vendor tries to out do the other.  My hope is that NetApp gets off the bench and starts marketing the SMT solution.  As I indicated earlier, both solutions are solid products however there are differences in each, which I think is important to understand when considering either solution.

Cisco, Cloud, EMC, Netapp, Storage, VMware , , , , , , , ,

Using HP SAN Virtualization Services Platform (SVSP) in a vSphere stretched cluster

7 Comments Posted by on April 26, 2010

Title:

Using HP SAN Virtualization Services Platform (SVSP) in a vSphere stretched cluster

Recently I was involved in evaluating the integration of HP’s StorageWorks SVSP solution with a vSphere stretched cluster design.

Hewlett Packard’s SVSP is a HP’s product that I classify as a “storage enabler”. Storage enablers add a layer between one or more physical SANs and the hosts accessing LUNs.  This layer adds features that do not exist with the existing SANs.    An example of this layer includes aggregating storage between dissimilar SANs, LUN replication, LUN cloning, and sync mirroring.

This particular customer is using vSphere’s HA feature as a DR solution. Their plan was to use vSphere’s HA feature in conjunction with geographically dispersed storage and the sync-mirror feature of SVSP to achieve high availability automation in the case of one of the sites hosting half of the cluster goes down.

The sync mirror feature takes an active LUN and synchronizes the data at the block level to a passive mirror of that LUN.

In this specific case, the customer had redundant dark fiber connections between sites with two XP24000s, one at each site.


After evaluating this proposed configuration, I found some significant problems with this approach.   Would it provide a DR solution?  Yes, but I feel a more appropriate question is… Will it provide a reliable and stable DR solution at a reasonable cost??

As I worked with the customer and did my own research I found there is no evidence that VMware officially supports stretched clusters and because of this, the actual supportability for them is questionable.  You do not want to have to call VMware for support and be surprised by their demand that you redesign your cluster to start getting support.

It is only recently that major players such as NetApp, EMC, and Cisco have been looking at how geographically dispersed datacenters can be configured to support core functional features of vSphere like vMotion.  The efforts show that this functionality is being looked at and tested.   I believe full support for this type of DR solution is coming however we are not quite there yet and with today’s high speed WAN/Internet links available, I see this as becoming an increased topic of discussion.

While working on this project, I performed a series of tests in order to document findings, determine actual DR success, and capture specific data which would be used at a later date to recommend future solutions.

To outline some of my findings and to provide an example of some of the tests I performed I’ve shared some info below:

Test 1 – Failure testing - vSphere HA functionality was tested and verified to work properly with local and remote storage during tests of hard shutdowns of ESX hosts.  Breaking the Fiber Channel ISL link between sites A & B for an extended period of time broke any VMs whose storage is hosted remotely.  It was noticed that the VMs were able to recover from storage disconnects less than 2 minutes, but anything longer than that and the VMs died a slow death.   Even at the less than 2 minute mark, this means that data corruption could still occur to a production box trying to read/write data to the SVSP LUN.  As of this writing, vSphere does not have any method to recognize that a VM has lost access to storage and compensate for it.   We can’t utilize multi-pathing in this capacity due to the separate sites (i.e. it’s not active/active storage).

Test 1 Findings – I recommend against hosting storage remotely.  Have active storage located at each site and each VM accessing its storage locally. This can be guaranteed with a split cluster design or turn off DRS.

Test 2 – Performance testing – When doing the testing we monitored “Total I/O per Second” & “Average I/O Response time (ms)”.  When the SVSP sync mirror link is broken, the active SVSP LUN has to have changes journaled and once communication is re-established with the passive LUN, it has to re-sync.  I/O performance drops for all VMs with VMDKs residing on the active SVSP LUNs during re-sync of sync-mirror LUN.  I noticed that during the re-sync process VMs that had their storage hosted by the active site had a performance drop of 8% and those hosted remotely of 20%.

Let’s look deeper though as to what this means from a VMware design aspect.

Problem #1 – Limits on cluster size and HA functionality at risk.

HA functionality is controlled by Primary and Secondary roles.  These roles are assigned to the first five hosts added to the cluster and can change whenever a host level HA event takes place, like putting host into maintenance mode or removing/adding a host to the cluster.   This entire process is automated by vSphere currently there is no supported way of controlling this process.  Because of this, stretched clusters should be limited to a maximum of 8 nodes.   If we go beyond this number and have five nodes at one site, it is possible that each of your primary HA nodes would be located at a single site.  This means the second site would be vulnerable to not having HA functionality if the first site were to go down.

Problem #2 – DRS has no site affinity awareness.

Currently DRS doesn’t have the capability to know which side of the cluster virtual machines reside at. DRS could potentially move VMs between sites and create a situation where bandwidth of the site links and applications within the VMs could be negatively affected.

It was my recommendation to our client that they abandon the stretched cluster design and implement a proven DR solution such as SRM.  In this specific environment, SRM is a perfect solution to provide the level of high availability they were looking for.  With active and passive storage at both sites, the local active storage at each site would replicate to the passive at the remote site.  This configuration would allow for the splitting of the cluster, allowing for better resource utilization across all hosts, direct insight and control as to where specific VM’s were running, and a fully supported solution by VMware.

Finally, implementing this type of design with SRM would mean you are only failing over 50% of your total VMs in the case of site failover thereby reducing the recovery complexity of any DR event.

SRM, Storage, vSphere, WWT , , , , ,

XenDesktop on VMware Infrastructure: Tips and Tricks

4 Comments Posted by on March 20, 2010

After running across issues with these types of deployments and not finding any helpful information on the web, I thought it would be good to stick some of it in a blog post.   Hopefully this will help others that are venturing into similar scenarios!

There are many reasons why (and why not!) to use this approach for VDI in your environment.  That being a whole other topic of debate, what we want to discuss here is the actual integration point between the two vendors.

How does XenDesktop hook into vCenter?   Why…   And what are its dependencies?

If you hit your favorite search engine and look for XenDesktop and VMware you will get plenty of results on how to point XD at the SDK URL of your vCenter server.  A few things I found lacking…

  1. What is vCenter used for?
  2. What will happen if vCenter goes down?
  3. Recommendations on making vCenter highly available for XenDesktop
  4. What are those odd tags in the notes field of my VMs now?

XenDesktop Metadata in vCenter

What is vCenter used for?

The XenDesktop pool management service on the DDC (Desktop Delivery Controller) relies on vCenter for virtual machine lifecycle maintenance.  This includes a query for the current power status of the VMs via vCenter including power on, off, suspend, and reset.  Finally, the XenDesktop Setup Wizard utilizes vCenter to automatically create new VMs based off of required templates.  This part is comparable to what View Composer does for VMware View environments.

So what will happen if vCenter goes down?

The DDC will no longer be able to query the status of the remaining VMs within the desktop pool, nor will it have the ability to power on any more VMs.  Eventually, as users log on to available desktops, all available VMs will be consumed and no more users will be serviced.  This also means the XenDesktop Setup wizard will fail!

  • The Good – Existing connections will be unaffected.
  • The Bad – New sessions have the potential of not being serviced.

Picture a 24/7 shop running this exact scenario:  XenDesktop implemented on a VMware infrastructure.

Midway through the first part of a work day, vCenter goes down and is unrecoverable.  Additional users continue to arrive to work, which causes the buffer of VMs that the pool management service typically maintains to be utilized and not replenished.  As desktops are consumed and since vCenter is unavailable, no new VM’s can be turned on.  This means we could have a majority of our workers sitting without desktops.  Bad stuff!

On a slightly less critical note, XenDesktop Setup Wizard will also fail should you try to use it while vCenter is unavailable.

So, how do I keep vCenter alive and available?

There are a few items I’ve found that can be utilized to help ensure vCenter is always available.

  • Still physical?   Get that thing virtual and minimize your downtime with HA. (Make sure it is set with the highest restart priority.)
  • Add some additional available desktops to your desktop pool.  This will help minimize down time and allow HA to restart vCenter on another ESX host by having more of a buffer available.
  • Backup your vCenter DB, no question – this must be done.  There are XenDesktop thumbprints placed in the VM note field that are extremely important (explained more below).
  • Look into VMware Fault Tolerance for vCenter if it is virtual.  One caveat of course: If it is a multi-vCPU vCenter deployment VMware FT will not work.
  • Look into vCenter heartbeat and add a second vCenter server.
  • Backup your vCenter DB, yes it is that important to say it twice.   It is quite amazing how few do this!

What are those odd tags in the notes field of my VMs and why do I care?

This is the XenDesktop VM metadata.  It helps tie all of this madness together.  And all of this is kept in the vCenter database.  And here is the third time, backup that database!  If that data is lost, have fun re-reregistering all of your VMs to your pools again.

  • CTXGuestOSID – this is the Guest OS SID.  This data is written by the DDC Farm Master. Once that data is there it is referenced by the DDC to keep the virtual machine and the Guest SID aligned for the PVS service.
  • CTXGuestMGTInfo – this is also written by the DDC Farm Master. This is what keeps the VM and the Pool management service aligned and happy.  Looking at this data identifies which pool it belongs to, as well as how the DDC can query power state of the VMs in that pool.

The vCenter server used to be a mid-tier application in my mind.  If it goes down, you may affect a couple admins – but the world still turns.  Because of the ties to XenDesktop, it must have a new focus.    You may have deployed the most bullet proof, highly available, and scalable XenDesktop deployment ever.  However, if vCenter is at the heart of the back-end virtual infrastructure you must ensure it is also equally highly available.

Thoughts, counter points?

~Jason

Citrix, vCenter, VMware, XenDesktop , , , , , ,

Flex10 with vSphere

5 Comments Posted by on February 14, 2010

I’ve been actively engaged with a health care client assisting them with a design for over 100 HP blades, in a C-7000 chassis with Flex10 and vSphere.   Having not found a whole lot of information out on the web, I figured I’d write about my experiences to open the doors on this design discussion and kick off things with our blog!

First a little background on Flex10.  Flex10 is an HP technology which allows you to split up a single 10Gb NIC into logically separated NICs (called FlexNICs).  At the same time, we are able to take that 10Gb connection and dynamically allocate it across the FlexNICs.  Bandwidth speeds can be specified from 500Mb to 10Gb on each FlexNIC.  As you can see from the picture, each blade server would have two physical 10Gb Flex Adapters which is logically split into four separate FlexNICs.  In total what the OS would see would be eight separate NIC’s.

Flex10_1

As with any blade chassis, one benefit of having blade servers in your environment is the ability to share network bandwidth by having a single (or multiple) network connections coming into the chassis which is then shared amongst all blades.  While in reality this will cause for a bandwidth over commitment, with today’s 10Gb network connections and the ability to add multiple uplinks, this 10Gb can be quickly expanded as bandwidth needs arise.  Interconnect modules/switches are installed in the back of the chassis to handle this network sharing.  In our case, a Flex10 Interconnect module is used.   Interconnect modules are installed in pairs and then each Flex10 Adapter is then mapped to the Flex10 Interconnect module.  It’s important to understand that the adapters are only connected to a single Interconnect thus the requirement and need to install both components in pairs to ensure redundancy.

Flex10_2
Now that we have a high level understanding of what Flex10 is and the various components, lets take a look at how this technology can be leveraged within a virtual environment.  Since the Flex10 adapters are logically split into four separate connections, ESX will see a total of eight vmnics which we can assign to virtual switches.  This is where the interesting and fun (depending on what you classify fun) part begins!
With any virtual switch design, it is necessary to ensure network redundancy, typically accomplished by assigning multiple vmnics to a single virtual switch.  To visualize how we might do this with Flex10 let’s consider the following.

Flex10_3

As you can see I’ve assigned one FlexNIC from each Flex adapter to a virtual switch.  I’ve also allocated bandwidth to the individual FlexNICs according to my needs.  This is an extremely useful benefit of the Flex10 allowing the separation/customization depending on specific environment needs.  It’s important to keep in mind the aggregate bandwidth of all FlexNICs on a single Flex adapter must equate to a total of 10Gb.  Hopefully up to this point, everything has made sense.  Part of any virtual switch configuration is the creation of port groups.  Here is where specific VLAN’s are specified to indicate what network segment is to be served by that specific port group.  In order for the specified VLAN’s to be assigned, we first need to look at how the Interconnect modules are configured from a networking perspective.

Interconnect modules are connected to upstream switches in a couple different manners.  Depending on the environment, it is important to be aware of limitations.  The first way we can configure uplinks is what HP calls “Shared Uplink”.  With shared uplinks you can take multiple ports and bond them together.  This in a sense creates an ether channel.  (Note: you cannot create an ether channel containing uplinks from each Interconnect module.  If an ether channel is created it must consist of multiple ports from the same Interconnect module.)

Some limitations you’ll want to be aware of when configuring Shared Uplinks are:

  • A total of 128 VLAN’s is currently supported by the Interconnect modules.
  • If you are to be setting things up in an active/active manner as I have above, then the max per Interconnect module is just 64 VLAN’s.
  • All VLAN’s must be specified on the Interconnect module in order for that VLAN to be assigned to a FlexNIC.
  • A total of just 32 VLAN’s is the max assigned number of VLAN’s per FlexNIC
  • You cannot assign the same VLAN number(s) to multiple FlexNICs residing on the same Flex Adapter.
  • A max of 4092 MTU can  be set when enabling jumbo frames (VMware best practices suggests setting an MTU to 9000 so it is important to pay attention to this note as setting it incorrectly can cause storage performance problems)

The second way we can configure uplinks is called “Tunneled Uplink”.  With this configuration, you are creating a trunk from the upstream switch and passing it all the way through to the FlexNIC.  The only limitation that I’m aware of with this type of configuration is there is no way to assign that trunk to multiple FlexNICs residing on the same Flex Adapter.  This is because the same VLAN would be presented to multiple FlexNICs which is also not supported in the Shared Uplink set.  I suppose you could have multiple Tunneled Uplinks assigned to different FlexNICs but some sort of pruning would have to exist on the trunk to ensure similar VLANs do not impact one another.

With the limitations above for Flex10, the best way I have found to effectively setup ESX virtual switches is to configure things as above, and then have a set of four uplinks (two per Interconnect module).  With this configuration, both a Shared Uplink set and Tunneled Uplink can be specified.  With the Shared model a single uplink is assigned from both Interconnect Modules and break out the various VLAN’s.  This Shared Uplink set would be dedicated to the Service Console, VMotion, NAS/iSCSI, and VM FT traffic.  Because of this, minimal VLAN’s would need to specified, for very easy setup and configuration.  This would then leave the remaining two uplinks which would be setup in a Tunneled Uplink.  This set of uplinks would be dedicated to virtual machine traffic and therefor would be sent straight to a FlexNIC where you could assign the necessary port groups.  To put a picture with all that, it would look like the following.

Flex10_4Flex10_5

By configuring things in this manner, vSwitch0,1,2 would belong to a Shared Uplink (denoted by the red cables) and vSwitch3 would belong to the Tunneled Uplink (denoted by green cables).  Redundancy is provided by assigning a FlexNIC from each Flex Adapter.  Easy enough right?  Well so I thought…

The final piece of this puzzle and where I’ve run into problems is related to how vSphere detects a network failure.  Because of the way blades work, they will always have a link to the Interconnect module meaning setting the redundancy on the virtual switch to be “Link Status” won’t work without some further configurations (below) since a network failover upstream won’t be relayed down to the individual FlexNICs.  The next logical answer would seem to be to use a Beacon Probe but in my testing this also will not work.  So now what?

Flex10 has two features called “Smartlink” and DCC (Dynamic Channel Control).  Smartlink and DCC enable the Interconnect modules to relay link status of the uplink switches to the individual FlexNICs however until very recent, this was not supported by ESX.  In order to take advantage of Smartlink and DCC, new drivers and firmware from VMware and HP must be applied.  HP’s firmware can be found here and will be applied to the Flex Adapter of each blade.  Once applied, VMware’s new driver found here will need to be installed.  I’ve listed steps below that you can follow to install them.

  1. Extract the contents of the ISO.  We’re after the zip file in the offline-bundle directory.
  2. Copy the zip file to your ESX host.
  3. Connect to your ESX host and issue the following command to install the new driver “esxupdate –bundle=<filename> update”
  4. Reboot and your new driver will be enabled!

Once the new firmware and driver have been installed, full support of Smartlink and DCC will be enabled meaning we can go back to a “Link Status” failover detection method within ESX!

So to bring it all together, I definitely see the benefits of the Flex10, especially in how you can control the bandwidth down to the individual FlexNICs.  I also like the logically separated FlexNICs as it allows us to be plenty creative in how we choose to configure our virtual switches.  Like all new features/technology, being on bleeding edge has its disadvantages such as the driver and firmware issue to be aware of but with a little work and knowledge, the benefits of Flex10 can be taken advantage of.  The only other item to consider is the requirement of having to individually assign VLAN’s on the Interconnect modules when utilizing Shared Uplinks.  It’s definitely something to keep in mind should you run into issues where lots of VLAN’s are required, however in most cases this can be corrected by setting up a Tunneled Uplink.  To date, HP is the only vendor out there allowing such splitting of network I/O in this manner and I’d love to hear people’s thoughts on the Flex10, their implementations, and use cases.

Closely related to this discussion is Cisco’s “Palo” card for their UCS platform, which is to compete with HP’s Flex10 but I’ll have to leave that to another discussion…

Cisco, ESX, iSCSI, Networking, Storage, VMware, vSphere , , , , , , , , ,

Newer posts →

Follow

Get every new post delivered to your Inbox.