Real World Cisco UCS Adapter Placement

I had the opportunity recently to deploy 18 B420 M3 blades across two sites.  Having only deployed half width blades over the last two years, I had to change my usual Service Profile configuration for ESXi hosts to ensure the vNICs and vHBAs were properly spread across the two VICs installed in each blade.  Each B420 had a VIC 1240 and a VIC1280.  The Service Profile for the blades includes six vNICs and two vHBAs.  The six vNICs were used to take advantage of QoS policies at the UCS-level.  The six vNICs configured included:

  • vNIC FabA-mgmt
  • vNIC FabA-vMotion
  • vNIC FabA-VM-Traffic
  • vNIC FabB-mgmt
  • vNIC FabB-vMotion
  • vNIC FabB-VM-Traffic


So the goal of the Adapter Placement Policy, in this case, would be to ensure that if a VIC failed, it would not cause a total traffic disruption for any particular type of traffic.  For instance, if the VIC 1280 failed, I would not want both VM Traffic vNICs to be mapped onto the 1280 and therefore cause all VM traffic to cease because of the failure of one piece of hardware.  Instead, I need to make sure that only one vNIC of a particular traffic type is mapped to any single VIC.

The Problem

Before creating the Adapter Placement Policy, the default Adapter Placement Policies ended up putting both ESXi management vNICs on Adapter 1 (the VIC 1240) and both vMotion vNICs on Adapter 3 (the VIC 1280).  This is what I’m trying to avoid.  By chance, the VM Traffic vNICs were spread across both Adapters “correctly,” but that could have happened to any vNIC.

The Fabric A ESXi management vNIC is on Adapter 1.


And the Fabric B ESXi management vNIC is also on Adapter 1.


The Fix

So how do we fix this?  The same way that we would with a physical server.  Ensure traffic types are spread across different PCIe slots or channels.  With the UCS, we use Adapter Placement Policies to accomplish this.

Create vNIC/vHBA Placement Policies

First I created a vNIC/vHBA Placement Policy.


The Virtual Slots will align or map to where your physical adapters are installed.  The VIC 1240 is always installed in Adapter Slot 1.  Check out the Configuring Server-Related Policies section of your relevant UCS version configuration guide for details on this.


The B420 M3 has three adapter slots, though.  While the 1240 can only be installed in slot 1, the 1280 can be installed in slots 2 or 3, as shown below.  The diagram below comes from the B420M3 Installation and Service Note.


In my case, the 1280 is installed in slot 3 and therefore, is shown as Adapter 3 in the Equipment tab.

The Selection Preference identifies which vNICs or vHBAs will be mapped onto which adapter.  In this case, I chose Assigned Only because I want the admins to think about where their vNICs are being placed when they create them, rather than just letting them be placed any adapter, possibly to adding a single point of failure to a particular traffic type.  Assigned Only means “only map vNICs/vHBAs to these Virtual Slots if they’re explicitly or statically assigned.

I found that choosing Round Robin or Linear didn’t actually make any difference as to how the vCons were mapped to the physical adapters.  According to the vCon to Adapter Placement for All Other Supported Servers section of the config guide, I should have seen this mapping change, however, I didn’t.  You can use the “show vcon-mapping” command to see this.


Fortunately, it doesn’t really matter in my case so much so I just let it be.  You just need to know what this mapping is with regard to vCon’s and Adapters.  You can use the command above or simply look at the Server adapters on the Equipment tab as shown below.  I see Adapter 1, which is the VIC 1240 and Adapter 3, which is the 1280.


So in total, we have this type of mapping between the hardware and UCSM configurations for my particular case:

  • Virtual Slot 1 –> vCon1 –> Adapter 1 –> VIC 1240
  • Virtual Slot 2 –> vCon2 –> Unused
  • Virtual Slot 3 –> vCon3 –> Adapter 3 –> VIC 1280
  • Virtual Slot 4 –> vCon4 –> Unused

With this information in hand,

  1. What vNICs and vHBAs I have configured
  2. What adapters I have and how they’re mapped from hardware into UCSM

I was able to draw out what I was trying to do.  While the 1240 and 1280 still have connections to each fabric, in order to provide hardware redundancy across the VICs I chose to map the Fabric A vNICs and vHBAs to the 1240 and the Fabric B vNICs and vHBAs to the 1280.  This is just one way to do it, but it’s clear and it’s consistent and it meets my objectives – so I’m good with that.


Modify vNIC/vHBA Placement

Now we assign the Placement Policy to the Service Profile and configure the actual placements.  To do this, we select the Service Profile and go to the Network Tab > Modify vNIC/vHBA Placement.


Select the new Placement Policy at the top and start configuring the vNIC/vHBA mappings.  Again, I worked out above where each vNIC and vHBA would be mapped so this is simply putting it in the configuration.


The full vCon 1 configuration is shown here:


And the full vCon 3 configuration is shown here:


A reset of the Service Profile re-mapped the vNICs and vHBAs.

Before the change, we can see the mapping wasn’t useful at all.


And after the change, it’s nice and orderly and spread across the hardware.


Or in the GUI, we can see the effects of the placement policies by viewing the Desired and Actual Placements.  The screenshots below are from different Service Profiles because I didn’t keep track of which ones I was taking screenshots of.  The first screenshot shows the vNIC placement before correction.  Again, the ESXi management vNICs are placed on Adapter 1 and the vMotion vNICs are placed on Adapter 3.


And after creating placement policies, the Fabric A vNICs are mapped to Adapter 1 and Fabric B vNICs are mapped to Adapter 3.  While I don’t show it, the vHBAs are mapped correctly, too.  You’ll just have to take my word for it. ;-)


10 Years, $10, 10 Days at Packt Publishing

Packt Publishing is celebrating 10 years publishing its technical tomes and they’re inviting everyone to celebrate with them.  While this post is coming out at the tail end of the promotion, you still have time to get in on the action.  It’s good until July 5th.

You can buy as many books as you like for $10 each.  Check out their deals here:

Initial thoughts: VMware vSphere 5.x Datacenter Design Cookbook

I was recentDatacenterDesignly given the privilege to review Packt Publishing’s recent book about vSphere design.  I was immediately pleased to see that recent VCDX (graduate? achiever?) Hersey Cartwright of #vBrownbag fame was the sole author.  I always appreciate knowing what I’m about to put in my brain came from a trustworthy source.  I see in his author bio, though, early in the book, that he’s “only” recognized as a VCAP, not a VCDX (VCAPs are all-stars to begin with, dont’ get me wrong).  So he must have at least started working on this before he achieved rock-star status.  I couldn’t help but think as I read on how much writing this book must have helped his VCDX attempt. 

I’ve read a lot, I mean a lot, of VMware books and articles and blog posts – just about everything I can get my hands on – and everything in this book I kept nodding along with. There were many times Hersey would broach a subject and I’d immediately look for him to cover those oh-so-important caveats.  Sure enough, he covered them.  I was very happy to see that we were on the same page.

So with respect to design books, this is essentially the 3rd of its kind I’ve read.  The first, of course, was the Sybex vSphere Design (both editions), then I was very pleased to read VMware Press’s Managing and Optimizing VMware vSphere Deployments, which, while not strictly design-focused, hit on many design features nonetheless.  Each is excellent and I recommend them.  What makes Hersey’s different is that it’s short and to the point (vSphere Design is over 500 pages).  This book is under 250 pages but packs in the relevant information you need be a good architect or designer.  Most importantly, let me emphasize this

Hersey doesn’t give you a fish in this book.  He teaches you to fish.

What I mean by that is in each section, he’s not simply listing the answers you’re looking for to design a redundant virtual network or to build reliable storage – he couldn’t possibly.  What I feel he does throughout is explain the concepts and then teaches you to ask better questions that lead to a good design.  That’s not quite anything like what I’ve read in any other VMware book.  I don’t feel Hersey wastes a sentence.  An additional feature of this book, that also makes it unique from others I’ve read, is that it discusses how to build documentation to support a vSphere design.  It’s not coincidence that Hersey mentioned each type of document that is likely needed in a successful VCDX defense.  Congratulations, Hersey – you’ve made a one-of-a-kind book. Thanks for sharing.

Initial thoughts: VMware vCenter Operations Manager Essentials

VMware vCenter Operations Manager EssentialsI recently had the opportunity to review this title by the good folks at Packt Publishing.  They provided me with a free e-copy for this review.  It was good timing, too, because a new project of mine involves more in-depth work with vCOPs than I’ve had so far.

My experience with vCOPs has been a couple installations and initial configurations for clients.  I have not customized vCOPs greatly and have mainly used it’s default capabilities to glean health, performance, and capacity metrics for reporting purposes.  So far, the topics I’ve found most useful include the troubleshooting performance sections in Chapter 4.  This is where many people will find value in vCOPs.  In addition, Chapter 5, covering capacity planning, is another area in which I’ve spent most of my time.

For someone like me who has dabbled in vCOPs, I’ve found this book useful more as a reference than one I need to read cover to cover.  I imagine, though, for those new to the product, the chapters on installation and initial configuration will be worth your time.

I also encourage you to visit Lauren’s blog, check her out on the twitters, and support her book. 

Book: VMware vCenter Operations Manager Essentials


Twitter: @malhoit

An Open Thank You to Pluralsight

This post is almost a year overdue.  I apologize to Pluralsight and the company formerly known as Trainsignal (whom Pluralsight acquired last year – I’ll refer singly to Pluralsight for the remainder) for the lateness of this post.

Thank you.  For the second year in a row, Pluralsight continued its over-the-top generosity by supplying those recognized as 2014 VMware vExperts with a free, yearly subscription to their immense IT and development training courses.

Thank you.  For the second year in a row I was blessed to be recognized as a vExpert for my contributions to the vCommunity and I received Pluralsight’s complementary subscription.

Thank you.  Many of us in IT know that our currency, the value to ourselves, our career, our clients and companies, is knowledge.  We value knowledge to a high degree.  Our livelihoods and those for which we work, to a large degree, rely on how well we know our craft.  Pluralsight’s gift of a free year of their premium IT training is incredible.  I value it so much because of the usefulness and success I’ve found with it.  Training in IT is crucial if you want to stick around very long.  One has to make it a priority if they want to succeed.  Pluralsight’s gift is like handing an IT guy a link to success.  Here you go – do great things.

Thank you.  Pluralsight, I know you didn’t have to do this. It means a lot to me that you think highly enough of this group of evangelists to give freely of your time and efforts in producing the highest quality training.

Thank you.  From a very grateful IT guy.

Getting Started with EMC Storage

EMC LogoSo I’ve been asked to get up to speed on EMC.  We’re really a Flexpod company, which includes Cisco, NetApp, and VMware, but we’re finding clients in our Dallas / Fort Worth market that are already EMC customers and would like to stay that way.  Many times, it makes more sense for these shops to stick with EMC.  One reason for this is that we find small to mid-size shops that already have the technical expertise to manage an EMC environment are sometimes reluctant to defenestrate the platform on which they’re most comfortable.  Selling them a NetApp would not be setting them up for as much success as we would if we left them with hardware already in their management comfort zone, assuming the sale met their needs in the first place.

And so here I am, ready and raring to learn EMC.  I’m already a NetApp dude and have a firm grasp on storage fundamentals, but this will be my first foray into EMC.  While I’m already using my google-fu to find training aids, I’d love to hear the recommendations from the vCommunity on how best to learn EMC products.

I’m already tracking on Joe Kelly’s Pluralsight training which I’m sure I’ll lean on heavily.  I also know that Nick Weaver has a simulator out there.  What other free learning tools would you recommend?

Book Review: Implementing VMware vCenter Server

Book Cover: Ipmlementing VMware vCenter Server

I was recently given the opportunity to review Packt Publishing’s recent release of Implementing VMware vCenter Server: A practical guide for deploying and using VMware vCenter, suitable for IT professionals.  At first glance, I wondered how an entire book could be written about vCenter alone.  While reading it, though, I was pleasantly surprised time and again when I saw how much good information was shared.  This book is an excellent primer for those new to vCenter and really, VMware in general.

Read the rest of this entry »


Get every new post delivered to your Inbox.

Join 1,238 other followers