March 6

Back To Basics: Private VLAN (4 of 9)

You know the more I get back into Distributed Switches – the more this is looking like a 10/20/30 part series…! This part concerns Private VLANs or PVLANs. Admittedly, its not a widely used function of many physical switches – but I have occasionally seen it used by ISP and Service Providers. So at some stage I’m going to have to renumber all the previous parts to this series to fix that up! So from now on this series is number of N, and I will fix that numbering when I think I’m finished! 🙂

The VMwareKB YouTube channel has short video which explains how PVLANs work which you might useful if your configuring this feature for the first time.

Private VLAN on a vNetwork Distributed Switch – Concept Overview

Private VLAN or PVLANs is an extension to the VLAN standard, and allows for the extension of a single VLAN into secondary PVLANs. These secondary PVLANs reside within the domain of a primary VLAN. As with VLANs, VMware’s implementation of PVLAN allows the Distributed Switch to be aware of the underlying physical switches PVLAN configuration. PVLANs are usual seen within the Service Providers, ISPs and government bodies – they are less prevalent in the corporate world, and hardly if ever configured at the SMB markets except in unique use cases.

There are three types of Secondary PVLAN:

  • Promiscuous
  • Isolated
  • Community

640px-Screen Shot 2014-02-26 at 15.26.07.png

Each type allows determines how packets are forwarded or not. A node attached to a Promiscuous PVLAN can send and receive packets to ANY other node residing on Secondary PVLAN associated with it. Typically, devices such as routers are attached to Promiscuous PVLANs to act as a gateway to further networks. An Isolated PVLAN has more limited communication properties, it can only communicate to and from its configured Promiscuous PVLAN. It cannot communicate to other Isolated PVLANs or to Community PVLANs – additionally, nodes WITHIN a Isolated PVLAN cannot communicate to each other either. Isolated PVLANs are typically used to create a DMZ for firewall purposes – as one compromised node, cannot be used as source to launch attacks to others. The Community PVLAN allows a node to send and receive packets from other ports within the same PVLAN, and can communicate to the Promiscuous PVLAN. For a Distributed Switch to interact with a PVLAN implementation it must support and be enabled for 802.1q Tagging. Physical switch can be confused by the fact that in some cases each MAC address is visible in more than one VLAN tag. For this reason Physical switch must trunk to the ESXi host and not be in a secondary PVLAN.

The configuration begins with defining the Promiscuous PVLAN, and then adding support with the various secondary PVLANs to which it is associated with. The PVLAN is define as flat list with just the PVLAN ID number in left-hand column of the UI with the Secondary PVLANs defined on the right-hand side. Once both Promiscuous, Isolated and Community PVLANs have been defined they can be utilised by the Distributed Portgroup

Define the Promiscuous PVLAN

Continue reading

Category: BackToBasics | Comments Off on Back To Basics: Private VLAN (4 of 9)
February 28

Back To Basics – Migrating to Enhanced LACP (Part 3 of 9)


Recommendation: If you are intend to use LACP consider deploying from day one rather than migrating from Distribute Uplink containers. The migration is simple but adds additional steps that would be unnecessary.

Image Source: KB2051826


Distributed Switches support an enhanced version of LACP which allows for the use of dynamic link aggregation. Prior to vSphere 5.5 only single Multiple Link Aggregation Groups (LAGs) could be created – with the release of vSphere 5.5 up to 64 LAGs can be created per Distributed Switch. Despite these new enhancements a number of configuration requirements to do exist in terms of its current implementation – consult KB2051307 for further details. There is more detailed documentation that outlines the differences between vSphere 5.0, 5.1 and 5.5 which you should consult if you are running in a mixed environment – these are details in KB2051316

However, expressed briefly LACP does currently:

  • Support is not compatible with software iSCSI multipathing.
  • support between two nested ESXi hosts is not possible (virtualized ESXi hosts).
  • LACP cannot be used in conjunction with the ESXi dump collector. For this feature to work, the vmkernel port used for management purposes must be on a vSphere Standard Switch.
  • Port Mirroring cannot be used in conjunction with LACP to mirror LACPDU packets used for negotiation and control.
  • The teaming health check does not work for LAG ports as the LACP protocol itself is capable of ensuring the health of the individual LAG ports. However, VLAN and MTU health check can still check LAG ports.

Continue reading

Category: BackToBasics | Comments Off on Back To Basics – Migrating to Enhanced LACP (Part 3 of 9)
February 26

Back To Basics: Managing Properties of Distributed Switch (Part 2 of 9)

Wooo-weee. Aren’t there a lot of settings and options on both Distributed Switch and the Portgroup. It can feel a bit daunting if its the first time you have looked, or if like me, you haven’t looked at them in a while. So many knobs, which I know to twiddlers everywhere are so appealing – like a big red button that says “Do not press this button”. You just can’t help yourself!

So I’m going to have spread this content over a much longer series of posts than I’d originally intended – but I think that’s better than one monster post that covers absolutely everything. In this first post on the settings of Distributed Switch, I’m focusing just on the Distributed Switch itself. Just the topology, MTU, CDP/LLDP side of the house – subsequent posts will look at:

  • Migrating to Enhanced LACP
  • Private VLAN
  • NetFlow
  • Port Mirroring
  • Health Check
  • Using Network I/O Controls and Network Resource Pools


and then (deep breathe, I will look at the setting on a Distributed Portgroup)

The Distributed Switch offers up a whole host of advanced settings applicable to the Switch and the portgroup. Not all of these settings will be applicable, important or supportable in your deployment. However, we have chosen to cover these in as much detail as possible – highlighting the options that appear to be most commonly required in most environments.


The topology view is a good way to get an overhead view on the Distributed Configuration. He was can see the three distributed portgroups; the IP addresses used by the FT VMkernel portgroup; The physical NICs associated with the hosts and the UpLink container. The small (i) information icon allows you to view the settings on each component.

Screen Shot 2014-02-12 at 14.57.13.png


The general properties options allow you to:

  • Rename the Distributed vSwitch
  • Decrease/Increase the number of UpLink contains, as well as rename them
  • Enable/Disable Network IO Control (Enabled by default)
  • Set a description

Screen Shot 2014-02-12 at 15.11.31.png

The Advanced propeties options allow you to:

  • Set the MTU size for Jumbo Frames
  • Configure the Cisco Discovery Protocol settings, and switch to Link Layer Discovery Protocol
  • Set a name and contact details for the Administrator

Screen Shot 2014-02-12 at 15.48.40.png

The MTU value is applied to all communications passing to/from the Distributed Switch – and its important all paths in the communication flow are configured for the correct MTU size. If the MTU size impact on a virtual machine portgroup the MTU size should be adjusted within the guest operating system to a matching size. This is to aviod a scenario called fragmentation. If a 9000 MTU ethernet packet encounters a 1500 system then the packet will be split into 6×1500 packets which will actually reduce performance, and increase the overhead on the device/system that carries out the fragmentation.

Assuming your physical switch supports Cisco Discovery Protocol (CPD) the support can be adjust to listen/advertise/both. Listen enables the vSphere Administrator to query and return information from the physical switch – this can be useful in diagnosing configuration mismatches. Likewise Advertise allows the Cisco administrator to query the Distributed Switch as if it was a physical switch. Bothallows for a combination of Listen/Advertise.

CDP is available for both Standard and Distributed Switches, but Link Layer Discovery Protocol (LLDP) is only available on version 5.0 Distributed Switches or higher. Assuming the physical switch supports one of these protocols, then you should see information in the (i) icon on a physical adapter like so:

Screen Shot 2014-02-12 at 16.26.28.png


Category: BackToBasics | Comments Off on Back To Basics: Managing Properties of Distributed Switch (Part 2 of 9)
February 25

It’s the most wonderful time of the year…

top-vblog-2014-2-cropYeah, I know – totally now unseasonal and inappropriate blogpost title. It’s that time of year when Eric Siebert starts the voting process for Top vBlog of the year. As you might know I was in the top ten for some years with my “RTFM Education” site. Last year was the first year really of “” proper. For a while I was having to blog over on the communities website, as I saw out a period of “non-compete” with TechTarget who acquire the ye olde site back in 2009. Anyway, I hope many of you will vote for me again – and keep me in that all important top 10 slot!

Probably my biggest contribution in 2013 was my mega-blogpost series all about vCloud Director. It ended up being some 70 odd parts in total. I do hope to carry on with a new series all about vCloud Automation Center. Recently, I’ve been side-tracked by trying to use Windows 2012 R2 Hyper-V and System Center Virtual Machine Manager 2012 R2 in anger. The theory was to get outside of my “VMware Bubble” to really see what the competition is capable (or should that be incapable!) off. I hope folks have found those post insightful and balanced. I tried to carry on with my “warts and all” style which has been a hallmark of my documentation in the past. Nine times out of ten, its my good self that is cause of all the problems – but I figure as I’m human I’m allowed to make mistakes, what matters is ‘fesssing up to those – in the hope that others will learn from those experiences. More recently, I’ve embarked on a Back To Basics series of posts  – revisiting every aspect of the vSphere 5.5 platform, together with companion PowerCLI examples. That was inspired by me feeling I was getting disconnected from core virtualization from VMware. I hadn’t serious written about vSphere since the 4.0 book I wrote – I’d become so focused on SRM, View and latterly cloud – that I felt I need to do something to keep my vSphere knowledge sharp and up-to-date. More recently I’ve been recently revisiting those Back To Basics posts with demo videos and discussion videos too!

On the community front I continue to attend and support the VMware User Groups program – not just by speaking myself but by trying to mentor the membership to speak as well, a process I’ve dubbed #FastForward. One of my most popular posts last year was about using AutoLab on, as well has hosting some “skinny linux” distributions on Both of these efforts are about helping the community learn more about VMware Technologies with low upfront cost, and will minimal resources.

I’ve got some other big plans for 2014 which lie at outside of the blog, but are content related. It’s early days yet so I need to keep quiet about those for now. But keep in touch with the blog to find out more in the coming months.

So please take a little time to spend your vote. This year the voting is weighted – so vote for me in the No1 slot is 10 points, a vote for the No2 slot is 9 points and so on…

Category: Announcements | Comments Off on It’s the most wonderful time of the year…
February 23

How to get a broken server door key out of HP ML350e


Yes, I know how esoteric and specific can a blogpost get, eh?

I’d only had my new HP ML350e series servers in a homelab until one day disaster struck. I accidentally broke a server key in the lock on one of the servers.

The ML350e has one of those key setups where you cannot leave the front door unlocked. When you unlock the server you can’t remove the key. The key also locks the outer-panel which is used to get to the main system for upgrades – and physically secures the server (bear in mind anyone with genuine physical access can pull the power cords and disconnect the network!). I’d prefer to leave the server unlocked, so I loose the key I can still have access. The trouble is in an unlocked mode, you cannot remove the key. I’d left the keys hanging in the locks with the front door pushed shut (Actually, I sometimes do this with my house door/key. Yes, I’m that forgetful. My mum calls me the absent minded professor).

With the server stood vertically, and side-by-side. It wasn’t long before shifting of the server caused one to bang into another, and bend the cheap and nasty key. Of course, I bent it back into shape. The third time this happened it sheered off right off. I tried to the pull remainder of the keyout, and only succeed in pushing it in further. I know. Utter Face Palm…


So how to get the key out? First I consulted the internet. This guys is great…

So sprayed the lock with the workmans friend – WD40. And then stuck this very thin screwdriver into the lock along side the key. The screwdriver is the kind you might use to unscrew the screws on an iPhone or such like. I found a place where it would slip down the side of the broken key – and then pulled it very quickly. I did this couple of times. The frictional force of the mini-screwdriver as it was removed dislodge the key enough so I could see the end of it. I bit more leverage with screwdriver helped bring the key out enough that I could use some tweezers to remove it… RESULT!


Moral? Lock the server. Secure the keys. Don’t bend keys, they sometimes don’t bend back. If your too abscent minded not to loose keys, loop them to the back of the server, or staple them to your forehead.

Category: Other | Comments Off on How to get a broken server door key out of HP ML350e
February 20

Back To Basics: Managing a Distributed vSwitch (Part 1 of 9)

Well, its the start of a new “chapter” in the Back To Basic series. This time focusing on networking. Hang on I hear you say, didn’t you cover that way back when. Your right I covered Standard Switches (vSwitches?) and ploughed on into Storage (so I get my Synology and IOMega NAS setup)… But how would you do all that with distributed switches instead.

Introduction to Distributed vSwitches


Distributed vSwitches were first introduced in vSphere 4.0 – and since then various enhancements have been made as each subsequent release of vSphere has been released. At their heart, Distributed vSwitches are method of centralizing the management of the virtual network into single plane. Every VMware ESXi host added to Distributed vSwitch inherits its configuration, and those settings are stored within vCenter, rather than on the ESXi host itself. This means adding new portgroups for a new VLAN for a cluster of ESXi hosts are relatively trivial affair. The VMware ESXi host “caches” its Distributed vSwitch to local storage so if the vCenter is unavailable for whatever reason network communications are unaffected. However, no management of the Distributed vSwitch is possible until it is restored. For this reason some virtualization admins prefer that infrastructure VMs such as vCenter, SQL, Domain Controller and other VMware services and appliance remain on standard vSwitch to allow for continued management even if vCenter is offline.

Distributed vSwitch also off some features which are easier to configure than with Standard vSwitches, such as adjusting the MTU for Jumbo Frame support – in addition there are some unique features these include:

  • Private Virtual LAN support (PVLAN)
  • Port Binding
  • Traffic Shaping for both inbound and outbound traffic
  • Port Policies and Port Mirroring
  • Network IO Control and Network Resource Pools
  • NetFlow
  • Network Rollback and Recovery
  • Health Check
  • Enhanced LACP Support
  • Additional load-balancing options on a Distributed Portgroup called “Route based on physical NIC Load”

This part covers the basics of creating the Distributed Switch, Adding Hosts and creating portgroups for both virtual machine and VMware ESXi host networking…


Since the first publication of this blogpost I’ve added some videos. The first is a “Show Me How” video which demo’s how to create a Distributed Switch using “Template Mode” in the vSphere Web Client, it then continues to show how to add VMKernel ports also using template mode to bulk assign the IP addresses required.

Native Format

The second video is a “Discuss The Options” video with @chriswahl  (Social media adventurer, virtualization whisperer, and global event speaker. VCDX 104. Author of  &) In this video he talks about whether you should distributed switch exclusively, or whether you should use a combo of Standard & Distributed Switches, after we talk about the different network control available (load-balancing/distribution, LAGs, NOIC, Jumbo Frames and so on) and which pay dividends and which aren’t helpful in your environment.



Creating Distributed Switch

Continue reading

Category: BackToBasics | Comments Off on Back To Basics: Managing a Distributed vSwitch (Part 1 of 9)
February 17

Hyper-V R2eality: VMs not so hot after all…

When I first got into virtualization with VMware, one of the most compelling advantages was the fact that the VM is just a process (albeit one running an OS inside it) and it liberated you from the constraints of physicalization. That’s a word I invented to describe the situation where someone is foolish enough to run x86 OSes directly on a physical server. When Virtualization 1.0 came on the scene it seemed a revelation that if I wanted more memory, disk, network or CPU – I merely had to power off the VM, click a spinner and power back on. Nowadays, even that seems rather quaint and charmingly old-fashioned. We have got so used to being able to add resources on the fly, you would assume that every vendor in this so-called era of the “commoditized hypervisor” would have this functionality. I mean especially Microsoft Hyper-V 2012 R2, right?

I’m going to do a quick compare of vSphere 5.5 with Windows Hyper-V 2012 R2. I’ll be using the same guest OSes and the latest and greatest versions – Hardware Level 10 with the vSphere Web Client and a Gen2 VM in Hyper-V. Sadly, what you will find is the Microsoft Virtual Machine is more like the Physical Machine, because to make changes often as not you have to power it down. To me that makes Microsoft Virtualization more like a 1.0 era solution. It’s all somewhat bizarre coming from a company that championed “Plug & Pray” in the 90s.

These comparison uses a Gen2 VM on Windows Hyper-V 2012 running Windows 2012 R2 inside it – remember Microsoft doesn’t’t offer any supported automated way to do this – as my previous post outlined:

Hyper-V R2eality: On the long, long road to Damascus – Hyper-V Gen1 conversion to Gen2

Adding a PCI Device

Continue reading

Category: Microsoft | Comments Off on Hyper-V R2eality: VMs not so hot after all…
February 13

Return to the homelab (Part 4)

If you have been following this series of posts for a while. You’ll know I have an issue with my new HP ML350e Gen8 servers. I’m using a generic Intel Gigabit card, when the card is enabled the fan runs at 30-40%, when disabled it runs at 6%. The difference in noise is palpable, and plus there’s a certain “revving” noise which is particularly irritating. In the end I range HP support about this issue – and they politely explained that as the NICs were not genuine HP parts, then the liability was mine. That goes to show two things – firstly checking the VMware HCL isn’t really enough, and if you here people talk about “commodity hardware” that’s a bit of myth when your dealing with large OEM provider – that’s not a dig at HP by the way, you could apply this to any enterprize provider. It’s just the way our industry is – right or wrong.

Anyway, in the end I decided I would try a firmware update across the various components. I’m not convinced this will make ANY difference, as I think this is chipset issue issue associated with thermal detection of the device itself. But I think its good practise to keep this stuff update since servers generally ship with Jurassic versions anyway…

ILO Download:

ILO4 Update to 1.32; (Released: Nov, 2013)

Downloads as cp021805.exe. You need Windows to extract it (sorry to Mac users like me). From the ILO itself – +Administrator +Firmware +Click Choose File, and Browse for the .bin file.

Screen Shot 2014-02-12 at 21.26.19

HP Intelligent Provisioning Download:

Update to 1.50 (Released: 10 Sep 2013)

Download .ISO image. Create remote console session via ILO, and boot to the DVD… Allow the automate system to flash the environment

Screen Shot 2014-02-12 at 21.23.39

HP ESXi Offline Bundle for VMware vSphere 5.5 Download:

Despite the fact you download a HP.iso from, updates do exists onto of that GA’d release. This takes the format of .ZIP file that you can download from HP’s website.

You may or not need this bundle. The HP .iso on has the date 2013.09.26, and this bundle was released at the same time. So this is there for people who used a generic version of ESXi (good luck with that!) or have done upgrades via VUM, and want to distribute this offline bundle via other methods.

Update 1.5 (Released: 26 Sep 2013)

Use esxcli, VMware Update Manager or PowerCLI to install the update.  Personally, I like to upload bundles like this to shared storage, and then invoke the esxcli command  like this:

esxcli software vib install -d /vmfs/volumes/


Did these updates fix my NIC card issue? Nope… 🙁

I thought I would take a look at the system BIOS to see if the ID of the card had changed. It handn’t. I was also given a type about disabling the thermostatic controls for the PCI buses from the BIOS. I’d tried this before to no available. But perhaps the firmware update had taken affect.

Screen Shot 2014-02-12 at 22.46.04

Sadly, not the device is still reported as an “Unknown PCI Device”. Disabling of the power management on the PCI bus held on >>Power Management Options >>Advanced Power Management Options >>PCI-E Gen3 Control

Screen Shot 2014-02-12 at 22.49.33

Some folks have talked of running some sort update to the NIC cards – but I haven’t a clue how to go about to doing that. Plus I run the risk of bricking perfectly functional NICs which might have some resale value on ebay – and may well help finance a card purchase that is compatible.

What I’m going to do is work with cards as is for part of my work on Distributed vSwitches. Once that’s done, I will remove the physical NICs and ebay them (with warnings for folks running VMware ESX 5.5 on the HP Series of servers). Then I might consider replacing them with proper HP parts – or else I might just run on 2xNIC setup rather than the 4xNIC setup that I’m used to…

Category: BackToBasics | Comments Off on Return to the homelab (Part 4)
February 11

Return of the HomeLab (Part 3)

In this 3rd (and I’m thinking final) part of the return to the homelab series I want to write about storage. I’ve followed other peoples adventures over the years, and seemed that the Synology series of NAS boxes keep on coming up trumps. I did consider “building my own” with something the Nexanta Community Edition – because there was the possibility of going 10Gps with Infiniband. However, with all things considered I know most of my storage traffic generated by deploying VMs from templates and from moving VMs from one type of storage to another using Storage vMotion. I bet my running VMs barely touch even a 1Gps interface…

The thing that impressed me about the Synology series is how they have implemented VAAI (and Microsoft ODX) extensions into the firmware. I opted for the Synology DS1513+ which is their 5-caddy series. I decided that I probably didn’t need the next level up to gain capacity or performance. For me the critical thing is performance, as I hate having to wait around for template clones to complete. My previous home lab used an IOMEGA NAS device using SATA drives. This was an earlier edition that didn’t support VAAI, because VAAI wasn’t even around then (I think). So it’s perhaps unfair to compare the two devices. The critical thing for me having walked away from the colo where enjoyed access to both NetApp and Equallogic (both SAS/SATA) was not to feel I was compromised on the storage performance front. So I took the radical decision to buy all SSD drives for the entire array, as well as paying for a 2GB RAM upgrade to the onboard cache. I pretty sure the only bottleneck in the Synology is the CPU or the maybe the network when I carry non-VAAI enabled tasks (like copying ISOs around…). By going down this route I hoped to remove any bottleneck from a storage perspective in the new homelab.

Continue reading

Category: BackToBasics | Comments Off on Return of the HomeLab (Part 3)
February 7

Back To Basics – Configuring Storage: PowerCLI (Part 6 of 6)

Well, this the end of my 6 part series on storage and vSphere 5.5. There’s is a part that’s “missing” all about “Vendor Storage Plug-ins”. I did some recent work with a vendor before I left the colo… but their software is still under beta at the moment… So there will a be a lost part 7 to release when the embargo ends…

Screen Shot 2013-10-21 at 14.01.16.png

Rescanning Storage

A simple one-liner PowerCLI command is all that is required to rescan many hosts storage configuration – care must be taken not to rescan every single host in vCenter, and limit rescans to clusters of ESX hosts

Get-Cluster -name “NYC-Gold01” | Get-VMhost | Get-VMHostStorage -RescanAllHBA

Configuring iSCSI Storage

This is script uses a .CSV file retrieve the FQDN/Hostname values.

Screen Shot 2013-12-18 at 17.59.12.png

The first block of PowerCLI enables the iSCSI Software Adapter together with a valid IQN, Target IP, Chapname & Password. The second block uses the ESXCLI namespace within PowerShell to set the VMKernel Port Bindings for iSCSI Load-balancing. Finally, a rescan is undertaken to find new storage and VMFS volumes. This script assumes the appropriate networking is in place.

Continue reading

Category: BackToBasics | Comments Off on Back To Basics – Configuring Storage: PowerCLI (Part 6 of 6)