Well, I’ve pretty quiet on the blogging front I guess since VMworld. In case you don’t know I start a new role in May, 2018 and I’ve really been heads down on that trying to wrap my head around a whole series of new processes and procedures – and as ever learning by one’s mistakes. As ever in the life the lesson you learn the hard way are the ones that tend to stay.

So in May of last year I joined a company called sureskills.com. My relationship with SureSkills went back to the days when I was freelance VMware Certified Instructor (VCI) and I was pimping my training skills across Europe, whilst writing books in hotels rooms, and speaking at VMUG. SureSkills was a big customer of mine – and in the early days I’d be there at least 1-2 weeks every month. The role I have at SureSkills is one that gets me involved in course development with a primary focus on VMware Official Curriculum materials. My first two courses that I worked with colleagues on was “VMware Sales Professional” (VSP) course covering VMware’s EUC/Mobility story focused on WorkspaceONE, and the second was a “VMware Technical Sales Professional” (VTSP) on Network Virtualisation – with specific focus on NSX Data Center (both the “T” and “V” editions). That just came to close this week. It quite something to hear one’s “script” voiced-over by American VO artist!

My new project is more familiar ground. This is an instructor-led classroom based course initially focused on the “Academy Program” and its actually falling under VMware’s “Virtualize Africa” initiative. Expressed simply “Virtualize Africa” is kind “leave no-one behind” project that aims to extend the benefits of VMware technology to every part of the globe. The course I developing with the input of VMware staffers is focused on supporting business-critical apps (Microsoft, Oracle, SAP) as well as nodding toward the next-generation of application frameworks such as containers, kubernetes – as well as vSphere Integrated Containers and VMware PKS. It’s a really exciting project because much of this content has never to this day been available in a training course – and usually resided in best practise docs, VMworld Sessions and VROOM! Style blog posts. Although, the remit is focused on the Virtualize Africa project – much of this content should be of equal interests to other geos. Anyway, I hope to share my insights and learnings now that I’m getting back into the weeds of technical material.

I recently decided to switch back to using USB Media for booting my VMware ESXi hosts. My main thought was that I want to use the local HHD for either some kind of VSA testing (such as StorMagic’s SvSAN) or else when I have the budget buy in SSDs to make a physical VSAN cluster, rather being dependent and reliant on a virtual nested VSAN setup. I went out and got some 32GB USB media to make sure they would be big enough to create the scratch partition.

I’m often wiping my VMware ESXi hosts to try out new builds – all to roll-backwards to previous builds for testing purposes. For that I used the UDA together with some kickstart scripts to do the bulk of the customisation. So times I just lay down a clean build with no customisation and use PowersHell scripting to build up the environment to the level I want it. I have that in a modular way that allows me to lay down, some but not all, of the configuration depending on my development needs.

Anyway, to script an install to USB media you can use this in the kickstart:

install --firstdisk=usb --novmfsondisk

The –firstdisk flag actually can take a series of arguments separated with a comma. It is possible to specify an order of particular disks type (not just the first USB, HHD/SSD or SAN based LUN) the installer searches for. And you can use to this set your own orders for how disks are discovered. For instance –firstdisk=usb,remote,local would first try to install to USB, then a LUN on a SAN, before lastly trying the local disk.

Also what’s supported is specifying the model of the disk like so –firstdisk=ST3120814A. I used this in the past with hyper-converged appliances that I was doing ground-zero resets on – and just so happened the bootdisk I was using had a unique model number. Of course this isn’t very helpful when a server contains disks that all have the same model number…

The other option is to use instead of install –firstdisk, but use install –disk=mpx.vmhba1:C0:T0:L0. This allows you to indicate the disk by which adapter its configured for, and the controller/target used (note these values are often 0, just the L number value that changes. For many years we have referred to this as the vmhba syntax or “Runtime Name”. In the past its been hard to trust 100% these values, but I think they are more reliable nowadays, as the way the VMware ESXi host boots the vmkernel is VERY different now.

Probably the easiest way to find the MODEL name/number and vmhba syntax is using the esxcli commands like so:

esxcli storage core path list

Here’s a cut and paste of the USB output – 

usb.vmhba32-usb.0:0-t10.SanDisk00Ultra00000000000000000000004C530001270222101242 UID: usb.vmhba32-usb.0:0-t10.SanDisk00Ultra00000000000000000000004C530001270222101242 

Runtime Name: vmhba32:C0:T0:L0 

Device: t10.SanDisk00Ultra00000000000000000000004C530001270222101242 

Device Display Name: Local USB Direct-Access (t10.SanDisk00Ultra00000000000000000000004C530001270222101242) 

Adapter: vmhba32 Channel: 0 Target: 0 LUN: 0 Plugin: 

NMP State: active Transport: usb 

Adapter Identifier: usb.vmhba32 

Target Identifier: usb.0:0 

Adapter Transport Details: Unavailable or path is unclaimed 

Target Transport Details: Unavailable or path is unclaimed Maximum IO Size: 32768

I imagined a system with multiple UBS media (!??!) would report C0:T0:L1 or else C0:T1:L1.

Oddly enough I found a second USB device added to my ESXi host doesn’t appear – not even when doing a manual install with ISO attached the HP ILO. I’m not sure why this is – but it may (or may not) be significant which USB slot the device gets inserted too, or that there are limits around the way the VMkernel enumerates USB device – perhaps only enumerating the first USB memory stick found?  This isn’t terribly important – but I will ask around my contacts and see what I can dig up. It’s a bit obscure, but I’m curious like that.

I’m running VSAN in a nested configuration, and I generally shutdown my homelab in the evening each day. I grew tired of doing the manual process of maintenance mode for these nested nodes before shutting them down, and then shutting down the host that they run on. I did a bit of googling for the PowersHell that does that – and was see quite complicated scripts. I’m pleased I went back to the documentation for PowerCLI which is the primary source for the cmdlets:

https://code.vmware.com/doc/preview?id=6702#/doc/Set-VMHost.html

I discovered that the Set-VMhost cmdlet supports a -VsanDataMigrationMode switch with three different options for:

• Full
• EnsureAccessibility
• NoDataMigration

1..4 | Foreach {
 $Num = "{0:00}" -f $_
 Set-VMHost -VMHost esx"$Num"nj.corp.local -State "Maintenance" -VsanDataMigrationMode NoDataMigration 
 }

1..4 | Foreach {
 $Num = "{0:00}" -f $_
 Stop-VMHost esx"$Num"nj.corp.local -Confirm:$False
 }

Nesting VMware ESXi has become easier and easier as the years roll by. In case you don’t know “nesting” is the term used for running VMware ESXi inside a VM for development purposes. It’s the basis of VMware’s popular “Hands-on-Lab”, as well as homelabs – and in recent years “nesting” has gone to the Cloud with service like Ravello on the Oracle Infrastructure Cloud (OIC).

In years gone by there were many settings needed at the Physical and Virtual layer – that required hand-edits to configuration files to make this work. With the advent of vSphere 6.5 U1 a lot of that disappeared. HOWEVER, there’s still plenty of work that needs to be done at the physical and virtual layer to allow for networking to work properly. As well as successfully passomh some of the “health checks” around technologies like VMware VSAN that has specific networking requirements for your web-client to light up with little green ticks of happiness.

Here’s a brief check list:

1. Physical Switch needs:
   • MTU of 9000,
   • with VLAN Tagging enabled as necessary
2. Physical ESXi host vSwitch needs
   • MTU of 9000
   • vSwitch security policy of Accept, Accept, Accept
   • Portgroups used by the virtual “nested ESX” enabled for 4096 to allow for VLAN tagging in the nested layer to pass-thru to the physical switch
3. Virtual Nested ESXi vSwitch needs
   • MTU of 9000
   • vSwitch security policy of Accept, Accept, Accept
   • Portgroups for VMs and VMkernel enabled for VLAN Tagging as necessary

Standard Switches work well with nesting, and have the added benefit of not being tied to a vCenter to the host. This makes blowing away the nested layer after your lab period ends a breeze. It’s slower and more awkward “clean-up” process if your using DvSwitches. To use the special MacLearn VIBs that improve network performance DvSwitches are needed at the physical layer – this isn’t an option if you main physical ESXi host is stand-alone and not managed vCenter.

1. Physical Switch Needs:

In my case I have a HP ProCurve 1810G – 24 G switch – its not a bad unit, whisper quiet and simple to configure:

I have simple VLAN configuration (mainly used to demonstrate/explain the VLAN Tagging concept) but I also VLAN off my VMotion traffic.

Physical ESXi host vSwitch needs

I set my MTU and Security policy on the properties of vSwitch0, which means all the portgroups inherit those settings. Its simple quick and easy.

To do that in the context of the VMware ESXi VMKernel you could use the ESXCLI command – and these commands could be part of a kickstart install script:

## - Enabling Jumbo Frames on vSwitch0 to pass VSAN "Configuration Assistant" tests.
esxcli network vswitch standard set -m 9000 -v vSwitch0

## - Lower security on vSwitch0 to allow traffic to flow in a nested environment. 
esxcli network vswitch standard policy security set -v vSwitch0 -f=true -m=true -p=true

The VLAN configuration can be set when the portgroup is being created as is the case with VMotion being on VLAN10 like so:

esxcfg-vswitch -A "VMotion" vSwitch0
esxcfg-vswitch -v 10 -p "VMotion" vSwitch0
esxcfg-vmknic -a "VMotion" -i [VMOT_IP] -n 255.255.0.0 -p "VMotion" vSwitch0
esxcli network ip interface tag add -i vmk1 -t VMotion

Note: In this case [VMOT_IP] is variable used as part of the UDA appliance.

For an existing portgroup for example the “VM Network” the VLAN value can be set to be one that passes thru the VLAN Tagging from the nested layer to the physical layer:

esxcli network vswitch standard portgroup set -p "VM Network" --vlan-id 4095

Alternatively, if your Physical ESXi hosts are managed by vCenter – you could do this with a PowersHell for each loop like so:

1..3 | Foreach {
 $Num = "{0:00}" -f $_
 $vswitch0 = Get-VirtualSwitch -VMHost esx"$Num"nyc.corp.local -Name vSwitch0
 Set-VirtualSwitch -VirtualSwitch $vswitch0 -MTU 9000 -Confirm:$false
 }

1..3 | Foreach {
 $Num = "{0:00}" -f $_
 $vswitch0 = Get-VirtualSwitch-VMHost esx"$Num"nyc.corp.local -Name vSwitch0
 Get-VirtualSwitch -Name $vSwitch0 | Get-SecurityPolicy | Set-SecurityPolicy -MacChanges $true
 Get-VirtualSwitch -Name $vSwitch0 | Get-SecurityPolicy | Set-SecurityPolicy -ForgedTransmitsInherited $true
 Get-VirtualSwitch -Name $vSwitch0 | Get-SecurityPolicy | Set-SecurityPolicy -AllowPromiscuous $true
 }

The VLAN configuration for an existing portgroup can be updated using the following:

1..3 | Foreach {
 $Num = "{0:00}" -f $_
 $vswitch0 = Get-VirtualSwitch -VMHost esx"$Num"nyc.corp.local -Name vSwitch0
 $VMNetworkPG = Get-VirtualPortGroup -VirtualSwitch $vSwitch0 -Name "VM Network"
 Set-VirtualPortGroup -VirtualPortGroup $VMNetworkPG -VLanId 4095
 }

Virtual Nested ESXi vSwitch needs

The setting on vSwitch0 on the Nested ESXi host mirror that of the physical… so everything is aligned from vESX>pESX>pSwitch everything is MTU 9000 with weakened security. The VLANs I have from 101-104 could be added at the physical and virtual level with the esxcli command like so:

esxcli network vswitch standard portgroup add --portgroup-name=VLAN101 --vswitch-name=vSwitch0

esxcli network vswitch standard portgroup set -p VLAN101 --vlan-id 101

or if you were wanting to do this via PowersHell, you could use the following method. This approach differs from my previous examples – it merely “gets” every ESXi host in a vCenter, and creates VLAN101, 102, 103, and VLAN 104.

101..104 | Foreach { $Num = $_ 

(Get-VMHost | sort-object name) | foreach 

{New-VirtualPortGroup -VirtualSwitch (Get-VirtualSwitch -Name vSwitch0 -VMHost $_) -Name VLAN$num -VLanId $num } 

}

A very common use of nested vSphere is to create a virtual/nested vSphere VSAN cluster – this is because not everyone can afford the servers and storage to build out VSAN on a physical level (3 hosts with 1xSDD, 1xHDD). For sometime its been possible to virtualise ESXi, as well as also marking specific VMDKs as either being HHD or SSD. Inside the nested ESXi environment you will need a VMKernel portgroup enabled for VSAN. A simple thing to do would be to enable either the Management Network or the VMotion network for dual usage. Alternatively, you could setup a net-new VMKernel portgroup who’s sole and only purpose is VSAN communications. That’s what I do (even if the traffic goes over the EXACT same nics – as it makes it clear what the usage of each portgroup/vmkernel port is.

You can do this with CLI using the following commands like so:

esxcfg-vswitch -A "VSAN" vSwitch0
esxcfg-vswitch -v 0 -p "VSAN" vSwitch0
esxcfg-vmknic -a "VSAN" -i [VSAN_IP] -n 255.255.255.0 -m 9000 -p "VSAN" vSwitch0
esxcli vsan network ipv4 add -i vmk2

Note: In my case [VSAN_IP] is a variable used in my kickstart sub-template as part of the UDA. The main thing is to know the number of vmk ports – vmk0 is always the default management port – in my script the next vmk port created is for VMotion, and that therefore make the VSAN port be vmk2.

or with PowersHell:

1..3 | Foreach {
 $Num = "{0:00}" -f $_
 $vswitch0 = Get-VirtualSwitch -VMHost esx"$Num"nj.corp.local -Name vSwitch0
 New-VMHostNetworkAdapter -VMHost esx"$Num"nyc.corp.local -VirtualSwitch $vswitch0 -PortGroup VSAN -IP 10.20.33.1"$Num" -SubnetMask 255.255.255.0 -VsanTrafficEnabled $true
 }

https://blogs.vmware.com/vsphere/2018/05/vsphere-6-5-update-2-now-available.html

DISCLAIMER: Firstly, and most importantly. This issue likely to effect a tiny proportion of customers. Despite what many think as soon as new version of vSphere hits the streets for most large organisations it can take anything between 12-24 months to actually complete an upgrade from one flavour of vSphere to another. Therefore by the time a customer who has rolled out vSphere 6.5 U2, the chances are that there WILL be an upgrade path to vSphere 6.7 probably in the shape of U1 or U2 release. The important thing to remember is if your intention is in the short-term to upgrade to vSphere 6.7 for whatever reason – is to be aware of the title of this blogpost. You may as well skip U2 and head straight to vSphere 6.7 if that’s the case…

UPDATE (10th May, 2018): It looks like the VAMI interface on the VCSA is set to deploy vSphere 6.5 U2. It’s set as just a “bugfix” and that its severity is set to “critical”. I’m research what the situation is with update manager.

WHY DOES THIS HAPPEN: This has happened on more than one occasion in recent memory. I don’t personally think that precedent can be used to justify this. A much simpler reason exists. The reality is VMware now has a very complicated, and richly interdependent and tightly-coupled series of software and services. Attempts have been made in the past to build a “Train Release” by which the core platform like vCenter/ESX are the “head” or “engine” of the train, and the related products that sit on top of it – are couple behind it on a release schedule that if correctly managed should arrive in the customer station at roughly the same time. Occasionally, a carriage gets de-coupled and is left in the sidings somewhere outside Colchester. For customers for whom that software package is critical it can and will delay and upgrade processes until the last part of the train gets into the station. This can be difficult where one SR calls for upgrade to fix one problem, only to “break” a piece of software elsewhere.

This kind of sequencing of software releases is very, very difficult to do in a large multi-national software company where software is shrinked-wrapped and installed on-prem. It’s actually a compelling argument for buying software like vSphere-as-a-service, and heading off into the Public Cloud. You cease to have worry about this kind of poop.

HAS THIS BEEN COMMUNICATED EFFECTIVELY. I’m sorry to say this but no. Sadly, an almost like Apple like cloak of secrecy still envelopes VMware. There are good reasons for this in some cases, but other situations like this – it actively and effectively works against good communication. As ordinary mortal I am supposed to read all the blogs, read all the release notes and listen to all the podcasts. I would say I’m pretty well connected to the VMWorld – but I didn’t know this was happening. Heck, I didn’t know that vSphere 6.5 U2 was on its way – or know there wasn’t upgrade path.

Apparently, to some that’s my fault, and my problem – and it isn’t the responsibility of VMware to shout from the rooftops as if Four Horses of the Apocalypse were on their way. Before I rant on and loose the plot a little – I will point you back to this disclaimer. Look. I totally get it, and totally understand why and how this happens. And in many, many, many ways the vSphere 6.5 U2 release is a good, good,good thing because it quickly deals with the issue associated with “WHY DOES THIS HAPPEN”. The train becoming de-railed from the railway lines.

For those who are outside of the vendorland much of this just seems bug-eyed weird. Although “shit-sandwich” has become such a defacto standard in our industry many seem quiet happy to suck it up. Personally, I like to set my standards higher than that. Case in point. The download page for vSphere 6.5 U2 has one of those little yellow “read the KB” stickers – which incidentally disappears when you click “Go To Downloads”.

The downloads page of vSphere 6.7 (at the time of writing) has a tiny notification at all.

People will say “buyer beware” and you should read the Release Notes. Honestly, who has time for that – Release Notes are like EULAs. Practically no-one reads them (I’m unable to quantify that statement I know!).

Of course, if you fail to read all the blogs, the release notes and these stickers – then its “your own fault chum” according to some. Personally, I think vendors have to realise that in todays busy world with lots of competing channels searching for our eyeball time – something a little more noticeable is required.

With that. Read my disclaimer again…