A selfish [virtualized] look at 2016-2017

(image credit to Ken Horn, https://twitter.com/kdmhorn)

So the last year and a half has been wild.

 

In 2016, I left my employer and pursued a new opportunity with a data protection company called Zerto, which is where I reside and work today as a solutions engineer. Zerto provides an enterprise-class solution for business continuity, disaster recovery, and application mobility, and as someone that has been doing data protection for about a decade, it’s very refreshing to work on a platform that a.) works well and b.) gets people excited about data protection, a traditionally dry topic.

I can get into Zerto and what it does in detail in later posts, but I want to reflect on the opportunities that this platform has given me, especially as it revolves around virtualization.

First off, my new employer has given me a huge opportunity to get into the inner workings of ESXi and vSphere as a whole. As a technologist, I’m here to discover new and interesting ways to solve problems, and vSphere is a rich platform to develop creative solutions with. I hold current VCP5 and VCP6 certifications but didn’t constantly touch many of the things that VCP gets into. With Zerto, I have had the opportunity to explore production installs of VSAN, get into the weeds of the vSphere API, and truly understand what those esoteric VMX files are really telling us about a VM!

Secondly, I have had the pleasure of meeting a ton of folks in the virtualization world. Fellow geeks and tinkerers that have the same goal of solving problems and doing cool stuff to further their missions are always fun to talk to. Jonathan Frappier (@jfrappier) and Mike Marseglia (@mmarsreally gave me the push I needed to get into the VMware and virtualization community, and from there I started networking with the Providence VMUG crowd as well as the Boston VMUG UserCon regulars.

Third, I now have a paper trail of presentations that I have given to the VMUGs at Rochester, Boston (Twice!), Hartford, and Providence. The great thing about VMUGs is that they are opportunities not just to talk about your company or product but to really connect regarding virtualization and the challenges and problems that you can solve with them. For example, my presentation at the Providence VMUG was a “back-to-basics” discussion regarding having a plan surrounding disaster recovery. Far too many organizations rely on the “throw your hands up and panic” mode of disaster recovery, and I discussed how to incorporate vSphere and Zerto into a higher-level holistic level of DR planning.

So here’s where and when I presented at VMUGs in 2016 and 2017:

So, with that said, where do we go from here? Public cloud integration. vSphere on AWS. 2016 and the first half of 2017 were pretty darn good… let’s see how the next year and a half shapes up!!

3ab7f81c-9ec3-4641-911c-cc6eafd47daf-original

-ajs

A horrible but functional way to shut down a home lab for the evening

stackoverflow

Warning: the following barf– er, bash script might make you ill.

Over the last couple of days I’ve been trying to figure out ways to improve my home lab experience, including automating the tedious shutdown procedure. Most ESXi environments are 24/7, but of course with a home lab there is typically no such need. The trick is to power off the guests gracefully with minimal effort. Enter: scripting!

I currently have five VMs deployed in my lab: two are powered off, one is an embedded ESXi host (that currently runs zero guests), one is a vCenter Server Appliance, and one is the vSphere Management Assistant. The vMA is where all the scripting magic occurs.

The script that I have come up with is admittedly very bad.

#!/bin/bash
# Shutdown script for home lab
#set -x

IFS=""
OUTPUT=( $(esxcli --sessionfile ~/tali01-session vm process list | grep "/vmfs" | cut -c17-256) )
OUTPUTARR=()

while read -r line; do
    OUTPUTARR=("${OUTPUTARR[@]}" "$line")
done <<< "$OUTPUT"

while true; do
    read -p "Are you sure you want to shut down the lab (Y/N)? " yn
    case $yn in
        [Yy]* )
            echo "Shutting down vSphere lab..."
            for ((i = 0; i < ${#OUTPUTARR[@]}; i++))
            do
                echo ${OUTPUTARR[$i]}
                vmware-cmd --sessionfile ~/tali01-session ${OUTPUTARR[$i]} getstate
                vmware-cmd --sessionfile ~/tali01-session ${OUTPUTARR[$i]} stop soft
                vmware-cmd --sessionfile ~/tali01-session ${OUTPUTARR[$i]} getstate
                sleep 60
            done; exit;;
        [Nn]* )
            echo "Exiting."
            exit;;
        * ) echo "Please answer yes or no.";;
    esac
done

What the script does is this: it invokes the special IFS variable in bash so that it doesn’t choke on whitespace, then it run grabs the output of esxcli. This is where it starts getting awful. –sessionfile is something you can use to authenticate to an ESXi host without having to repeatedly type in a username and password. Here’s the problem: it expires if you don’t use the file within 30 minutes. Already we encounter problem one: you have to generate a new sessionfile every time you want to use this script by way of using –savesessionfile.

Problem number two immediately follows. We’re grepping for config files that we can use in order to power down VMs. I use cut in order to choose the exact part of the string that I want to follow. This is a horrendous way of grabbing a part of a string, because if the output ever changes in how esxcli vm process list works, you’ll get an erroneous cut back. Furthermore, notice that I cut up to character 256… what if our vmx file goes well beyond that by even a single character?

Up next: reading a multi-line output into an array. I banged my head against this part for the last two days. It should be so simple, and ultimately it was a modification of the ubiquitous “x = x + 1” or “x++” method. Take an empty but initialized array, add a single read line, and append it to the array. Repeat until done.

Next up: the actual shutdown. Prompt the user to make he or she actually wants to shut down the lab. If affirmative, start walking through each config file stored in the array and softly power off the guest, waiting 60 seconds before looping back to the beginning. Repeat for each system.

That’s it. The concept is simple enough but there are a few other reasons why the script sucks, such as the fact that it shuts down the very guest it exists on, and there’s no intelligence behind what it shuts down first; it so happens to collect the config files in the order that I want them to shut down but that is horribly hacky and not scalable whatsoever.

I had a good friend of mine (whom is a bazillion times the programmer that I am) look the script over and said that “you told bash to stop being bash”. I thought it was funny.

So, where do we go from here?

  1. Figure out a way to prioritize the shutdown of certain VMs. I have barely scratched the surface of the commands available on vMA, so there’s got to be a way to tag or otherwise identify VMs for position in a shutdown list.
  2. Port this to PowerCLI or figure out some way to make it so that the very VM it runs on isn’t shut down when executing the script.
  3. Come up with options, such as “fast” shutdown where the sleep time is 30 seconds or less, “hard” shutdown where we want everything to go down as quickly as possible (ie yank the power cord), and so on.
  4. The authentication method with the ESXi server is downright miserable. Use something like –credstore instead to make the authentication persistent.

If you have any suggestions that focus on either features or scripting, let me know!

-ajs

esxcli troubles – order matters!

As part of my efforts to renew my VCP5-DCV certification to the new and shiny VCP6-DCV (not to mention avoiding having to take another week-long class in order to qualify to do so), I have been spinning up a lab environment to mess around with. Like when I was studying for VCP5-DCV, I am using Mastering VMware vSphere 6 by Nick Marshall as my go-to guide.

In my lab, I deployed an instance of the vSphere Management Assistant (vMA) so that I can run vSphere CLI commands such as esxcli. The deployment of vMA went without any problems, but I was having a bunch of trouble getting it to authenticate with vCenter.

esxcli –server=192.168.1.85 –vihost=192.168.1.249 -username=administrator@vlab.local network ip interface list

(It may whine about not having a certificate in the credential store… if so, add the certificate.

/usr/lib/vmware-vcli/apps/general/credstore_admin.pl add -s vcenter.ip.address.here -t [certificate])

Then I started getting a weird authentication error…

authenticationproblem1

That’s weird. I tried the same password multiple times just to make sure I wasn’t typing it in wrong. No dice. I started looking online and found this thread:

https://community.spiceworks.com/topic/395441-vmware-cannot-complete-login-due-to-an-incorrect-username-or-password

Basically, this guy’s fix was to disconnect the host and reconnect it in vCenter. I tried that. It still failed.

Finally, I found the answer. https://pubs.vmware.com/vsphere-50/index.jsp?topic=%2Fcom.vmware.vcli.getstart.doc_50%2Fcli_jumpstart.3.4.html

After installation, run ESXCLI commands against a specific host by first specifying all dispatcher options. If the target server is a vCenter Server system, specify the target ESXi host before any ESXCLI namespaces, commands, and supported options.

It was purely a matter of what options in the command come first. I ran the command again like this:

authenticationproblem2

 

Tada! Order DOES matter to esxcli. The book example will not work as formatted.

Up next: scripting the nightly shutdown of the lab.

 

-ajs