Tobias Kreidl | Citrix XenServer

Posts by: "Tobias Kreidl"

Introduction to Saving VM Parameters: How I Metadata Backup

By Tobias Kreidl On 5 października 2017 · Add Comment

XenServer Virtual Machines (VMs) certainly need no introduction, but even if you do not pardon the pun above, they still contain a lot of specialized and individualized information about their sizes, network connections, and myriad other settings that are generally not readily exposed, yet are integral to the operation and functionality of each VM. This blog entry is not intended to take a deep dive into the several hundred parameters that are defined for VMs, but rather just talk a bit about how to save, extract and potentially restore VM information based on them.

VM Metadata Backups

The purpose of backing up the metadata from a VM is to help you understand how it’s configured without having to search through the list of parameters accessible via various “xe” commands or XenAPI calls that require some programming efforts, plus also allow you to potentially track changes to your VMs without necessitating a full VM export/backup each time. You may not want or need to restore a VM from a full backup, but rather just revert a few parameters back to older values. You might also want to monitor what sorts of changes have taken place over time and equate those to performance or other metrics. In short, a number of reasons to maintain relatively frequent metadata backups of VMs can be justified.

Getting VM Metadata

There are a number of ways to obtain VM metadata settings. One of these is with the standard “xe” command to extract parameters, either individually, in a comma-separated string of multiple queries, or all of them such as with:

# xe vm-list uuid|name-label=UUID|NAME-LABEL params=all

Here, either the UUID or name-label can be used to select the VM.

Some parameters can also not only be obtained via “xe vm-param-get” but also changed, using the complementary “xe vm-param-set” operator. This gives you access to modifying around 30 parameters and reading over 80 of them.

The XenCenter console, xsconsole, provides a direct way to back up and restore VM metadata. From the "Backup, Restore and Update" menu, you can next navigate to the "Backup Virtual Machine Metadata" option and choose from available SRs onto which you wish to create a metadata backup of all available VMs. Note, though, that it will only evidently create metadata backups of running VMs! Likewise, the restore operation can only be performed on various subsets of VMs and not on an individual VM. Conversely, the metadata restore operation apparently can be applied to both running and halted VMs, but again cannot be performed on an individual VM.

Another option is to make use of the XenAPI library and extract tokens using XenAPI calls and access them for example using constructs such as these:

vm_meta_status = gather_vm_meta(vm_object, full_backup_dir)

vm_record = session.xenapi.VM.get_record(vm_object)

vm_out = open ('%s/vm.cfg' % tmp_full_backup_dir, 'w')

vm_out.write('name_label=%s\n' % vm_record['name_label'])

vm_out.write('name_description=%s\n' % vm_record['name_description'])

vm_out.write('memory_dynamic_max=%s\n' % vm_record['memory_dynamic_max'])

vm_out.write('VCPUs_max=%s\n' % vm_record['VCPUs_max'])

vm_out.write('VCPUs_at_startup=%s\n' % vm_record['VCPUs_at_startup'])

This can be time-consuming if you want to keep identifying and modifying code to deal with any additions or changes, plus you may periodically have to update your API libraries.

Yet another option is to make use of the not-well-documented features within the “xe” command set associated with vm-export and vm-import utilities. It is possible to export just the metadata from a VM using the following syntax:

# xe vm-export metadata=true uuid=UUID-OF-VM) filename=/full_path/OUTPUT_FILE.XVA

This will create what is in essence a tar file containing a single file that captures more than 300 parameters! The XML code has to be extracted from this tarball, which contains a single file that is always named ova.xml and can be pulled from the XVA files with a basic tar command in which specifying the file to extract as ova.xml is optional, since it’s the one and only file within the tar file:

# tar –xf OUTPUT_FULE.XVA

tar: ova.xml: implausibly old time stamp 1969-12-31 17:00:00

Note that you may get this rather interesting message regarding the timestamp, which can be ignored. It may also turn out that the output file has absolutely no access permissions set, so you may want to run a “chmod 600 ova.xml” (or 644, etc.) to make it readable. You may also wish to rename it so it’s unambiguous and/or less likely to be overwritten.

For exported XVA files that are gzipped, you can extract the ova.xml file in a single operation with:

# tar xzf OUTPUT_FULE.XVA .gz

tar: ova.xml: implausibly old time stamp 1969-12-31 17:00:00

Once extracted, let’s take a look at the first part of the ova.xml file, which takes on a rather “ugly” appearance:

There is a wealth of information in here, but it’s not in a very friendly format. Fortunately, this can be readily rectified with the handy xmllint utility already present on XenServer (at least on 7.X):

# tar -xOf /ubuntu12-xs66-specialchars.XVA | xmllint –format - >/tmp/output_VM.xml

The ”-O” flag causes the output to be redirected to stdout and hence it can be piped to the xmllint utility, which in turn can generate a very nicely formatted and properly indented XML file. Note that the “-“ before the redirection “>” operator signifies the output of xmllint to go to stdout and if desired, the command can be abbreviated as such, in which case the output will just appear on the terminal. It will be several hundred lines long, so you may as well redirect the output into a file where you can more conveniently deal with reviewing that amount of information.

Here is what the first part of the formatted file really looks like:

OK, Great -- Now What?

Given the ability to now parse and peruse the XML metadata file associated with a particular VM, one could contemplate creating periodic backups of the VM metadata to have on hand in case one needs to reconstruct something or check if anything had changed. That’s all fine and good, but other than using “xe” commands or other means to change individual parameters, how does having these data help in the event of wanting to reconstruct or restore a VM?

The bottom line is that this feature has limited direct applications, though it does have a few. Consider the case of trying to use an XVA file that only contains the VM’s metadata to restore a VM. Note that the original VM must of course still exist or there will be nothing present to associate the VM storage with if a version of this VM cannot be found. However, if it is present, consider the following results:

# xe vm-import preserve=true filename=/ubuntu12-xs66-specialchars.XVA

The VM cannot be imported unforced because it is either the same version or an older version of an existing VM.

vm: 9538882a-c7e4-b8e5-c1f9-0d136f4a81b1 (TST-ubuntu12-vmtst3-xs66)

existing_version: 0

version_to_import: 0

Perhaps as anticipated, it will fail as the “preserve=true” flag will first check for a duplicate VM and upon discovering it, flags it as a command that would overwrite the existing VM. That’s a good thing. Leaving off that flag, we next try:

xe vm-import filename=/ubuntu12-xs66-specialchars.XVA metadata=true

This should yield success, but what kind of success? What happens is the VM created using the “metadata=true” flag produces a new VM copy with the same name, but a different UUID for the VM that is “Created by template provisioner.” What has happened is just that a fast clone has been created. You will see that if you delete such a VM created that way, it will not show any storage devices associated with it and in XenCenter, it will therefore not ask you if you want to delete the associated storage.

This is not entirely without use, however as you can still make use of this VM and perhaps even compare its characteristics to the original. Furthermore, you can export the VM, and import it as a new VM in which case it will gain the properties of a full clone. At that point, dependence on the original no longer exists.

This exercise might be useful in debugging or checking parameter-based performance or other issues between the original and subsequent metadata modifications. Such headers may also be useful just for tracking historical uses of VMs, checking to see what IP addresses may have been assigned, and numerous other things.

The Full XVA Export

The discussion up to this point should result in a mental lightbulb turning on and raising the question, well, if I restore a full export of a VM, isn’t all this information already in there? Since clearly it has to be for a vm-import to work properly, an examination of a full XVA export will indeed reveal that it consists of many fairly small files, numbering at times many thousands, but always starting with our old friend, ova.xml, as we see from this sample output that lists the contents instead of extracting it:

# tar -tvf /exports/test-export.xva |less

---------- 0/0 29935 1969-12-31 17:00 ova.xml

---------- 0/0 1048576 1969-12-31 17:00 Ref:367/00000000

---------- 0/0 40 1969-12-31 17:00 Ref:367/00000000.checksum

---------- 0/0 1048576 1969-12-31 17:00 Ref:367/00000001

---------- 0/0 40 1969-12-31 17:00 Ref:367/00000001.checksum

---------- 0/0 1048576 1969-12-31 17:00 Ref:367/00000002

---------- 0/0 40 1969-12-31 17:00 Ref:367/00000002.checksum

---------- 0/0 1048576 1969-12-31 17:00 Ref:367/00000003

---------- 0/0 40 1969-12-31 17:00 Ref:367/00000003.checksum

---------- 0/0 1048576 1969-12-31 17:00 Ref:367/00000004

---------- 0/0 40 1969-12-31 17:00 Ref:367/00000004.checksum

…

etc.

The nice aspect of having the XVA file self-contained with all the metadata, as well as the data contents of the VM, makes this standalone file easy to move around and utilize as a backup.

The other nice aspect is that you, in fact, can use it as its own metadata storage mechanism and extract only that part of it if desired without needing to create a separate metadata backup (unless, of course, you want to do that more often and independently of a a vm-export). To extract just the metadata from this file shown above, all you need to do is to specify the embedded tar file name:

tar -xvf /exports/test-export.xva ova.xml

We now have the same metadata file content we had when running the vm-export command combined with the “metadata=true” option.

In Summary

First off, it cannot be overstated that your XenServer environment should be backed up frequently and fastidiously, including both the pool metadata as well as the individual metadata for VMs. Even if you already have full exports of your VMs, having additional metadata can be useful for auditing purposes, as well as making it possible to check on parameters that are hard or impossible to glean through other means.

Nobody that I know was ever accused of creating too many backups.

Czytaj dalej →

XenServer – Octoblu – SNMP Integration: The Sum of the Parts

By Tobias Kreidl On 5 października 2017 · Add Comment

Czytaj dalej →

Changes to Hotfixes and Maintenance for XenServer Releases

By Tobias Kreidl On 14 września 2017 · Add Comment

XenServer (XS) is changing as a product with more aggressive release cycles as well as the recent introduction of both Current Release (CR) and Long Term Service Release (LTSR) options. Citrix recently announced changes to the way XenServer updates (hotfixes) will be regulated and this article summarizes the way these will be dealt with looking forward.

To first define a few terms and clarify some points, the CR model is intended for those who always want to install the latest and greatest versions, and these are being targeted for roughly quarterly releases. The LTSR model is primarily intended for installations that are required to run with a fixed release. These are by definition required to be under a service contract in order to gain access to periodic updates, known as Cumulative Updates or CUs. The first of these will be CU1. Since Citrix is targeting support for LTSR versions for ten years, this incurs quite a bit of cost over time and is much longer than a general CR cycle is maintained, hence the requirement for active support. The support for those running XS as part of running licensed versions of XenApp/XenDesktop will remain the same, as these CR licenses are provided as part of the XenApp/XenDesktop purchase. There will still be a free version of XenServer and the only change at this point will be that on-demand hotfixes for a given CR will no longer be available once the next CR is released. Instead, updates to quarterly releases will be needed to access and stay on top of hotfixes. More will be discussed about this later on. End-of-life timelines should help customers in planning for these changes. Licensed versions of XenServer need to fall under active Customer Success Services (CSS) contracts to be eligible for access to the corresponding types of updates and releases that fall outside of current product cycles.

At a Glance

The summary of how various XS versions are treated looking forward is presented in the following table, which provides an overview at a glance.

XenServer Version	Active Hotfix Cut-Off	Action Required
6.2 & 6.5 SP1	EOL June 2018	None. Hotfixes will continue to be available as before until EOL.
7.0	December 2017	Upgrade or buy CSS.
7.1	December 2017	Upgrade or buy CSS (to stay on the LTSR stream by deploying CU1)
7.2	At release of next CR	Upgrade to latest CR, or buy CSS to obtain 4 months more of hotfixes (and thus skip one CR)

In the future, the only way to get hotfixes for XS 7.2 without paying for a CSS will be to upgrade to the most current CR once released. Customers who do pay for CSS will also be able to access hotfixes on 7.2 for a further four months after the next CR is released. This would in principle allow you to skip a release and transition to the next one beyond it.

What Stays the Same

While changes are fairly significant for especially users of the free edition of XS, some aspects will not change. These include:

· There will still be free versions of XenSever released as part of the quarterly CR cycle.

· The free version of XS can still be updated with all released hotfixes until the next CR becomes generally available.

· XS code still remains open source.

· Any hotfixes that have already been released will remain publically available.

· There will be no change to XS 6.2 through 6.5 SP1. They are only going to get security hotfixes at this point, anyway, until being EOLed in June 2017.

In Summary

First off, stay calm and keep on maintaining XS pretty much as before except for the cut-offs on how long hotfixes will be available for non-paid-for versions. The large base of free XS users should not fear that all access to hotfixes is getting completely cut off. You will still be able to maintain your installation exactly as before, up to the point of a new CR. This will take place roughly every quarter. This is a pretty small amount of extra effort to keep up, especially given how much easier updating has become, in particular for pools. Most users would do well to keep up with current releases to benefit from improvements and to not have to apply huge numbers of hotfixes whenever installing a base version; the CR releases will include all hotfixes accumulated up to that point, which will be more convenient.

What To Do

Any customers who already have active CSS contracts will be unaffected by any of these changes. Included are servers that support actively licensed XenApp/XenDesktop instances, who by the way, get all the XS Enterprise features incorporated into the latest XS releases – for all editions of XA/XD. Customers on XS 7.0 and 7.1 LTSR who have allowed their software maintenance to lapse and have not renewed/upgraded to CCS subscriptions should seriously consider that route.

Citrix will provide additional information as these changes are put into place.

Czytaj dalej →

From the Field: XenServer 7.X Pool Upgrades, Part 2: Updates and Aftermath

By Tobias Kreidl On 5 lipca 2017 · Add Comment

The XenServer upgrade process with XenServer 7 can be a bit more involved than with past upgrades, particularly if upgrading from a version prior to 7.0. In this series I’ll discuss various approaches to upgrading XS pools to 7.X and some of the issues you might encounter during the process based on my experiences in the field.

Part 1 dealt with some of the preliminary actions needed to be taken into consideration and the planning process, as well as the case for a clean installation. You can go back and review it here. Part 2 deals with the alternative, namely a true upgrade procedure, and what issues may arise during and after the procedure.

In-Place Upgrades to XS 7

An in-place upgrade to XS 7.X is a whole different beast compared to where the OS is overwritten from scratch. Mostly, it depends on whether or not you wish to retain the original partition scheme or go with the newer, larger, and more flexible disk partition layout. The bottom line is despite the added work and potential issues, you may as well go through with the update as it will make life easier later down the road.

Your choices here are to retain the current disk partitioning layout or to switch to the newer partition layout. If you want to know why the new XS disk layout changed, review the “Why was the XenServer 7 Disk Layout Changed?” section in Part 1 of this blog series.

In my experience, issues found with in-place upgrades cover five areas – I’m going to cover them as follows:

Pre-XS 7 Upgrade, Staying the Course with the Original Partition Layout
The 72% Solution
The (In)famous Dell Utility Partition
XS 7.X to XS 7.X Maintaining the Current Partition Layout
XS 6.X or7.X to XS 7.X With a Change to the New Partition Layout (plus possible issues on Dell Servers with iDRAC modules)

Pre-XS 7 Upgrade, Staying the Course with the Original Partition Layout

If you choose to stick with the original partition layout, the rest of your installation experience – whether you go with the rolling pool upgrade, or conventional upgrade – will be pretty much the same as before.

As with any such upgrade, make sure to check ahead of time that the pool is ready for this undertaking:

XenCenter has been upgraded
Assure the proper backups of VMs, metadata, etc. have been performed
That all VMs are agile or in the case of a rolling pool upgrade, at least shut down on all local SRs.
All hosts must not only be running the same version of XenServer, but must have the identical list of hotfixes applied.

Depending on the various ages of the hosts and how hotfixes were applied, you could run into the situation where there is a discrepancy. This can be particularly frustrating if an older hotfix has since been superseded and the older version is no longer available, yet shows up in the applied hotfix list. Though I’d only recommend this in dire circumstances (such as this, where the XenServer version is going to be replaced by a new one anyway) there are ways to make XenServers believe they have had the same specific patches applied by manipulating the contents of /var/update/applied directory (see, for example, this forum thread).

One thing you might run into is the following issue, covered in the next section, which incidentally appears to be independent of whichever partition layout you use.

The 72% Solution

This is not really so much of a solution, but rather a warning as to what you may encounter and what to do before you potentially run into this or what to do afterwards should you encounter it.

This issue apparently crops up frequently during the latter part of the installation process and is characterized by a long wait with a screen showing 72% completion of the installation that can last anywhere from about five minutes to over an hour in extreme cases.

One apparent cause of this is if you have an exceedingly large number of message files on your current version of XS. If you have the chance, it would be worthwhile taking the time to examine the contents of the area where these are stored and manually clean up any ancient versions within those areas under /var/lib/xcp/blobs/ in particular as most of these reside under the “messages” and “refs” subdirectories, paying attention also to the symbolic links.

If you do get stuck in the midst of a long-lasting installation you can escape out by pressing ALT+F2 to get into the shell and check the installer logs under /tmp/install-log to see if it has thrown any errors (thanks, Nekta -- @nektasingh -- for that tip!). If all looks OK, continue to wait until the process completes.

The (In)famous Dell Utility Partition

Using the rolling pool upgrade for the first time, I ran into a terrible situation in which the upgrade proceeded all the way to the very end and just as the final reboot was about to happen, I got this error popping up on the console:

Which read:

An unrecoverable error has occurred. The error was:

Failed to install bootloader: installing for i386-pc platform.

/usr/sbin/grub-install: warning: your embedding area is unusually small. core.img won’t fit in it..

/usr/sbin/grub-install: warning: Embedding is not possible. GRUB can only be installed in this setup by using blocklists. However blocklists are UNRELIABLE and their use is discouraged..

/usr/sbin/grub-install: error: will not process with blocklists.

You can imagine the reaction.

What caused this and what can you do about it? After all, this was something I’d never encountered before in any previous XS upgrades.

It turns out that the reason for this is the Dell utility partition (see, for example, this link). The Dell utility partition is a five-block partition that Dell puts on for its own purposes at the beginning of many of its servers as they ship. This did not interfere with any installs up to and including XS 6.5 SP1, hence this came to me as a total surprise when I first encountered in during a XS 7 upgrade.

And while this wasn't an issue initially or in one particular upgrade which for whatever reason managed to squeak by without any errors being reported whatsoever, it's too small to hold the initial configuration needed to do the installation under most circumstances when installing XS 7.

What's bad is that the XenServer installation doesn't apparently perform any pre-checks to see if that first partition on the disk is big enough. This is the case for both UEFI and Legacy/BIOS boots.

The solution was simply to delete that sda1 partition altogether using fdisk and re-install. Deleting the partition can be performed live on the host prior to the installation process.

You can then successfully bypass this issue. I have performed this surgical procedure on a number of hosts with success and have not experienced any adverse effects; you do not even require a vorpal sword to accomplish this task.

Possibly future upgrade processes will be better accommodating and should perform a pre-check for this along with looking at other potential inconsistencies such as the lack of VM agility or uniformity of applied hotfixes.

XS 6.X or 7.X to XS 7.X With a Change to the New Partition Layout

This is the trickiest area where most, including me, seem to encounter issues.

One point that should be clarified right away is:

Any local partition on or partly contained on the system disk to be upgraded is going to get destroyed in the process.

There are no two ways about this. Hence, plan ahead: either

Storage Xenmotion any local VMs to pooled storage, or
Export them.

If you want or need to preserve your existing local storage SRs, you’ll have to stay with the original partition scheme.

Should you decide to update the new partition scheme, before the upgrade, you will need to perform the following action:

# touch /var/preserve/safe2upgrade

The upgrade process (to preserve the pool) will need to be performed using the rolling pool upgrade. The caveat is that if anything goes wrong during any part of the upgrade process, you will have to exit and try to start over. The end point can vary from it being simple to reconvene from where you left off, to having things in a broken state, depending on the circumstances.The pre-checks performed are supposed to catch issues beforehand and allow you to take care of them before launching into the upgrade process, but these do not always trap everything! Above all, make sure that:

All VMs are agile, i.e. that there are no VMs running or even resident on any local storage
High Availability (HA) and Workload Balancing (WLB) are disabled
You have created metadata backups to at least one pooled storage device or exported it to an external location
You have plenty of space to hold VMs on whatever hosts remain in your pool, figuring you’ll have one out at any given point and that initially, only the master will be potentially able to take on VMs from whichever hosts is in the process of being upgraded

Preferably also have recent backups of all your VMs. It’s generally also a good idea to keep good notes about your various host configurations, including all physical and virtual networks, VLANs, iSCSI and NFS connection and the like. I’d recommend doing an “export resource data” for the pool, which has been available as a utility since XS 6.5 and can run from XenCenter.To export resource data:

In the XenCenter Navigation pane, click Infrastructure and then click on the pool.
From the XenCenter menu, click Pool and then select Export Resource Data.
Browse to a location where you would like to save report and then click Save.

You can chose between an XLS or CVS output. Note that this feature is only available on XenCenter with paid-for licensed version of XenServer. However, it can also be run via the CLI for any (including free) version of XS 6.5 or newer using:

# xe pool-dump-database file-name=target-output-file-name

Being prepared now for the upgrade, you may still run into issues along the way, in particular if you run out of space to move VMs to a different server or if the upgrade process hangs. If the master server manages to make it through the upgrade and the upgrade process fails at some point later, one of the first consequences will be that you will no longer be able to add any external hosts to the pool because the pool will be in a mixed state of hosts running different XS versions. This is not a good situation and makes it very difficult under some circumstances to recover from.

Recommended Process for Changing XenServer Partition Layout

Through various discussions on the XenServer discussion forum as well as some of my own experimentation, this is what I recommend as the most solid and reliable way of doing an upgrade to XS 7.X with the new partition layout coming from a version that still has the old layout. It will take quite a bit longer, but is more reliable.

In addition to all the preparatory steps listed above:

You will be needing to eject all hosts from the pool at one point or another, so make sure you have carefully recorded the various network and other settings, in particular some which are not retained as such in the metadata exports, such as the individual iSCSI network connections or NFS connections.
Copy your /etc/fstab file and also be sure to check for any other customizations you may have done, such as cron jobs or additions to /etc/rc.local (and also note that rc.local does not run on its own under XS 7.X, so you will need to manually enable it to do so – see the section “Enabling rc.local to Run On Its Own” below).

Once your enhanced preparation is complete:

Start with your pool master and do a rolling pool upgrade to it. Do not attempt to upgrade to the new partition layout! After it reboots, it should still have retained the role of pool master. Note that the rolling pool upgrade will not allow you to pick which host it will upgrade next beyond the master so be certain that any of these hosts can have all its VMs fit on the pool master. If necessary move some or all the VMs on the pool master onto other hosts within the pool before you commence the upgrade procedure.
Pick another host and migrate all its VMs to the pool master.
Follow this procedure for all hosts until the pool is completely upgraded to the identical version of XS 7.X on all pool members. You can either continue to use the rolling pool upgrade or if desired, switch to manual upgrade mode.
Making sure you’ve carefully recorded all important storage and network information on that host, migrate all VMs off a non-master host and eject it from the pool.
Touch the file /var/preserve/safe2upgrade and plan on the local SR being wiped and re-created. Then shutdown the host and perform a standalone installation to that just ejected host. It will have the new partition layout.
Reduce the network settings on this host to just a single primary management interface NIC, one that matches the same one it had initially. Rejoin this just upgraded host back into the pool. Many of the pool metadata settings should automatically get recreated, in particular any of the pooled resources. Update any host-specific network settings as well as other customizations.
Continue this process for the remainder of the non-master hosts.
When these have all been completed, designate a new pool master using the command “xe pool-designate-new-master host-uuid=new-master-uuid”. Make sure the transition to the new pool master is properly completed.
Eject what was the original pool master and perform the standalone upgrade and rejoin the host to the pool as before.
Celebrate your hard work!

While this process will take easily two times as long as a standard upgrade, at least you know that things should not break so badly that you may have to spend hours of additional time making things right. As a side benefit (if you want to consider it as such), it will also force you to take stock of how your pool is configured and require you to take careful inventory, In the event of a disaster, you will be grateful to have gone through this process as it may be very close to what you may have to go through under less controlled circumstances!I have done this myself several of times and had it work correctly each time.

An Additional Item: If the Console Shows No Network Configured on Dell Servers with iDRAC modules

This condition can show up unexpectedly and while normally something like this can be handled if the host is found to be in emergency mode with an “xe pool-recover-slaves” command, that’s not always the case. And even more oddly, if instead you ssh in and run xsconsole from the CLI, all looks perfectly normal, including all the network settings that appear present and correct and also match the settings visible in XenCenter. This condition, as far as I know, seems unique to Dell servers and was seen with iDRAC 7, 8 and 9 modules. Here's what it looks like:

The issue here appears to have been a change in behavior that kicked in starting with XenServer 7.1 and hence may not even be evident in XS 7.0.

The fix turned out to be an upgrade all the BIOS/firmware and iDRAC configurations. In this particular case, I made use of the method I described in this blog entry and that took care of it. Note that this still does not seem to consistently address this issue, in particular with some older hardware (e.g., R715 with an iDRAC 6, even after updating to DSU 1.4.2, BIOS 3.2.1, but being stuck at iDRAC version 1.97 – apparently not upgradeable).

Enabling rc.local To Run On Its Own

The rc.local file is not automatically executed by default under RHEL/CentOS 7 installations, and since XS 7 is based on CentOS 7, it is no exception. If you wish to customize your environment and assure this file is executed at boot time, you will have to manually enable /etc/rc.local to run automatically on reboot on XenServer 7.X and to do so, will need to run these two commands:

# chmod u+x /etc/rc.d/rc.local
# systemctl start rc-local

You can verify that rc.local is now running with the following command, which in turn should produce output similar to what is shown below:

# systemctl status rc-local

   rc-local.service - /etc/rc.d/rc.local Compatibility
   Loaded: loaded (/usr/lib/systemd/system/rc-local.service; static; vendor preset: disabled)
   Active: active (exited) since Sun 2017-06-18 09:09:50 MST; 1 weeks 6 days ago
   Process: 4649 ExecStart=/etc/rc.d/rc.local start (code=exited, status=0/SUCCESS)

A Final Word

Once again, I will reiterate that feedback is always appreciated and the XenServer forum is one good option. Errors, on the other hand, should be reported to the XenServer bugs site with as much information as possible to help the engineers understand and reproduce the issues.

I sincerely hope some of the information in this series will have been useful to some degree.

I would like to most gratefully acknowledge Andrew Wood, fellow Citrix Technology Professional, for the review of and constructive additions to this blog.

Czytaj dalej →

From the Field: XenServer 7.X Pool Upgrades, Part 1: Background Information and Clean Install

By Tobias Kreidl On 4 lipca 2017 · Add Comment

The XenServer upgrade process with XenServer 7.0 can be a bit more involved than with past upgrades, particularly if upgrading from a version prior to 7.0. In this series I’ll discuss various approaches to upgrading XenServer (XS) pools to 7.X and some of the issues you might encounter during the process based on my experiences in the field.

Part 1 deals with some of the preliminary actions that need to be taken into consideration and the planning process, as well as the case for a clean installation. Part 2 deals with the alternative, namely a true in-place upgrade procedure, and what issues may arise during and after the procedure.

Why Upgrade to XenServer 7?

XenServer (XS) 7 has been out for a little over a year now and is currently up to release 7.2. A number of improvements in scaling, functionality and features are available in XS 7, which should encourage users to upgrade to take advantage of them.

First Step in Any XS Upgrade

The first step in any XenServer upgrade is to upgrade your XenCenter to at least the minimum version required to support the newest installation of XenServer you are or will be accessing.

This step is crucial. Failure to do this causes puppies to die. This is because older versions of XenCenter won’t connect to newer versions of XS and in many cases, new features are built into the newer versions of XenCenter that are not available in older versions and support important components in the newer XS releases, including hotfix and rolling pool upgrade options, changes in licensing, and others such as supported pooled storage options. In short, your upgrade process will rapidly come to a screeching halt and there may be potential damage done unless you are working with the proper version of XenCenter.

It is also highly recommended to make backups of anything and everything prior to upgrading. More on that will be discussed later.

Summary of Pre-Upgrade Checks

From experience, the following are key activities to perform before upgrading any XenServer pool:

XenCenter has been upgraded to at least the version that does or will match your newest pool

Assure the proper backups of virtual machines (VMs), metadata, etc. have been performed
Make sure that all VMs are agile (can be migrated to other hosts within the pool) or in the case of a rolling pool upgrade, at least shut down on all local SRs
All hosts must not only be running the same version of XenServer, but must have the identical list of hotfixes applied
Clean up old messages in /var/lib/xcp/blobs/

I’d also recommend doing an “export resource data” for the pool, which has been available as a utility since XS 6.5 and can run from XenCenter.

To export resource data:

In the XenCenter Navigation pane, click Infrastructure and then click on the pool.
From the XenCenter menu, click Pool and then select Export Resource Data.
Browse to a location where you would like to save the report and then click Save.

You can chose between an XLS or CVS output. Note that this feature is only available on XenCenter with paid-for licensed versions of XenServer. However, it can also be run via the CLI for any (including free) version of XS 6.5 or newer using:

# xe pool-dump-database file-name=

Benefits of a XenServer 7 Upgrade

XenServer 7 is a significant jump from XS 6.5 SP1 in many ways, including improvements in scale and performance. It also has added a number of very nice new features (see what's new in XS 7; what's new in XS 7.1; XS 7.2 ).

While any upgrade can be a bit intimidating, ones that jump to a whole new edition can be more complicated and given the changes involved here, this case was no exception. I would also recommend trying to verify your hardware is present on the HCL though it’s often slow to get populated. If in doubt and you have the extra resources, take as close as possible a spare server and try to do an installation on it first to make sure it’s compatible. If you don’t have such a machine and have the spare capacity, eject one of the hosts from your pool and use it to test out the upgrade. Before performing an upgrade on a production server or pool, it is always recommended to try this out first on a test configuration.

I do not recommend at this time the rolling pool or standalone upgrade that involves switching at the same time from the old to the new partition layout. Overall experiences within the user community have not been consistently positive.

It is hoped that the preparatory steps that are recommended before undertaking an upgrade do not prove to be too daunting. This series is intended to help with those preparations as well as give guidelines and tips how to go about the process and what to do if issues arise. There are clearly a number of ways to tackle upgrades and there are undoubtedly numerous other possible approaches. One might, for example, storage XenMotion VMs to other pools or make use of temporary pooled storage as part of the process. The detaching and attaching pooled storage to servers already running a newer version of XenServer is another possible route to explore.

Experimentation is always something that should be encouraged, albeit not on production systems!

Feedback is always appreciated and the XenServer forum https://discussions.citrix.com/forum/101-xenserver/ is one good option. Errors, on the other hand, should be reported to the XenServer bugs site with preferably sufficient detail to allow the engineers to be able to understand and reproduce the issues.

XS 7.X Leveraging a Clean Installation as an Upgrade Component

A clean install can become part of an upgrade procedure if for example an upgraded pool already exists and individual hosts or hosts from a whole different pool are to be merged into a pool running a newer version of XenServer. Hence talking about this option is relevant to upgrading. You can potentially storage Xenmotion VMs over to a new pool, then eject and dissolve individual pool members, perform a clean installation on them, and join the hosts to the new pool. Of course the target pool must have adequate resources to hold and run the VMs being moved over until the new hosts can be configured and added. This process might involve leveraging temporary external storage, for example, by creating an NFS SR.

The easiest installation scenario is a clean install, which has thankfully always been the case and this approach may work well in some instances. This is obviously suited best for a standalone server or as mentioned above, creating a server that will be joined to an existing pool.

Regardless of which upgrade process you undertake, be sure to review your VMs afterwards to see if XenTools need to be updated.

A Few Words About the XenServer 7 New Disk Layout

The main item to be aware of is that a clean installation will always create the new GPT partition layout as follows:

Size	Purpose
4GB	log partition (syslogd and other logs)
18GB	backup partition for dom0 (for upgrades)
18GB	XenServer host control domain (dom0) partition
0.5GB	UEFI boot partition
1GB	swap partition

This adds up to 41.5 GB. There can be a sixth partition which is optionally used for creating a local storage repository (SR) or a portion of a local SR if spanning it to other storage devices. See also James Bulpin's XS 7 blog for more details about the XS 7 architecture.

Here is what a real-life partition table looks like after performing an installation. Note that the space for a local SR ends up in partition 3 (in this case, a default LVM partition 94.6 GB in size):

# fdisk -l /dev/sda

WARNING: fdisk GPT support is currently new, and therefore in an experimental phase. Use at your own discretion.

Disk /dev/sda: 146.2 GB, 146163105792 bytes, 285474816 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk label type: gpt

# Start End Size Type Name

1 46139392 83888127 18G Microsoft basic

2 8390656 46139391 18G Microsoft basic

3 87033856 285474782 94.6G Linux LVM

4 83888128 84936703 512M BIOS boot parti

5 2048 8390655 4G Microsoft basic

6 84936704 87033855 1G Linux swap

Why was the XenServer 7 Disk Layout Changed?

The XS 6.5 SP1 and earlier partition scheme was much smaller and simpler:

4 GB XenServer host control domain (dom0) partition

4 GB backup partition for dom0 (for upgrades)

The rest of the space was again available for use as a local SR in a third partition.

The default partition layout was changed under XS 7 for a number of reasons. For one, most storage devices these days start with a capacity of few hundred GB and so it makes sense to be able to allocate more space for the XenServer installation itself beyond just 8 GB. In that regard, running out of space has been a common problem caused by either a lot of hotfixes being stored or the accumulation of a lot of log files. Isolating /var/log to its own partition now helps keep XS from crashing in case it fills up. Before, /var/log was just another subdirectory on the system “/” partition and inevitably resulted in a system crash when the partition became 100% full. A separate UEFI boot area was also added. Plus, the Linux swap space was assigned to its own partition. In short, there was no reason not to make use of more space and provide an improved partitioning scheme that would also scale better for larger pools containing a lot of VMs.

Note: One idiosyncrasy of XS 6.5 SP1 is that it switched to a GUID partition table in order to use more than 2 TB of local storage if available, but this has evidently been undone in 7.0 since GPT is no longer limited as such when running a 64-bit OS based on CentOS 7.

Note: There are changes in the minimum host physical system requirements, including a recommended minimum of 46 GB of disk space. This will impact environments that used smaller partitions or SSDs for the XS OS.

Stage Complete: Level Up!

Well done – you’ve completed Part 1. In Part 2, I’ll deal with the alternative to leveraging a clean install -- a true in-place upgrade procedure, and what issues may arise during and after the procedure.

Czytaj dalej →

XENSERVER HIGH-AVAILABILITY ALTERNATIVE HA-LIZARD

By Tobias Kreidl On 29 grudnia 2016 · Add Comment

WHY HA AND WHAT IT DOES

XenServer (XS) contains a native high-availability (HA) option which allows quite a bit of flexibility in determining the state of a pool of hosts and under what circumstances Virtual Machines (VMs) are to be restarted on alternative hosts in the event of the loss of the ability of a host to be able to serve VMs. HA is a very useful feature that protects VMs from staying failed in the event of a server crash or other incident that makes VMs inaccessible. Allowing a XS pool to help itself maintain the functionality of VMs is an important feature and one that plays a large role in sustaining as much uptime as possible. Permitting the servers to automatically deal with fail-overs makes system administration easier and allows for more rapid reaction times to incidents, leading to increased up-time for servers and the applications they run.

XS allows for the designation of three different treatments of Virtual Machines: (1) always restart, (2) restart if possible, and (3) do not restart. The VMs designated with the highest restart priority will be the first to be attempted to restart and all will be handled, provided adequate resources (primarily, host memory) are available. A specific start order, allowing for some VMs to be checked to be running before others, can also be established. VMs will be automatically distributed among whatever remaining XS hosts are considered active. Where necessary, note that hosts that contain expandable memory will be shrunk down to accommodate additional hosts and those hosts designated to be restarted will also be run with reduced memory, if necessary. If additional capacity exists to run more VMs, those designated as “start if possible” will be brought online. Whichever VMs that are not considered essential typically will be marked as “do not restart” and hence will be left “off” had they been running before, requiring any of those desired to be restarted to be done manually, resources permitting.

XS also allows for specifying the minimum number of active hosts to remain to accommodate failures; larger pools that are not overly populated with VMs can readily accommodate even two or more host failures.

The election of what hosts are “live” and should be considered active members of the pool follows a rather involved process of a combination of network accessibility plus access to an independent designated pooled Storage Repository (SR) that serves as an additional metric. The pooled SR can also be a fiber channel device, being independent of Ethernet connections. A quorum-based algorithm is applied to establish which servers are up and active as members of the pool and which -- in the event of a pool master failure -- should be elected the new pool master.

WHEN HA WORKS, IT WORKS GREAT

Without going into more detail, suffice it to say that this methodology works very well, however requiring a few prerequisite conditions that need to be taken into consideration. First of all, the mandate that a pooled storage device be available clearly means that a pool consisting of hosts that only make use of local storage will be precluded. Second, there is also a constraint that for a quorum to be possible, it is required to have a minimum of three hosts in the pool or HA results will be unpredictable as the election of a pool master can become ambiguous. This comes about because of the so-called “split brain” issue (http://linux-ha.org/wiki/Split_Brain) which is endemic in many different operating system environments that employ a quorum as means of making such a decision. Furthermore, while fencing (the process of isolating the host; see for example http://linux-ha.org/wiki/Fencing) is the typical recourse, the lack of intercommunication can result in a wrong decision being made and hence loss of access to VMs. Having experimented with two-host pools and the native XenServer HA, I would say that an estimate of it working about half the time is about right and from a statistical viewpoint, pretty much what you would expect.

This limitation is, however, still of immediate concern to those with either no pooled storage and/or only two hosts in a pool. With a little bit of extra network connectivity, a relatively simple and inexpensive solution to the external SR can be provided by making a very small NFS-based SR available. The second condition, however, is not readily rectified without the expense of at least one additional host and all the connectivity associated with it. In some cases, this may simply not be an affordable option.

ENTER HA-LIZARD

For a number of years now, an alternative method of providing HA has been available through the program package provided by HA-Lizard (http://www.halizard.com/) , a community project that provides a free alternative that is neither dependent on external SRs nor requires a minimum of three hosts within a pool. In this blog, the focus will be on the standard HA-Lizard version and because of the particularly harder-to-handle situation of a two-node pool, it will also be the subject of discussion.

I had been experimenting for some time with HA-Lizard and found in particular that I was able to create failure scenarios that needed some improvement. HA-Lizard’s Salvatore Costantino was more than willing to lend an ear to the cases I had found and this led further to a very productive collaboration on investigating and implementing means to deal with a number of specific cases involving two-host pools. The result of these several months of efforts is a new HA-Lizard release that manages to address a number of additional scenarios above and beyond its earlier capabilities.

It is worthwhile mentioning that there are two ways of deploying HA-Lizard:

1) Most use cases combine HA-Lizard and iSCSI-HA which creates a two-node pool using local storage while maintaining full VM agility with VMs being able to run on either host. In this case, DRBD (http://www.drbd.org/) is implemented in this type of deployment and it works very well making use of the real-time storage replication.

2) HA-Lizard, only, is used with an external Storage Repository (as in this particular case).

Before going into details of the investigation, a few words should go towards a brief explanation of how this works. Note that there is only Internet connectivity (the use of a heuristic network node) and no external SR, so how is a split brain situation then avoidable?

This is how I'd describe the course of action in this two-node situation:

If a node sees the gateway, assume it's alive. If it cannot, assume it's a good candidate for fencing. If the node that cannot see the gateway is the master, it should internally kill any running VMs and surrender its ability to be the master and fence itself. The slave node should promote itself to master and attempt to restart any missing VMs. Any that are on the previous master will probably fail though, because there is no communication to the old master. If the old VMs cannot be restarted, eventually the new master will be able to restart them regardless after a toolstack restart. If the slave node fails by not being able to communicate with the network, as long as the master still sees the network and not the slave’s network, it can assume the slave needs to fence itself, kill off its VMs and assume that they will be restarted on the current master. The slave needs to realize it cannot communicate out, and therefore should kill off any of its VMs and fence itself.

Naturally, the trickier part comes with the timing of the various actions, since each node has to blindly assume the other is going to conduct a sequence of events. The key here is that these are all agreed on ahead of time and as long as each follows its own specific instructions, it should not matter that each of the two nodes cannot see the other node. In essence, the lack of communication in this case allows for creating a very specific course of action! If both nodes fail, obviously the case is hopeless, but that would be true of any HA configuration in which no node is left standing.

Various test plans were worked out for various cases and the table below elucidates the different test scenarios, what was expected and what was actually observed. It is very encouraging that the vast majority of these cases can now be properly handled.

Particularly tricky here was the case of rebooting the master server from the shell, without first disabling HA-Lizard (something one could readily forget to do). Since the fail-over process takes a while, a large number of VMs cannot be handled before the communication breakdown takes place, hence one is left with a bit of a mess to clean up in the end. Nevertheless, it’s still good to know what happens if something takes place that rightfully shouldn’t!

The other cases, whether intentional or not, are handled predictably and reliably, which is of course the intent. Typically, a two-node pool isn’t going to have a lot of complex VM dependencies, so the lack of a start order of VMs should not be perceived as a big shortcoming. Support for this feature may even be added in a future release.

CONCLUSIONS

HA-Lizard is a viable alternative to the native Citrix HA configuration. It’s straightforward to set up and can handle standard failover cases with a selective “restart/do not restart” setting for each VM or can be globally configured. There are a quite a number of configuration parameters which the reader is encouraged to research in the extensive HA-Lizard documentation. There is also an on-line forum which serves as a source for information and prompt assistance with issues. This most recent release 2.1.3 is supported on both XenServer 6.5 and 7.0.

Above all, HA-Lizard shines when it comes to handling a non-pooled storage environment and in particular, all configurations of the dreaded two-node pool configuration. From my direct experience, HA-Lizard now handles the vast majority of issues involved in a two-node pool and can do so more reliably than the non-supported two-node pool using Citrix’ own HA application. It has been possible to conduct a lot of tests with various cases and importantly, and to do so multiple times to ensure the actions are predictable and repeatable.

I would encourage taking a look at HA-Lizard and giving it a good test run. The software is free (contributions are accepted) and it is in extensive use and has a proven track record. For a two-host pool, I can frankly not think of a better alternative, especially with these latest improvements and enhancements.

I would also like to thank Salvatore Costantino for the opportunity to participate in this investigation and am very pleased to see the fruits of this collaboration. It has been one way of contributing to the Citrix XenServer user community that many can immediately benefit from.

Czytaj dalej →

XenServer High-Availability Alternative HA-Lizard

By Tobias Kreidl On 29 grudnia 2016 · Add Comment

WHY HA AND WHAT IT DOES

WHEN HA WORKS, IT WORKS GREAT

ENTER HA-LIZARD

It is worthwhile mentioning that there are two ways of deploying HA-Lizard:

2) HA-Lizard, only, is used with an external Storage Repository (as in this particular case).

CONCLUSIONS

Czytaj dalej →

NAU VMbackup 3.0 for XenServer

By Tobias Kreidl On 8 kwietnia 2016 · Add Comment

By Tobias Kreidl and Duane Booher

Northern Arizona University, Information Technology Services

Over eight years ago, back in the days of XenServer 5, not a lot of backup and restore options were available, either as commercial products or as freeware, and we quickly came to the realization that data recovery was a vital component to a production environment and hence we needed an affordable and flexible solution. The conclusion at the time was that we might as well build our own, and though the availability of options has grown significantly over the last number of years, we’ve stuck with our own home-grown solution which leverages Citrix XenServer SDK and XenAPI (http://xenserver.org/partners/developing-products-for-xenserver.html). Early versions were created from the contributions of Douglas Pace, Tobias Kreidl and David McArthur. During the last several years, the lion’s share of development has been performed by Duane Booher. This article discusses the latest 3.0 release.

A Bit of History

With the many alternatives now available, one might ask why we have stuck with this rather un-flashy script and CLI-based mechanism. There are clearly numerous reasons. For one, in-house products allow total control over all aspects of their development and support. The financial outlay is all people’s time and since there are no contracts or support fees, it’s very controllable and predictable. We also found from time-to-time that various features were not readily available in other sources we looked at. We also felt early on as an educational institution that we could give back to the community by freely providing the product along with its source code; the most recent version is available via GitHub at https://github.com/NAUbackup/VmBackup for free under the terms of the GNU General Public License. There was a write-up at https://www.citrix.com/blogs/2014/06/03/another-successful-community-xenserver-sdk-project-free-backup-tools-and-scripts-naubackup-restore-v2-0-released/ when the first GitHub version was published. Earlier versions were made available via the Citrix community site (Citrix Developer Network), sometimes referred to as the Citrix Code Share, where community contributions were published for a number of products. When that site was discontinued in 2013, we relocated the distribution to GitHub.

Because we “eat our own dog food,” VMbackup gets extensive and constant testing because we rely on it ourselves as the means to create backups and provide for restores for cases of accidental deletion, unexpected data corruption, or in the event that disaster recovery might be needed. The mechanisms are carefully tested before going into production and we perform frequent tests to ensure the integrity of the backups and that restores really do work. A number of times, we have relied on resorting to recovering from our backups and it has been very reassuring that these have been successful.

What VMbackup Does

Very simply, VMbackup provides a framework for backing up virtual machines (VMs) hosted on XenServer to an external storage device, as well as the means to recover such VMs for whatever reason that might have resulted in loss, be it disaster recovery, restoring an accidentally deleted VM, recovering from data corruption, etc.

The VMbackup distribution consists of a script written in Python and a configuration file. Other than a README document file, that’s it other than the XenServer SDK components which one needs to download separately; see http://xenserver.org/partners/developing-products-for-xenserver.html for details. There is no fancy GUI to become familiar with, and instead, just a few simple things that need to be configured, plus a destination for the backups needs to be made accessible (this is generally an NFS share, though SMB/CIFS will work, as well). Using cron job entries, a single host or an entire pool can be set up to perform periodic backups. Configurations on individual hosts in a pool are needed in that the pool master performs the majority of the work and it can readily change to a different XenServer, while individual host-based instances are also needed when local storage is also made use of, since access to any local SRs can only be performed from each individual XenServer. A cron entry and numerous configuration examples are given in the documentation.

To avoid interruptions of any running VMs, the process of backing up a VM follows these basic steps:

A snapshot of the VM and its storage is made
Using the xe utility vm-export, that snapshot is exported to the target external storage
The snapshot is deleted, freeing up that space

In addition, some VM metadata are collected and saved, which can be very useful in the event a VM needs to be restored. The metadata include:

vm.cfg - includes name_label, name_description, memory_dynamic_max, VCPUs_max, VCPUs_at_startup, os_version, orig_uuid
DISK-xvda (for each attached disk)

vbd.cfg - includes userdevice, bootable, mode, type, unplugable, empty, orig_uuid
vdi.cfg - includes name_label, name_description, virtual_size, type, sharable, read_only, orig_uuid, orig_sr_uuid

VIFs (for each attached VIF)

vif-0.cfg - includes device, network_name_label, MTU, MAC, other_config, orig_uuid

An additional option is to create a backup of the entire XenServer pool metadata, which is essential in dealing with the aftermath of a major disaster that affects the entire pool. This is accomplished via the “xe pool-dump-database” command.

In the event of errors, there are automatic clean-up procedures in place that will remove any remnants plus make sure that earlier successful backups are not purged beyond the specified number of “good” copies to retain.

There are numerous configuration options that allow to specify which VMs get backed up, how many backup versions are to be retained, whether the backups should be compressed (1) as part of the process, as well as optional report generation and notification setups.

New Features in VMbackup 3.0

A number of additional features have been added to this latest 3.0 release, adding flexibility and functionality. Some of these came about because of the sheer number of VMs that needed to be dealt with, SR space issues as well as with changes coming to the next XenServer release. These additions include:

VM “preview” option: To be able to look for syntax errors and ensure parameters are being defined properly, a VM can have a syntax check performed on it and if necessary, adjustments can then be made based on the diagnosis to achieve the desired configuration.
Support for VMs containing spaces: By surrounding VM names in the configuration file with double quotes, VM names containing spaces can now be processed.
Wildcard suffixes: This very versatile option permits groups of VMs to be configured to be handled similarly, eliminating the need to create individual settings for every desired VM. Examples include “PRD-*”, “SQL*” and in fact, if all VMs in the pool should be backed up, even “*”. There are however, a number of restrictions on wildcard usage (2).
Exclude VMs: Along with the wildcard option to select which VMs to back up, clearly a need arises to provide the means to exclude certain VMs (in addition to the other alternative, which is simply to rename them such that they do not match a certain backup set). Currently, each excluded VM must be named separately and any such VMs should de defined at the end of the configuration file.
Export the OS disk VDI, only: In some cases, a VM may contain multiple storage devices (VDIs) that are so large that it is impractical or impossible to take a snapshot of the entire VM and its storage. Hence, we have introduced the means to backup and restore only the operating system device (assumed to be Disk 0). In addition to space limitations, some storage, such as DB data, may not be able to be reliably backed up using a full VM snapshot. Furthermore, the next XenServer release (Dundee) will likely support up to as many as perhaps 255 storage devices per VM, making a vm-export even more involved under such circumstances. Another big advantage here is that currently, this process is much more efficient and faster than a VM export by a factor of three or more!
Root password obfuscation: So that clear-text passwords associated with the XenServer pool are not embedded in the scripts themselves, the password can be basically encoded into a file.

The mechanism for a running VM from which only the primary disk is to be backed up is similar to the full VM backup. The process of backing up such a VM follows these basic steps:

A snapshot of just the VM's Disk 0 storage is made
Using the xe utility vdi-export, that snapshot is exported to the target external storage
The snapshot is deleted, freeing up that space

As with the full VM export, some metadata for the VM are also collected and saved for this VDI export option.

These added features are of course subject to change in future releases, though typically later editions generally encompass the support of previous versions to preserve backwards compatibility.

Examples

Let’s look at the configuration file weekend.cfg:

# Weekend VMs
max_backups=4
backup_dir=/snapshots/BACKUPS
#
vdi-export=PROD-CentOS7-large-user-disks
vm-export=PROD*
vm-export=DEV-RH*:3
exclude=PROD-ubuntu12-benchmark
exclude=PRODtestVM

Comment lines start with a hash mark and may be contained anywhere with the file. The hash mark must appear as the first character in the line. Note that the default number of retained backups is set here to four. The destination directory is set next, indicating where the backups will be written to. We then see a case where only the OS disk is being backed up for the specific VM "PROD-CentOS7-large-user-disks" and below that, all VMs beginning with “PROD” are backed up using the default settings. Just below that, a definition is created for all VMs starting with "DEV-RH" and the default number of backups is reduced for all of these from the global default of four down to three. Finally, we see two excludes for specific VMs that fall into the “PROD*” group that should not be backed up at all.

To launch the script manually, you would issue from the command line:

./VmBackup.py password weekend.cfg

To launch the script via a cron job, you would create a single-line entry like this:

10 0 * * 6 /usr/bin/python /snapshots/NAUbackup/VmBackup.py password
/snapshots/NAUbackup/weekend.cfg >> /snapshots/NAUbackup/logs/VmBackup.log 2>&1

This would run the task at ten minutes past midnight on Saturday and create a log entry called VmBackup.log. This cron entry would need to be installed on each host of a XenServer pool.

Additional Notes

It can be helpful to break up when backups are run so that they don’t all have to be done at once, which may be impractical, take so long as to possibly impact performance during the day, or need to be coordinated with when is best for specific VMs (such as before or after patches are applied). These situations are best dealt with by creating separate cron jobs for each subset.

There is a fair load on the server, comparable to any vm-export, and hence the queue is processed linearly with only one active snapshot and export sequence for a VM being run at a time. This is also why we suggest you perform the backups and then asynchronously perform any compression on the files on the external storage host itself to alleviate the CPU load on the XenServer host end.

For even more redundancy, you can readily duplicate or mirror the backup area to another storage location, perhaps in another building or even somewhere off-site. This can readily be accomplished using various copy or mirroring utilities, such as rcp, sftp, wget, nsync, rsync, etc.

This latest release has been tested on XenServer 6.5 (SP1) and various beta and technical preview versions of the Dundee release. In particular, note that the vdi-export utility, while dating back a while, is not well documented and we strongly recommend not trying to use it on any XenServer release before XS 6.5. Doing so is clearly at your own risk.

The NAU VMbackup distribution can be found at: https://github.com/NAUbackup/VmBackup

In Conclusion

This is a misleading heading, as there is not really a conclusion in the sense that this project continues to be active and as long as there is a perceived need for it, we plan to continue working on keeping it running on future XenServer releases and adding functionality as needs and resources dictate. Our hope is naturally that the community can make at least as good use of it as we have ourselves.

Footnotes:

Alternatively, to save time and resources, the compression can potentially be handled asynchronously by the host onto which the backups are written, hence reducing overhead and resource utilization on the XenServer hosts, themselves.
Certain limitations exist currently with how wildcards can be utilized. Leading wildcards are not allowed, nor are multiple wildcards within a string. This may be enhanced at a later date to provide even more flexibility.

This article was written by Tobias Kreidl and Duane Booher, both of Northern Arizona University, Information Technology Services. Tobias' biography is available at this site, and Duane's LinkedIn profile is at https://www.linkedin.com/in/duane-booher-a068a03 while both can also be found on http://discussions.citrix.com primarily in the XenServer forum.

Czytaj dalej →

Review: XenServer 6.5 SP1 Training CXS-300

By Tobias Kreidl On 13 listopada 2015 · Add Comment

A few weeks ago, I received an invitation to participate in the first new XenServer class to be rolled out in over three years, namely CXS-300: Citrix XenServer 6.5 SP1 Administration. Those of you with good memories may recall that XenServer 6.0, on which the previous course was based, was officially released on September 30, 2011. Being an invited guest in what was to be only the third time the class had been ever held was something that just couldn’t be passed up, so I hastily agreed. After all, the evolution of the product since 6.0 has been enormous. Plus, I have been a huge fan of XenServer since first working with version 5.0 back in 2008. Shortly before the open-sourcing of XenServer in 2013, I still recall the warnings of brash naysayers that XenServer was all but dead. However, things took a very different turn in the summer of 2013 with the open-source release and subsequent major efforts to improve and augment product features. While certain elements were pulled and restored and there was a bit of confusion about changes in the licensing models, things have stabilized and all told, the power and versatility of XenServer with the 6.5 SP1 release is at a level now some thought it would never reach.

FROM 6.0 TO 6.5 – AND BEYOND

XenServer (XS for short) 6.5 SP1 made its debut on May 12, 2015. The feature set and changes are – as always – incorporated within the release notes. There are a number of changes of note that include an improved hotfix application mechanism, a whole new XenCenter layout (since 6.5), increased VM density, more guest OS support, a 64-bit kernel, the return of workload balancing (WLB) and the distributed virtual switch controller (DVSC) appliance, in-memory read caching, and many others. Significant improvements have been made to storage and network I/O performance and overall efficiency. XS 6.5 was also a release that benefited significantly from community participation in the Creedence project and the SP1 update builds upon this.

One notable point is that XenServer has been found to now host more XenDesktop/XenApp (XD/XA) instances than any other hypervisor (see this reference). And, indeed, when XenServer 6.0 was released, a lot of the associated training and testing on it was in conjunction with Provisioning Services (PVS). Some users, however, discovered XenServer long before this as a perfectly viable hypervisor capable of hosting a variety of Linux and Windows virtual machines, without having even given thought to XenDesktop or XenApp hosting. For those who first became familiar with XS in that context, the added course material covering provisioning services had in reality relatively little to do with XenServer functionality as an entity. Some viewed PVS an overly emphasized component of the course and exam. In this new course, I am pleased to say that XS’s original roots as a versatile hypervisor is where the emphasis now lies. XD/XA is of course discussed, but the many features available that are fundamental to XS itself is what the course focuses on, and it does that well.

COURSE MATERIALS: WHAT’S INCLUDED

The new “mission” of the course from my perspective is to focus on the core product itself and not only understand its concepts, but to be able to walk away with practical working knowledge. Citrix puts it that the course should be “engaging and immersive”. To that effect, the instructor-led course CXS-300 can be taken in a physical classroom or via remote GoToMeeting (I did the latter) and incorporates a lecture presentation, a parallel eCourseware manual plus a student exercise workbook (lab guide) and access to a personal live lab during the entire course. The eCourseware manual serves multiple purposes, providing the means to follow along with the instructor and later enabling an independent review of the presented material. It adds a very nice feature of providing an in-line notepad for each individual topic (hence, there are often many of these on a page) and these can be used for note taking and can be saved and later edited. In fact, a great takeaway of this training is that you are given permanent access to your personalized eCourseware manual, including all your notes.

The course itself is well organized; there are so many components to XenServer that five days works out in my opinion to be about right – partly because often question and answer sessions with the instructor will take up more time than one might guess, and also, in some cases all participants may have already some familiarity with XS or other hypervisor that makes it possible to go into some added depth in some areas. There will always need to be some flexibility depending on the level of students in any particular class.

A very strong point of the course is the set of diagrams and illustrations that are incorporated, some of which are animated. These compliment the written material very well and the visual reinforcement of the subject matter is very beneficial. Below is an example, illustrating a high availability (HA) scenario:

XS6.5SP1_course_image.jpg

The course itself is divided into a number of chapters that cover the whole range of features of XS, enforced by some in-line Q&A examples in the eCourseware manual and with related lab exercises. Included as part of the course are not only important standard components, such as HA and Xenmotion, but some that require plugins or advanced licenses, such as workload balancing (WLB), the distributed virtual switch controller (DVSC) appliance and in-memory read caching. The immediate hands-on lab exercises in each chapter with the just-discussed topics are a very strong point of the course and the majority of exercises are really well designed to allow putting the material directly to practical use. For those who have already some familiarity with XS and are able to complete the assignments quickly, the lab environment itself offers a great sandbox in which to experiment. Most components can readily be re-created if need be, so one can afford to be somewhat adventurous.

The lab, while relying heavily on the XenCenter GUI for most of the operations, does make a fair amount of use of the command line interface (CLI) for some operations. This is a very good thing for several reasons. First off, one may not always have access to XenCenter and knowing some essential commands is definitely a good thing in such an event. The CLI is also necessary in a few cases where there is no equivalent available in XenCenter. Some CLI commands offer some added parameters or advanced functionality that may again not be available in the management GUI. Furthermore, many operations can benefit from being scripted and this introduction to the CLI is a good starting point. For Windows aficionados, there are even some PowerShell exercises to whet their appetites, plus connecting to an Active Directory server to provide role-based access control (RBAC) is covered.

THE INSTRUCTOR

So far, the materials and content have been the primary points of discussion. However, what truly can make or break a class is the instructor. The class happened to be quite small, and primarily with individuals attending remotely. Attendees were in fact from four different countries in different time zones, making it a very early start for some and very late in the day for others. Roughly half of those participating in the class were not native English speakers, though all had admirable skills in both English and some form of hypervisor administration. Being all able to keep up a common general pace allowed the class to flow exceptionally well. I was impressed with the overall abilities and astuteness of each and every participant.

The instructor, Jesse Wilson, was first class in many ways. First off, knowing the material and being able to present it well are primary prerequisites. But above and beyond that was his ability to field questions related to the topic at hand and even to go off onto relevant tangential material and be able to juggle all of that and still make sure the class stayed on schedule. Both keeping the flow going and also entertaining enough to hold students’ attention are key to holding a successful class. When elements of a topic became more of a debatable issue, he was quick to not only tackle the material in discussion, but to try this out right away in the lab environment to resolve it. The same pertained to demonstrating some themes that could benefit from a live demo as opposed to explaining them just verbally. Another strong point was his adding his own drawings to material to further clarify certain illustrations, where additional examples and explanations were helpful.

SUMMARY

All told, I found the course well structured, very relevant to the product and the working materials to be top notch. The course is attuned to the core product itself and all of its features, so all variations of the product editions are covered.

Positive points:

Good breadth of material
High-quality eCourseware materials
Well-presented illustrations and examples in the class material
Q&A incorporated into the eCourseware book
Ability to save course notes and permanent access to them
Relevant lab exercises matching the presented material
Real-life troubleshooting (nothing ever runs perfectly!)
Excellent instructor

Desiderata:

More “bonus” lab materials for those who want to dive deeper into topics
More time spent on networking and storage
A more responsive lab environment (which was slow at times)
More coverage of more complex storage Xenmotion cases in the lecture and lab

In short, this is a class that fulfills the needs of anyone from just learning about XenServer to even experienced administrators who want to dive more deeply into some of the additional features and differences that have been introduced in this latest XS 6.5 SP1 release. CXS-300: Citrix XenServer 6.5 SP1 Administration represents a makeover in every sense of the word, and I would say the end result is truly admirable.

Czytaj dalej →

Introduction to Saving VM Parameters: How I Metadata Backup

XenServer – Octoblu – SNMP Integration: The Sum of the Parts

Changes to Hotfixes and Maintenance for XenServer Releases

From the Field: XenServer 7.X Pool Upgrades, Part 2: Updates and Aftermath

From the Field: XenServer 7.X Pool Upgrades, Part 1: Background Information and Clean Install

XENSERVER HIGH-AVAILABILITY ALTERNATIVE HA-LIZARD

XenServer High-Availability Alternative HA-Lizard

NAU VMbackup 3.0 for XenServer

Review: XenServer 6.5 SP1 Training CXS-300

AnyThin’

Citrix XenServer 7.2