Thursday, September 30, 2010

Developing in a Virtual World

Back in December of last year I described how I set up my most recent laptop with VMWare, installing my various development environments in virtual machines instead of on my host operating system (Creating a More Manageable Development Environment). It's been about ten months since I originally setup that machine, and this seems like a fine time to share how the experience has been going.

Overall, I can say that I am very happy with how this is working. I have not had to reinstall my host operating system during the past ten months, and I think that this is the first time that I can think of when I've gone this long. I still have the early, clean image of this machine that I created shortly after performing my base install, but the thought of restoring my machine to that state simply has not crossed my mind.

In the past, after four or six months, my operating system had slowed down to the point where I felt it necessary to restore from an early image of my system. But not now. I am pleasantly surprised that my host operating system is humming along fine, thank you. And I even have several applications installed on the host that many developers blame for some of their ills, such as iTunes.

As for the virtual machines, they are doing fine as well. I currently have quite a collection of virtual machines that I use, some more than others. As I described in the previous post, I create a separate virtual machine for each development environment. For example, I have one for RAD Studio 2010, one for RAD Studio 2007, and another for Visual Studio 2008. I recently installed virtual machines for Visual Studio 2010 and Delphi XE. There are also some separate VMs for 32-bit operating systems, though most of them are 64-bit. There are others, but you get the picture.

My reasoning for this approach is multi-fold. First, by keeping each virtual machine pure (a single development environment) I prevent any possible cross contamination that might compromise one or more of the environments. It's these unintended side effects that I blame for the increasing instability of my previous OS installations. Second, I can test my applications under different conditions, such as 32-bit and 64-bit versions of Windows 7, or under Windows XP.

The third benefit is that I can keep backups of these individual VMs. In fact, I keep several backups of each. There is the clean backup that contains my original installation. I can always fall back to this if something goes terribly wrong, such as a Windows update that hoses one of the VMs. I also keep a current backup of each VM, and carry that around with me on a portable USB drive when I travel.

If something happens to my VM, or even to my laptop, I have a solution. For example, if my laptop simply quits on me, I could load my virtual machine onto any other machine on which VMWare is loaded, and I am ready to go.

Finally, and this is one that I have not actually had to confirm yet, but my thinking is that I can use these VMs on my next laptop. In other words, I will never have to install Delphi XE on Windows 7 ever again. On my next laptop, I will simply copy over the VM, and I should be good to go. Sure, I may have to install it on the next version of Windows (whatever that will be called), but that should only happen once, and then I will be set with Delphi XE and that version of the operating system (this assumes that I will be supporting clients who are using Delphi XE on that new operating system. If not, then I'll only have to install the more current versions of Delphi on that OS).

Lessons Learned


I wanted to do more than simply report that this approach seems to be working well. I also wanted to share some of the tricks and insights that I've gained during this experiment. Note, however, that I am using VMWare (VMWare 7.1.2 to be precise). While other virtual machines may support features similar to those I describe here, I am referring specifically to how they are working for me with VMWare.

Memory


Let's begin with memory and the virtual machines. When I initially set up each virtual machine, I configured each to use 3 gigabytes of RAM. My thinking was that I wanted the virtual machine performance to be similar to running my development environment in the host operating system.

My laptop has 8 GB of RAM (a minimum, if you ask me, for taking this particular route), which means that using VMs that use 3GBs limits you to running one at a time (sure, you could run two at a time, but I'd be careful about that. If your VMs require so much memory that you leave little left for the host operating system, bad thing can and will happen).

Over the past several days I have cranked several of the VMs down to 2 GB of RAM. Honestly, I really haven’t noticed a difference. The big change, however, is that I can now run two VMs simultaneously, and still have plenty of breathing room for my host OS.

License to Thrill


One thing that I did not go into in any depth in my first post was the licensing issues that arise when you use multiple VMs. In short, you need to have licenses for the software that you use in your virtual machines. This includes those for the operating systems themselves, as well as for the other software that you install on those virtual machines.

Let's consider the operating system first. Having the one copy of Windows that shipped with your computer is not enough. That's probably only one license, and it is being used as your host OS. Depending on the operating system that you are using for your virtual machines, you may need one additional license for each VM.

For example, if you want to install Windows in a virtual machine, you must have a separate license for each virtual machine. No, you cannot rationalize that you only run one at a time. Read the Windows end user license agreement (EULA). It specifically covers virtual machines, and in almost every case, you will need a separate license for each one.

Fortunately, for us developers, there is a rock solid solution from Microsoft, and it is called Microsoft Developers Network, or MSDN. Your MSDN subscription includes the rights to use certain Microsoft products under certain conditions. And even the lowest level of MSDN subscription, MSDN Operating Systems, includes licenses for the current version of Windows. These licenses are specifically designed for testing and development. And, guess what, if you are using your VMs for building and testing software, you are using those licenses for testing and development.

(Disclaimer: If you have specific questions about the various MSDN subscriptions, do not take my word for it. Go to the Microsoft site and read what they say about the various MSDN subscriptions. It is possible that my subscription is different from your subscription.)

At least with the MSDN subscription that I have, I get up to an initial 10 licenses of Windows 7 Ultimate for use in testing and development environments. However, if I use those licenses up, I can request another 10, and another 10 after that, though I cannot imagine needing that many VMs.

My rather low-level subscription only gives me access to the most current operating system. However, some of the more deluxe subscriptions of MSDN give you access to current and past operating systems, which, depending on the type of testing you need to do, might be perfect. Likewise, if your testing and development involves other products, like SharePoint, Azure, Microsoft Office, SQL Server, and other Microsoft products, there are MSDN subscriptions that includes testing and development licenses for those products as well.

As far as AntiVirus is concerned, I initially purchased a 3-license pack of Norton Antivirus. That worked fine, until I wanted to install my fourth VM. At that point I got frustrated and began looking for an alternative, as I didn't want to shell out another $100 for 3 more licenses.

Fortunately, there is a free alternative from Microsoft called Microsoft Security Essentials. It works on Windows 7, Windows Vista, and XP, and I am now a big fan. It installs quickly, has a pretty small footprint, and doesn't pester me as much as Norton did. In fact, I am so happy with it that I removed Norton AntiVirus from my host OS (even though I still had more than half a year left on my license), and installed Windows Security Essentials there.

Quickie Installations


One feature of VMWare that I really didn't start using until recently was the ability to install software into a VM from an ISO image (an archive format used for optical disks). You can install either an entire operating system, or any other software, using an ISO image, which saves a lot of time.

For example, I recently created a new VM using Windows 7 Ultimate x86. I downloaded the ISO image from the MSDN Web site, and saved it to the hard disk on my host OS. I then created a new virtual machine in VMWare. When prompted, I indicated that I was going to install from an ISO image. Before even starting, VMWare prompted me for my product key, which I entered. It then proceeded to install the operating system, and was done in no time.

Next, I wanted to install Delphi XE under this VM. I had an ISO image of that installer, and instructed VMWare to treat that image on my host hard drive as the CD drive in the VM. Again, the installation was fast and easy.

Snapshots and Linked Clones


Another one of the benefits of VMs is that you can easily perform "what if" tests without much risk of doing harm. Two of the tools provided in VMWare for this purpose are snapshots and linked clones.

Imagine that someone has recommended that you try a new profiling tool, but you do not have much experience with it. Wouldn't it be nice to install it without having to worry about what side effects it might cause, or what trash it may leave behind if you end up uninstalling it?

The most foolproof technique, in my book, is to create a linked clone. A linked clone is a new VM based on an existing one. The big deal here is that a linked clone is very small, since it is relying on a snapshot of an existing VM. Once you have created the linked clone, which takes less than a minute, you can install the suspect piece of software in it and test away.

Once you are satisfied that the software you are testing is going to work for you, you can re-install it into your regular VM and delete the clone. If you decide that the software is not for you, simply delete the linked clone. In any case, what you installed into the linked clone cannot affect the VM you cloned it from.

Snapshots are designed to give you a similar capability. A snapshot is a placeholder into the state of a virtual machine. In theory, you should be able to create a snapshot, and then install the software you want to test. If things go badly, you simply revert the virtual machine to the snapshot, and no more pain.

This all sounds well and fine, and I'm sure it works, but I don't have a whole lot of experience with snapshots, so I am still a little gun shy.

Shared Folders


The final feature that I've started using extensively is shared folders. A shared folder is a location (drive, folder) on your host OS that can be seen by one or more virtual machines.

This is the place where I put files and programs that I am likely to use from any given virtual machine, such as printer drivers, 7-zip, and other utilities. I then use the Shared Folders option from the Hardware tab of the VMWare Virtual Machine Settings dialog to configure folder sharing.

For each VM, I select the Always enabled radio button. I then use the Browse button to select the shared folder. I also enable the Map as a network drive in the guest operating system checkbox. When done, the shared folder appears as a network share in each of the VMs.

Even without this feature, it is easy to copy and paste between VMs. However, having files that you typically use always available from within each VM is even better.

Still, There Are Drawbacks


Don't get me wrong, I have no regrets about going down this road. So far, so good. But there are some issues that I still grapple with.

For one, there are those infernal updates. Holy smokes! Every time Microsoft comes out with a Windows update I've got all these different VMs nagging me to update them. And, once or twice, the update has messed things up a bit.

I've addressed this issue by altering my behavior. I don't update Windows every time it asks. In fact, I normally ignore Windows updates until I have some time to really deal with them. Next, before I do the updates, I copy the virtual machines that I am using in my daily work to one of my backup drives. Then, I systematically open each one of these VMs and perform the update.

For those VMs that I use only on occasion, I wait until I need them. For example, if I am asked to teach a class on Delphi 7, I will backup and then update that VM in preparation for the class.

As a result, some VMs don't get updated that often. However, if an update causes a problem, I've always got that pre-update copy that I can rely upon.

Conclusion


I'm running into a lot of developers who are approaching their development environment this way, so I'm not so cocky as to think that I made this thing up. However, I am happy to share my experiences in this realm, and hope that if you are still developing in your base OS that you might have gained some insight into whether this approach might better serve you or not

10 comments:

  1. I've been working this way for about 5 years now. While I originally started as you are now, with a VM per development environment, these days I dedicate a VM to each of my customers. The VM contains all software I need to service that customer and that's all. It's great being able to suspend the VM exactly at the point I'm up to and then come back straight to that point weeks/months later when the customer rings again.

    I also use WinXP as my guest OS which allows me to keep my VMs with just 1GB of RAM. I rarely receive out of memory errors. When I upgrade to a new laptop that supports >4GB RAM I'll maybe bump that up to 1.5GB just to be sure.

    Don't be scared of snapshots, in my 5 years I've had only one SNAFU with them that wasn't caused by my own stupidity. If you're making regular backups of your VMs you'll be fine. You should favour making your snapshots while the VMs are turned off, live snapshots work fine but eat a lot of disk space.

    You can improve the performance of running a VM on a laptop by copying it to an external USB/eSATA HD and running it from there. Sounds as though it would be slower but by moving it to a different drive, freeing your C drive for host OS pagefile and system file access only, you get a more responsive VM.

    You can do a similar trick with your linked clones. Leave the base VM on the C drive and move the linked clone to an external drive. All writes to the linked clone will happen on the external drive while the majority of the reads will come from the base VM on the C drive.

    There's also a "Working directory" option in the VM settings which lets you achieve a similar result when dealing with just a single VM and it's snapshots.

    MaxiVista is a great little software package for increasing the number of monitors you can attach to your laptop. You install it on the host, attach to the additional monitors, then start VMWare and it can use all the monitors attached to your laptop (my laptop connects to 1 monitor directly and 2 monitors of the PC next to me via the network).

    I have AV on the host but I don't bother about it for any VM that doesn't receive email.

    I know I'm missing a few more tips. I'll come back if I think of them. Thanks for coming and visiting us at ADUG this year.

    Lachlan

    ReplyDelete
  2. Thanks for sharing this.
    Also, another nice thing I like about VM is the fact it's very useful to test the installation procedure of the application or application suite your wrote. If the application is more complex then a single executable, when there are a few executables, various folder created, driver to install, shortcut to create, program and icon association, etc... to be able to test and re-test your installation procedure of your baby, it's nice to be able to try that in virtual machine that will also quickly allow you to keep retrying on brand fresh new machine. you are sure that if it works, it's not because a sub-previous installation attempt did something your forgot and not done in the version after. It's really useful for that instead of trying to find a colleague that has a computer where your application never run and attempt to test your install on his machine!!! ;-)

    ReplyDelete
  3. Awesome post and this has inspired me to fully utilise my VM to its capacity.

    One suggestion... Snapshots could be the cure for your windows update drawback. The following method should work:

    1. Leave windows updates on for download/notification only.
    2. When you are ready to install them, firstly create a snapshot for each VM called "Windows Updates dd/mm/yyyy"
    3. Go ahead and install the updates.
    4. If any VM is corrupted after the windows updates, simply revert to the snap shot and problem solved!

    Thanks for such a detailed post.
    Rick.

    ReplyDelete
  4. Great post which basically sums up my experience too. I've got the same problems as you with updates. Have you considered going to the next step with VSphere. It looks like it better handles those.

    ReplyDelete
  5. Don't be afraid of snapshots. It just works.

    When you do something you may want to revert from just make a snapshot, it is a lot quicker than cloning of backing up a VM. Going back to an old snapshot is also pretty quick.

    About Windows licenses: What about having 1 VM with a lot of branched snapshots? You can easily have a branch with D7, one with Visual Studio, etc. Then you need just 1 VM. I wonder what Microsoft says about that, but it would not surprise me if they aren't sure either.

    ReplyDelete
  6. I also have been running in this mode, and it has saved me countless hours. One specific example would be when my laptop completely died. I was able to pop out my HD drive and copy the VM to a replacement laptop (a sata to usb converter just happened to be also available). I was up and running on new hardware without any problems, and able to make my deadline.

    ReplyDelete
  7. I couldn't agree more! Thank you for this great summary.

    With snapshots, you can fork/branch your development environment (just like you do with your code in the VCS). If you need to change your dev.env. due to some new features, you can easily switch between the experimental dev.env. snapshot and the production dev.env. snapshot.

    Another feature that none of you have mentioned (and I have to admit that my own experience is limited), is the capability to set up a virtual network.

    ReplyDelete
  8. Three things that speed up VMs a lot:

    - put your VMs on a different drive than the one your host OS boots from (I haven't had a DVD/CD player in my laptop for at least 5 years; swapping that for an extra HDD is so much more convenient, if I need a DVD player, I can use a USB powered external one)

    - Use solid-state 'drives' or hybrid drives (like the Seagate Momentus XT ST95005620AS (Hybrid), 500GB SATA-300, 32MB, 7200rpm drive) to speed up HDD access.

    - Put as much RAM in your host as possible.

    --jeroen

    ReplyDelete
  9. I have used VMs for this kind of work for a while (probably about 5 years or so). The VM software is getting better and better - I use VirtualBox which is pretty fast and stable although shared folders was always very slow - much faster in the most recent (Oracle) releases.

    I even use it on my Netbook - much more convenient that a full size notebook and with 4GB and 64Bit Windows 7 the performance of the guest OS is pretty OK too.

    I rationalised the licencing issue by always assuming the restriction was on how many copies of the OS were in use at any one time. Although I have several instances of (for example) XP using the same licence I only use one of those instances at any one time.

    ReplyDelete
  10. As I mentioned in the original post, licensing is an issue that is relevant when it comes to virtual machines. It depends on your end user license agreement (EULA). Some licenses permit you to install your software on one or more machines (and many agreements specify what a machine is, and in some cases, a virtual machine has the same status as your host). All I am saying is that if you want to ensure that you are legit, sofware-wise, read the EULA.

    ReplyDelete