Violence is the only solution (sometimes)

Step 0: Backup everything you'd hate to lose.

This is a must! The steps I describe below are crazy stuff. Don't try them without a full backup and/or a promise from your VDS provider to restore your node free of charge when it's absolutely hosed.

Step 1: Know your environment.

The Gigahost VDS runs on OpenVZ. An OpenVZ VDS (which they call a "virtual environment", VE) gives you just one partition (a simfs shim to the underlying filesystem in the host node) to work with, and the swap partition is also shared from the host node which you can't touch, so the conventional partition switcheroo (i.e. install a rescue distro in a new partition, boot from that partition, wipe out original partition, install Debian there) is out of the question.

And this being a VDS and not an actual machine, I do not have physical access to the console or the CD-ROM drive, of course. The only way I can get to my VDS is by ssh.

So the only option is to replace the distro in-place, in the same partition, while the old distro is running, which at first sounds like absolute insanity. But if you think about how Linux works, it's entirely doable, with care.

I'm going to outline what I did in my specific case (OpenVZ VE, from CentOS to Debian), but I guess the technique can be easily generalised to many other situations where you have no physical access and no extra partition to spare.

Step 2: Get Debian.

The general idea here is to create an image of an entire Debian installation under some subdirectory in your original VE. I'm going to use /newdist. Just mkdir /newdist and treat it as the root where your Debian distro will reside. (For those familiar with the partition switcheroo operation, this is like mounting a new partition on /newdist and installing Debian into that partition.)

You have quite a few options to go about actually creating a Debian installation under /newdist:

• If you like the natural fragrance of a pristine download, you can use debootstrap to build the base install under /newdist from scratch. This will take quite a while, though. You also need to chroot into the debootstrapped installation, do your base-config, and install the ssh package so that you can gain access after the switch.
• If you already have a Debian box running, you can slurp the entire directory tree from root downwards from that machine into /newdist.
• There is an even easier option in certain cases, and luckily for me, OpenVZ is one of them. There are a number of pre-built VE images for OpenVZ. Among these you will find a suitable Debian image for your CPU arch. The best thing about using these images is that the bootscripts and network scripts etc. are already tweaked with OpenVZ-specific changes.

So I downloaded the i386 Debian VE image, untarred it under /newdist, and it's almost good to go.

Step 3: Prepare the Debian environment.

The Debian tree you have just slurped into /newdist is either a fresh base image or an image intended for some other box. At the very least you should look into the following things to make sure the image is set up for your specific environment:

• Root password. If you're nuts like me, you can keep your existing root password by replacing the encrypted root password in /newdist/etc/shadow with that in /etc/shadow. (Make sure both your old and new distros support the same sort of encrypted passwords, though.) Or chroot into the Debian tree and do a proper password change from within.
• ssh host keys. I'll assume your Debian distro has ssh installed. Make sure it has a set of host keys in /newdist/etc/ssh. If you want to, you can keep the host keys from your old distro by copying /etc/ssh/ssh_host*_keys into /newdist/etc/ssh.
• Networking. For most VDS plans, you should have been assigned a static IP. Edit /newdist/etc/network/interfaces accordingly. You may need to run route and look at your old distro's network config scripts to see if any funky routing is needed, as is often the case for VDS environments. For OpenVZ VE's, your /newdist/etc/network/interfaces would most probably look like this:

  auto venet0 lo
  iface venet0 inet static
    address your.static.ip.here
    netmask 255.255.255.255
    broadcast 0.0.0.0
    up route add -net 191.255.255.1 netmask 255.255.255.255 dev venet0
    up route add default gw 191.255.255.1
  iface lo inet loopback

• Devices. Your VDS may need special devices. Compare /newdist/dev and /dev to see if any devices need to be created. MAKEDEV accordingly. (My OpenVZ-customised Debian already took care of this.)
• Boot scripts: Your VDS may require special scripts to be run at boot, shutdown or reboot. For example, my VDS has a script in /etc/rc6.d to write to a certain file just before reboot, without which the reboot wouldn't work. Copy any such custom scripts to /newdist/etc/rc*.d. (You can usually tell the custom scripts from the regular ones supplied by distro packages by the fact that custom scripts in /etc/rc*.d are often files instead of symlinks.)
• /newdist/etc/fstab. In OpenVZ you don't really need a fstab. (Mine is blank.) Your environment may vary. Compare your /newdist/etc/fstab and /etc/fstab to make sure there aren't any showstoppers.
• Other config files - things that are nice to take care of now, but not life-threatening: /newdist/etc/resolv.conf, /newdist/etc/hostname, /newdist/etc/localtime, /newdist/etc/timezone etc.

Your Debian installation in /newdist should now be ready to completely replace CentOS! But not so fast, cowboy...

Step 4: Make sure you're not about to be screwed.

We're about to switch from CentOS into Debian. Roughly, the game plan for the switch (Don't do it yet! I'm only describing it so I can explain stuff) is: (1) mkdir /olddist; (2) For everything under root except /lib, /proc, /newdist and /olddist, move under /olddist; (3) For every directory under /newdist except /lib and /proc, move up to root; (4) Update /lib using the library files from /newdist/lib.

The one problem we absolutely don't want to run into during the process is missing dynamic libraries; if this ever happens, we're irreversibly stranded. (We have no physical access and no alternative rescue distro to boot into, remember.) So we need to make very sure we don't move /lib at point (2), and take extreme care to update /lib in-place at point (4). It's also a good idea to get into the habit of running commands using absolute paths -- between point (2) and point (3), /bin/mv will have become /olddist/bin/mv so you can no longer depend on $PATH to find mv for you.

We also don't want non-essential services running, which may give us problems by locking files, writing to the filesystem, and creating extra processes which try to link to dynamic libraries that are no longer there etc.. So identify and stop those services. Do a runlevel to see which level you're in. If in level 3, ls -aF /etc/rc.d/rc3.d and find out which services are started in this level. Terminate all of them except ssh by doing /etc/init.d/the_service stop one by one. If your box was set up by a diabolical hacker, some services might have been set up via a "respawn" entry in /etc/inittab. You need to find these as well, edit out those lines from /etc/inittab, and do a telinit q to re-read /etc/inittab and stop those services.

Step 5: Switch!

The fun part. Feel the adrenaline rush! I've already laid out the game plan just now, but here's a more detailed breakdown:

• cd /; mkdir /olddist;
• Do a ls -aF / and write down with pen and paper the contents of the root directory. Yes, pen and paper. This way you really slow down and think about what you are doing.
• Cross out /lib, /proc, /newdist and /olddist from your list.
• /bin/mv your_list_of_things_in_root_without_lib,_proc,_newdist_and_olddist /olddist
• cd /newdist
• /olddist/bin/mv your_list_of_things_in_/newdist_without_lib_and_proc /
• cd /; /olddist/bin/tar -C /newdist -cf - lib | tar --preserve -xf -

The switch is done! You now have Debian in the root, and the old CentOS in /olddist. But the system is a bit wacky. For example, init thinks that /etc/inittab is in /olddist/etc/inittab; ssh can't create any pseudo ttys in /dev/pts. If you plan to stay for a while and do some final checking, you'd better do these to make sure you can get back in in case ssh drops your connection:

• /bin/mount -t devpts devpts1 /dev/pts
• Do a ps ax to find out the pid of the old ssh service - Be careful! Make sure this is the parent ssh process, not the child that immediately runs your shell! Do a kill -TERM pid_of_ssh_parent. Immediately, launch the Debian version of ssh with /etc/init.d/ssh start
• While the old ssh session is still active, try logging into a new ssh session. If the new session succeeds, you can safely close the old one.

Then continue with whatever housekeeping you want to do before the next step, which is...

Step 6: Reboot!

Once you're ready to press the big red button, do your /sbin/init 6, or call your tech support, or do whatever your VDS requires you to do in order to reboot.

Hopefully, in a few minutes, your new Debianised (civilised!) VDS will be ready for you. You may now retain /olddist for keepsake, backup old data from it, or just blow it away. At this stage, you may also want to compare /lib with /newdist/lib and remove all the extra fluff left around in /lib by the old distro, before you upgrade the libc6 package after which /lib will be mangled beyond recognition.

Enjoy your brand new Debian node. Or, if you now find out your host won't boot, enjoy the recovery process. (You have backed up, haven't you?)

Assumptions

To keep the scenario simple, I'm assuming you don't need to back up any user accounts. If you have such needs, you'll have to setup /newdist/etc/passwd and /newdist/etc/shadow properly before the mass migration, and take care to preserve stuff from /root and /home/* as required.

I'm also assuming your existing distro is equipped with the usual stuff in the usual place - /bin/mv, /bin/tar, /(s)bin/mount, dynamically linked to a reasonably modern libc in /lib etc.. Distros that are too old / esoteric may not work.