Table of Contents

"Upgrading" host vicki from Debian 5.0 "lenny" to Debian 6.0 "squeeze"

Some earlier communications on the matter:

pre-"upgrade" overview of hosts:

draft/outline of upgrade plans/procedure & background:

# hostname --fqdn; pwd -P; expand -t 2 < 0010_a_general_plan_or_outline
vicki.sf-lug.com
/root/upgrades/5.0_lenny_to_6.0_squeeze
SF-LUG & BALUG: System OS upgrades *soon*(?) - volunteer(s)?

Jim, et. al.,

Do we have a quorum of volunteers (or should we also try to add a person
or two)?  In this case, I'm specifically thinking colo box, physical
access and associated systems administration stuff (there's also lots
that can be done mostly remotely).

Anyway, I see some fairly major upgrades due in our near future.
Impacted are:
SF-LUG:
sflug (guest on vicki, hosts [www.]sf-lug.com)
vicki (host for the above)
BALUG:
vicki (noted above, hosts the immediately below)
balug-sf-lug-v2.balug.org (guest on vicki, hosts lot of BALUG
   production)
aladfar.dreamhost.com. (hosted, will be upgraded/replaced for us, hosts
   [www.]balug.org, etc.)

Security support for Debian 5.0 "lenny" ends *soon* (2012-02-06).
To the extent feasible, we should upgrade the relevant systems soon,
preferably before that date, if that's doable, but if not, soon
thereafter.

Also planning out, reviewing & discussing those upgrades, etc. at:
o Noisebridge Linux Discussion 2012-01-25
o SF-LUG 2012-02-05

Roughly, I have in mind (what I'd like to do):
o There isn't any official supported upgrade path from i386 to amd64
o the Silicon Mechanics physical box is and will run amd64/x86_64
o the Silicon Mechanics physical box supports hardware virtualization
o suitably backup (including on-disk as feasible)
o generally prepare for upgrades
o do "upgrades" as follows:
   o vicki:
     o DONE: backup / move / "shove" stuff around beginning of disk
       suitably out-of-the-way (on-disk backups / access to existing
       data)
       IMPLEMENTED AS:
       sd[ab]1 md0 RAID1 >243MiB available for use for upgrade/install
        (/boot data copied to /boot.2012-01-30.tar.gz)
       sd[ab]2 md1 RAID1 >16GiB available for use (data relocated; md1
         removed from LVM; md1 data wiped to all binary zeros)
     o install Debian 6.0.4 amd64, using beginning area(s) of disks
       (md[01] (sd[ab]12) and area preceding sd[ab]1 (boot blocks, MBR,
       partition table) - partition layout to be preserved, all data on
       all partitions to be preserved except sd[ab]12 via md[01] will be
       used for /boot and LVM2 respectively)
       general architecture layout mostly quite as before (everything
       mirrored, separate /boot, rest under LVM2, separate filesystems,
       etc.)
     o install/configure vicki as above to fully support qemu-kvm, Note
       that on amd64, and with hardware virtualization, that will allow
       vicki to support i386 and amd64 images under qemu-kvm.
   o sflug & balug-sf-lug-v2.balug.org:
     o once the above vicki upgrades are done, sflug and
       balug-sf-lug-v2.balug.org can be dealt with remotely, however it
       may be desirable, in the interest of time, to convert sflug to
       run under qemu-kvm and verify such is operational before leaving
       the site.
     o at minimum, before departing site, it should be ensured that host
       vicki reboots properly to provide remote ssh access to it, and
       that it is suitably configured to run i386 and amd64 images under
       qemu-kvm.
     o sflug & balug-sf-lug-v2.balug.org can each be dealt with
       separately by their primary/lead sysadmin(s) as may be desired, in
       general for them, I'd probably recommend proceeding as follows:
       o get the existing xen guests converted to qemu-kvm and then
         running again, more-or-less as they were (will require some
         adjustments - most notably boot bits)
       o upgrade guests to Debian 6.0.4 (or latest 6.0.x)
       o optional: change guests from i386 to amd64, use above guests
         as reference installations, and do an install/merge to get the
         guest(s) as desired to amd64 architecture.

Security support of Debian GNU/Linux 5.0 (code name "lenny") will be
terminated 2012-02-06.
Debian released Debian GNU/Linux 5.0 alias "lenny" 2009-02-14.
Debian released Debian GNU/Linux 6.0 alias "squeeze" 2011-02-06.

references:
http://lists.debian.org/debian-security-announce/2011/msg00238.html
# 

some notes/points/questions/observations/commentary/etc. from planning (meeting(s), etc.)

disk layout, etc. details:

# hostname --fqdn; pwd -P; more 00[23]* | expand -t 2
vicki.sf-lug.com
/root/upgrades/5.0_lenny_to_6.0_squeeze
::::::::::::::
0020_vicki-pre-disk-analysis
::::::::::::::
//hard drive partitions, we have:
# 2>>/dev/null sfdisk -uS -l /dev/sda; 2>>/dev/null sfdisk -uS -l /dev/sdb

Disk /dev/sda: 30401 cylinders, 255 heads, 63 sectors/track
Units = sectors of 512 bytes, counting from 0

   Device Boot    Start       End   #sectors  Id  System
/dev/sda1            63    498014     497952  fd  Linux raid autodetect
/dev/sda2        498015  35648234   35150220  fd  Linux raid autodetect
/dev/sda3      35648235 488392064  452743830   5  Extended
/dev/sda4             0         -          0   0  Empty
/dev/sda5      35648298  92213099   56564802  fd  Linux raid autodetect
/dev/sda6      92213163 148777964   56564802  fd  Linux raid autodetect
/dev/sda7     148778028 205342829   56564802  fd  Linux raid autodetect
/dev/sda8     205342893 261907694   56564802  fd  Linux raid autodetect
/dev/sda9     261907758 318472559   56564802  fd  Linux raid autodetect
/dev/sda10    318472623 375037424   56564802  fd  Linux raid autodetect
/dev/sda11    375037488 431602289   56564802  fd  Linux raid autodetect
/dev/sda12    431602353 488167154   56564802  fd  Linux raid autodetect
/dev/sda13    488167218 488392064     224847  fd  Linux raid autodetect

Disk /dev/sdb: 30401 cylinders, 255 heads, 63 sectors/track
Units = sectors of 512 bytes, counting from 0

   Device Boot    Start       End   #sectors  Id  System
/dev/sdb1            63    498014     497952  fd  Linux raid autodetect
/dev/sdb2        498015  35648234   35150220  fd  Linux raid autodetect
/dev/sdb3      35648235 488392064  452743830   5  Extended
/dev/sdb4             0         -          0   0  Empty
/dev/sdb5      35648298  92213099   56564802  fd  Linux raid autodetect
/dev/sdb6      92213163 148777964   56564802  fd  Linux raid autodetect
/dev/sdb7     148778028 205342829   56564802  fd  Linux raid autodetect
/dev/sdb8     205342893 261907694   56564802  fd  Linux raid autodetect
/dev/sdb9     261907758 318472559   56564802  fd  Linux raid autodetect
/dev/sdb10    318472623 375037424   56564802  fd  Linux raid autodetect
/dev/sdb11    375037488 431602289   56564802  fd  Linux raid autodetect
/dev/sdb12    431602353 488167154   56564802  fd  Linux raid autodetect
/dev/sdb13    488167218 488392064     224847  fd  Linux raid autodetect

//or if we present that data a bit differently, to show just how
//identical the partitioning on the two /dev/sd[ab] disks is:
Disk /dev/sd[ab]: 30401 cylinders, 255 heads, 63 sectors/track
Units = sectors of 512 bytes, counting from 0

   Device Boot    Start       End   #sectors  Id  System
/dev/sd[ab]1            63    498014     497952  fd  Linux raid autodetect
/dev/sd[ab]2        498015  35648234   35150220  fd  Linux raid autodetect
/dev/sd[ab]3      35648235 488392064  452743830   5  Extended
/dev/sd[ab]4             0         -          0   0  Empty
/dev/sd[ab]5      35648298  92213099   56564802  fd  Linux raid autodetect
/dev/sd[ab]6      92213163 148777964   56564802  fd  Linux raid autodetect
/dev/sd[ab]7     148778028 205342829   56564802  fd  Linux raid autodetect
/dev/sd[ab]8     205342893 261907694   56564802  fd  Linux raid autodetect
/dev/sd[ab]9     261907758 318472559   56564802  fd  Linux raid autodetect
/dev/sd[ab]10    318472623 375037424   56564802  fd  Linux raid autodetect
/dev/sd[ab]11    375037488 431602289   56564802  fd  Linux raid autodetect
/dev/sd[ab]12    431602353 488167154   56564802  fd  Linux raid autodetect
/dev/sd[ab]13    488167218 488392064     224847  fd  Linux raid autodetect

# 2>>/dev/null sfdisk -uS -d /dev/sda; 2>>/dev/null sfdisk -uS -d /dev/sdb
# partition table of /dev/sda
unit: sectors

/dev/sda1 : start=       63, size=   497952, Id=fd
/dev/sda2 : start=   498015, size= 35150220, Id=fd
/dev/sda3 : start= 35648235, size=452743830, Id= 5
/dev/sda4 : start=        0, size=        0, Id= 0
/dev/sda5 : start= 35648298, size= 56564802, Id=fd
/dev/sda6 : start= 92213163, size= 56564802, Id=fd
/dev/sda7 : start=148778028, size= 56564802, Id=fd
/dev/sda8 : start=205342893, size= 56564802, Id=fd
/dev/sda9 : start=261907758, size= 56564802, Id=fd
/dev/sda10: start=318472623, size= 56564802, Id=fd
/dev/sda11: start=375037488, size= 56564802, Id=fd
/dev/sda12: start=431602353, size= 56564802, Id=fd
/dev/sda13: start=488167218, size=   224847, Id=fd

# partition table of /dev/sdb
unit: sectors

/dev/sdb1 : start=       63, size=   497952, Id=fd
/dev/sdb2 : start=   498015, size= 35150220, Id=fd
/dev/sdb3 : start= 35648235, size=452743830, Id= 5
/dev/sdb4 : start=        0, size=        0, Id= 0
/dev/sdb5 : start= 35648298, size= 56564802, Id=fd
/dev/sdb6 : start= 92213163, size= 56564802, Id=fd
/dev/sdb7 : start=148778028, size= 56564802, Id=fd
/dev/sdb8 : start=205342893, size= 56564802, Id=fd
/dev/sdb9 : start=261907758, size= 56564802, Id=fd
/dev/sdb10: start=318472623, size= 56564802, Id=fd
/dev/sdb11: start=375037488, size= 56564802, Id=fd
/dev/sdb12: start=431602353, size= 56564802, Id=fd
/dev/sdb13: start=488167218, size=   224847, Id=fd

//excepting extended partition, all logical and non-zero length primary
//partitions paired up between the sda and sdb devices partitions as md
//raid1 devices:
# mdadm --verbose --examine --scan
ARRAY /dev/md0 level=raid1 num-devices=2 UUID=aa643e53:bf543ced:313266d4:d5715d2d
   devices=/dev/sdb1,/dev/sda1
ARRAY /dev/md1 level=raid1 num-devices=2 UUID=042ceb88:cc906844:9895a7df:5145afdc
   devices=/dev/sdb2,/dev/sda2
ARRAY /dev/md2 level=raid1 num-devices=2 UUID=0246205e:28218c5d:abb3665b:0b743010
   devices=/dev/sdb5,/dev/sda5
ARRAY /dev/md3 level=raid1 num-devices=2 UUID=2d2f4ea7:c64ec7bc:abb3665b:0b743010
   devices=/dev/sdb6,/dev/sda6
ARRAY /dev/md4 level=raid1 num-devices=2 UUID=865ccab9:d4b974f9:abb3665b:0b743010
   devices=/dev/sdb7,/dev/sda7
ARRAY /dev/md5 level=raid1 num-devices=2 UUID=658546bb:0f1cd14a:abb3665b:0b743010
   devices=/dev/sdb8,/dev/sda8
ARRAY /dev/md6 level=raid1 num-devices=2 UUID=a36c8141:20c78911:abb3665b:0b743010
   devices=/dev/sdb9,/dev/sda9
ARRAY /dev/md7 level=raid1 num-devices=2 UUID=fa9405b0:a35f0051:abb3665b:0b743010
   devices=/dev/sdb10,/dev/sda10
ARRAY /dev/md8 level=raid1 num-devices=2 UUID=28693012:1c28e9e4:abb3665b:0b743010
   devices=/dev/sdb11,/dev/sda11
ARRAY /dev/md9 level=raid1 num-devices=2 UUID=bdc04439:43e908da:abb3665b:0b743010
   devices=/dev/sdb12,/dev/sda12
ARRAY /dev/md10 level=raid1 num-devices=2 UUID=c828b7de:3f56cc42:abb3665b:0b743010
   devices=/dev/sdb13,/dev/sda13

///dev/md0 is used for boot:
#fgrep /boot /etc/fstab
/dev/md0        /boot           ext3    nosuid,nodev,ro,noatime        0       2
# df -k /boot
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/md0                241036     57135    171457  25% /boot


//md[1-6] used for LVM
# for tmp in /dev/md[1-9] /dev/md[1-9][0-9]; do echo $(pvdisplay "$tmp" | fgrep -e 'PV Name' -e 'VG Name'); done; unset tmp
PV Name /dev/md1 VG Name vg00
PV Name /dev/md2 VG Name vg-balug
PV Name /dev/md3 VG Name vg-sflug
PV Name /dev/md4 VG Name vg-balug
PV Name /dev/md5 VG Name vg-balug
PV Name /dev/md6 VG Name vg-local
  No physical volume label read from /dev/md7
  Failed to read physical volume "/dev/md7"

  No physical volume label read from /dev/md8
  Failed to read physical volume "/dev/md8"

  No physical volume label read from /dev/md9
  Failed to read physical volume "/dev/md9"

  No physical volume label read from /dev/md10
  Failed to read physical volume "/dev/md10"
//are md[7-9] and/or md10 in use for anything?
//not used for dom0 swap
//not mounted
//not referenced in /etc/fstab
//not used by xen guests or dom0 for guests
//fuser and fuser -m show nothing having them open
//apparently /dev/md[7-9] and /dev/md10 not in use (free/available)
::::::::::::::
0030_vicki_initial_disk_prep
::::::::::::::
//before:
sda1 sdb1 md0 /boot
sda2 sdb2 md1 vg00
sda5 sdb5 md2 vg-balug
sda6 sdb6 md3 vg-sflug
sda7 sdb7 md4 vg-balug
sda8 sdb8 md5 vg-balug
sda9 sdb9 md6 vg-local
sda10 sdb10 md7 (unused)
sda11 sdb11 md8 (unused)
sda12 sdb12 md9 (unused)
sda13 sdb13 md10 (unused)
//after:
sda1 sdb1 md0 /boot
sda2 sdb2 md1 (unused)
sda5 sdb5 md2 vg-balug
sda6 sdb6 md3 vg-sflug
sda7 sdb7 md4 vg-balug
sda8 sdb8 md5 vg-balug
sda9 sdb9 md6 vg-local
sda10 sdb10 md7 vg00
sda11 sdb11 md8 (unused)
sda12 sdb12 md9 (unused)
sda13 sdb13 md10 (unused)
# 

host networking bits

IP Addresses

# hostname --fqdn; pwd -P; expand < 0050_networking
vicki.sf-lug.com
/root/upgrades/5.0_lenny_to_6.0_squeeze
//still quite accurate:
http://www.wiki.balug.org/wiki/doku.php?id=system:ip_addresses
IPv4 addresses (this subnet):
208.96.15.248/29 network:
208.96.15.248 network
208.96.15.249 Default Gateway
208.96.15.250 "vicki" dom0 (Xen host - Silicon Mechanics box primary IP)
208.96.15.251 (temporarily?) in use by sflug domU
208.96.15.252 sflug domU (Xen "guest" of host "vicki", sf-lug.com., etc.)
208.96.15.253 (useable - reserved for future use(?))
208.96.15.254 balug domU (Xen "guest" of host "vicki", for BALUG use (balug-sf-lug-v2.balug.org, etc.)
208.96.15.255 broadcast
//DNS servers (colo provided): 216.93.160.11 216.93.160.16

//and specific bits shown from host and guests:
$ hostname; /sbin/ifconfig | sed -ne '/HWaddr/{p;n;/inet addr/p;}'; cat /etc/resolv.conf; netstat -nr
vicki
eth0      Link encap:Ethernet  HWaddr 00:30:48:91:97:90
          inet addr:208.96.15.250  Bcast:208.96.15.255  Mask:255.255.255.248
peth0     Link encap:Ethernet  HWaddr 00:30:48:91:97:90
vif2.0    Link encap:Ethernet  HWaddr fe:ff:ff:ff:ff:ff
vif5.0    Link encap:Ethernet  HWaddr fe:ff:ff:ff:ff:ff
search sf-lug.com
nameserver 216.93.160.11
nameserver 216.93.160.16
nameserver 64.81.79.2
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
208.96.15.248   0.0.0.0         255.255.255.248 U         0 0          0 eth0
0.0.0.0         208.96.15.249   0.0.0.0         UG        0 0          0 eth0
$

$ hostname; /sbin/ifconfig | sed -ne '/HWaddr/{p;n;/inet addr/p;}'
balug-sf-lug-v2.balug.org
eth0      Link encap:Ethernet  HWaddr 00:16:3e:4f:52:43
          inet addr:208.96.15.254  Bcast:208.96.15.255  Mask:255.255.255.248

$ dig -t A sf-lug.com. +short
208.96.15.252
$ dig -t A www.sf-lug.com. +short
208.96.15.252
$

$ hostname; /sbin/ifconfig | sed -ne '/HWaddr/{p;n;/inet addr/p;}'
sflug
eth0      Link encap:Ethernet  HWaddr 00:16:3e:7d:0c:67
          inet addr:208.96.15.252  Bcast:208.96.15.255  Mask:255.255.255.248
eth0:0    Link encap:Ethernet  HWaddr 00:16:3e:7d:0c:67
          inet addr:208.96.15.251  Bcast:208.96.15.255  Mask:255.255.255.248
$

//qemu-kvm doesn't automagically do quite as much bridge setup for us,
//so we'll need to do a bit more of that manually, for Debian 6.0 squeeze,
//we'll need at least package bridge-utils on vicki
//relevant networking file bits should look like:

//based on relatively similar Debian GNU/Linux 6.0.4 amd64 host
$ cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo br0
iface lo inet loopback

# The primary network interface
allow-hotplug eth0
iface br0 inet static
        bridge_ports eth0
        address 208.96.15.250
        netmask 255.255.255.248
        network 208.96.15.248
        broadcast 208.96.15.255
        gateway 208.96.15.249
        # dns-* options are implemented by the resolvconf package, if installed
        dns-nameservers 216.93.160.11 216.93.160.16
        dns-search sf-lug.com
$
# 

xen --> qemu-kvm (specific example sflug):

# hostname --fqdn; pwd -P; expand -t 2 < 0060_vicki_sflug_xen2qemu-kvm
vicki.sf-lug.com
/root/upgrades/5.0_lenny_to_6.0_squeeze
The bits on converting sflug from Xen to qemu-kvm
This isn't everything, this is mostly just the less than trival bits
(detailed) example given is sflug, balug would be fairly similar
o sflug is Debian GNU/Linux 5.0.9 i386 (excepting some slight bits that
  may predate that - e.g. some existing low-level bits quite
  interdependent with existing Xen host domU, e.g. kernel - but
  nevertheless, even those bits are at least Debian GNU/Linux 5.0.x
  i386)
o existing sflug has sda1 (/) and sda2 (swap), but no sda presented from
  host to it
o create an sda as follows:
  o LVM volume suitably sized on host to house sflug / filesystem (may
    be resized) and its existing swap
  o partition the above to act like and be configured as an sda (when
    presented from host) with suitably sized sda1 and sda2, using Linux
    and Linux swap types respectively, and with sda1 set as bootable
  o use losetup (with -o and --sizelimit options) to create loop devices
    to access the sda1 and sda2 partitions within the above (note that
    such needs to be sufficiently current losetup - such is well the
    case under Debian 6.0.x, but 5.0.9 lacks the --sizelimit option)
  o use dd with output of the above loop devices and input of existing
    sflug root (/) and swap respectively (or resized root (/)
    filesystem, as applicable)
  o UUIDs should be unique, so we adjust accordingly
  o use tune2fs to change UUID of target root (/) filesystem,
  o use mkswap to recreate target swap
  o use blkid to determine UUID and label(s) of the above targets
  o mount target root filesystem and inspect/edit /etc/fstab appropriately:
    o adjust UUIDs in /etc/fstab as appropriate
    o if/as appropriate, temporarily and suitably comment out anything
      that shouldn't be initially mounted
  o note that the above sets the (virtual) sda up nearly to be bootable,
    but not quite, since those bits weren't written to such a virtual
    drive (nor needed under xen) for the existing xen sflug
  o boot an installation sflug qemu-kvm using virtual sda as noted
    above, using virt-install(1), and suitably adjusted configuration
    approximately as follows - and with CD-ROM (virtual) of:
    Debian GNU/Linux 5.0.9 "Lenny" - Official i386 CD Binary-1 20111001-17:16
    and with suitable ssh X-11 forwarding enabled, etc.:
      DISPLAY=localhost:10.0 XAUTHORITY=/home/mpaoli/.Xauthority \
      virt-install \
              --name=sflug \
              --ram=256 \
              --os-type=linux \
              --os-variant=debianlenny \
              --network=bridge=br0 \
              --hvm \
              --virt-type kvm \
              --cdrom=/var/local/pub/mirrored/cdimage.debian.org/debian-cd/5.0.9/i386/iso-cd/debian-509-i386-CD-1.iso \
              --disk path=/dev/vg-sflug/sflug-sda,format=raw,bus=scsi \
              --wait=-1
    boot the (virtual) guest from CD into Graphical rescue mode
    ...
    Device to use as root file system: /dev/sda1
    Execute a shell in /dev/sda1
  o from outside the chroot(8), bind mount the already mounted bits
    we'll need:
    # mount -o bind /cdrom /target/cdrom
  o and within chroot(8) to keep from driving myself batty:
    sh-3.2# set -o vi
    sh-3.2# FCEDIT=nvi
  o and some other environment bits to avoid problems in our chroot(8)
    with aptitude and friends:
    cd / && exec env -i SHELL=/bin/sh TERM="$TERM" USER="$USER" \
    > PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
    > HOME=/root /bin/sh
  o and again to keep from driving myself batty:
    sh-3.2# set -o vi
    sh-3.2# FCEDIT=nvi
  o edit /etc/apt/sources.list - comment out all active entries
  o use apt-crom to update /etc/apt.sources.list, e.g.:
    sh-3.2# apt-cdrom -m -d=/cdrom add
  o create a backup of /boot "just in case" (and/or for reference):
    sh-3.2# mkdir /boot.bak && (cd /boot && tar -cf - .) |
    > (cd /boot.bak && tar -xf -)
  o (mostly) update aptitude as feasible:
    note that it may be necessary to update /etc/apt/sources.list (e.g.
    to use archive.debian.org for Debian GNU/LInux 5.0.9)
    sh-3.2# aptitude update
    sh-3.2# aptitude safe-upgrade
  o package changes freeing ourselves from xen:
    sh-3.2# aptitude install grub linux-image-2.6-686 libc6 libc6-xen_ \
    linux-image-2.6-xen-686_ linux-image-2.6.26-2-xen-686_ \
    linux-modules-2.6-xen-686_ linux-modules-2.6.26-2-xen-686_
  o install/configure grub:
    sh-3.2# grub-install --no-floppy /dev/sda
  o suitably create/adjust /boot/grub/menu.lst, e.g.:
    sh-3.2# update-grub
  o get our new root filesystem to consistent state - umount it, or
    remount it ro:
    from outside chroot(8):
    # umount /target/cdrom
    from inside chroot(8):
    sh-3.2# exit
  o From the rescue menu, choose:
    Reboot the system
  o if all's well, should get to grub boot prompt,
    boot single user mode
    sanity check system
    # cd / && exec shutdown -h now
  o reconfigure guest to:
    add networking (if not already done)
    add access to cdrom image
    add access to any additional storage as appropriate, e.g.:
    inspect/adjust/update /etc/fstab
  o boot guest
  o suitably update /etc/apt/sources.list
  o sanity check services running from guest (e.g. sshd, DNS & Apache
    from Internet)
  o configure guest to restart upon host reboot, e.g.:
    # virsh autostart sflug
  o shutdown guest(s), reboot host, check that all properly comes up
# 

vicki host install/"upgrade" procedure outline + select details:

# hostname --fqdn; pwd -P; expand -t 2 < 0070_vicki_upgrade_steps
vicki.sf-lug.com
/root/upgrades/5.0_lenny_to_6.0_squeeze
vicki upgrade key steps (outline of general procedure - fair bit of
detail, but skipping details of many of the more obvious/routine steps)

(remaining) pre steps:
o remount /boot ro
o backup /boot data
o backup MBR (440 bytes)
o shutdown VM guests
o attach KVM
o shutdown vicki

install(/merge/upgrade) steps:
o attach bootable USB image: debian-6.0.4-amd64-CD-1.iso
o boot USB
o Advanced options
o Graphical expert install
o ...
o Configure locales, add all en_US* locales
o default locale: en_us.UTF-8
o ...
o Load installer components from CD, include:
  o cfdisk-udeb
  o choose-mirror
  o multipath
  o openssh
  o parted-udeb
  o partman-reiserfs
  o reiserfs-modules
o Configure the network
  o Auto-configure network with DHCP:
    o no
  o IP address: 208.96.15.250
  o Netmask: 255.255.255.248
  o Gateway: 208.96.15.249
  o Name server addresses: 216.93.160.11 216.93.160.16
  o Hostname: vicki
  o Domain name: sf-lug.com
o Set up users and passwords
  o Allow login as root: yes (will change that later)
  o Create a normal user account now?: No
o Configure the clock
  o ...
  o Pacific
o Partition disks
  o Manual
    ONLY TOUCH PARTITIONS #1 AND #2 ON EACH DISK, DO NOT RECREATE OR MOVE
    THEM, THEY MUST REMAIN EXACTLY WHERE THEY ARE AND THE OTHER
    PARTITIONS TOTALLY UNTOUCHED
    o Configure software RAID
    o Keep current partition layout and conigure RAID?: Yes
    o Create MD device
    o RAID1
    o Number of active devices for the RAID1 array: 2
    o Number of spare devices for the RAID1 array: 0
    o select partition 1 from both disks
    o Keep current partition layout and conigure RAID?: Yes
    o Create MD device
    o RAID1
    o Number of active devices for the RAID1 array: 2
    o Number of spare devices for the RAID1 array: 0
    o select partition 2 from both disks
    o Keep current partition layout and conigure RAID?: Yes
    o Finish
    o RAID1 device #0 #1
      o Ext3
      o /boot
      o vicki-boot
    o RAID1 device #1 #1
      o physical volume for LVM
    o Configure the Logical Volume Manager
    o Write the changes to disks and configure LVM?: Yes
    o Create volume group
    o Volume group name: vicki
    o /dev/md1
    o Create logical volume
      logical volume name & size
      root / 1G
      usr 2G
      var 4G
      home 2G
      (we'll add swap later, not now:
        swap1 512M
        swap2 512M
        swap3 512M
        swap4 512M
      )
    o Partition disks
      LVM mount fs type
      root / ext3
      usr /usr ext3
      var /var ext3
      home /home ext3
    o RAID1 device #0 #1
      o Ext3
      o /boot
      o vicki-boot
    o Finish partitioning and write changes to disk
    (continue without swap)
o Install base system
after that, and before kernel, etc.:

ID "merging":
At this point, target filesystems are mounted on/under /target, including
/etc/{passwd,shadow,group,gshadow} files (although root password isn't in
there yet).  Here's where we start UID/GID reconciliation and alignment
and merging in, etc. of IDs from "old" vicki.
merge/reconcile /etc/{passwd,shadow,group,gshadow}

activate and use console session on tty2 (Ctrl+Alt+F2)

start additional md (mdadm) devices, e.g.:
~ # mdadm --assemble /dev/md2 /dev/sd[ab]5
~ # mdadm --assemble /dev/md3 /dev/sd[ab]6
~ # mdadm --assemble /dev/md4 /dev/sd[ab]7
~ # mdadm --assemble /dev/md5 /dev/sd[ab]8
~ # mdadm --assemble /dev/md6 /dev/sd[ab]9
~ # mdadm --assemble /dev/md7 /dev/sd[ab]10
~ # mdadm --assemble /dev/md8 /dev/sd[ab]11
~ # mdadm --assemble /dev/md9 /dev/sd[ab]12
~ # mdadm --assemble /dev/md10 /dev/sd[ab]13

Scan for volume groups:
~ # vgscan
and activate:
~ # vgchange -a y vg00
~ # vgchange -a y vg-local
~ # vgchange -a y vg-sflug
~ # vgchange -a y vg-balug

mount "old" root filesystem and copy ID information to handy location,
and while we're at it, ssh keys:
~ # mkdir /tmp/mnt
~ # mount -o ro /dev/vg00/root /tmp/mnt
~ # (umask 077 && mkdir /target/var/tmp/etc /target/var/tmp/etc/ssh)
~ # (cd /tmp/mnt/etc && cp -p passwd shadow group gshadow /target/var/tmp/etc/)
~ # (cd /tmp/mnt/etc/ssh && cp -p *key* /target/var/tmp/etc/ssh/)
unmount "old" root filesystem and remove our temporary mountpoint:
~ # umount /tmp/mnt && rmdir /tmp/mnt

chroot into target we're building:
~ # cd / && exec chroot /target
optionally give us more "friendly" shell, etc.:
# cd / && exec /bin/bash --posix --login
bash-4.1# PS1='# '
# set -o vi; FCEDIT=vi VISUAL=vi EDITOR=vi export FCEDIT VISUAL EDITOR

mount /proc filesystem:
# mount /proc

VERY CAREFULLY merge in ID information from /var/tmp/etc/* files into
corresponding /etc/* files:
# vipw
# pwconv
# vipw -s
# vigr
# grpconv
# vigr -s
etc. as needed

and NOTE ANY CHANGES WE'LL NEED TO MAKE,
e.g.:
o any changes on NEW filesystem(s) and/or
o any changes on OLD filesystem(s)
At last dry run check on 2012-02-25, the following issues were found,
and their recommended actions:
login/UID conflicts/issues:

libuuid
  new UID/GID on target
  adjust any old data before allowing import, multichown IDspec:
  107,100,107,101

100
  uid conflict (libuuid vs. old Debian-exim)
  change Debian-exim to available <1000 UID
  adjust any old data before allowing import,
  multichown IDspec (replacing Debian-exim with new target UID, e.g. 125):
  100,125

group/GID conflicts/issues:

libuuid
  new GID on target: 101
  adjust any old data before allowing import,
  ,,107,101

crontab
  new GID on target: 102
  adjust any old data before allowing import,
  ,,101,102

102
  gid conflict (vs. old Debian-exim)
  change Debian-exim to available <1000 UID
  adjust any old data before allowing import,
  ,,102,125

umount /proc filesystem from within chroot and exit chroot:
# umount /proc && exit

o continue with installation (kernel, etc.)
o include non-free (in case we need it for, e.g. firmware)
o Software selection
  o Choose software to install:
   select only:
   x SSH server
   x Standard system utilities
   (we'll add other stuff later)
o Is the system clock set to UTC?: Yes
...
reboot to single user
empty contents of /tmp
edit /etc/fstab to use tmpfs for tmp:
tmpfs /tmp tmpfs nosuid,nodev 0 0
mount /tmp
recopy our "old" ssh keys to /etc/ssh/
reboot

Should be able to login via ssh as "regular" user and su to root,
verify that, and if okay, disable root login via ssh

start additional md (mdadm) devices, e.g.:
# mdadm --assemble /dev/md2 /dev/sd[ab]5
# mdadm --assemble /dev/md3 /dev/sd[ab]6
# mdadm --assemble /dev/md4 /dev/sd[ab]7
# mdadm --assemble /dev/md5 /dev/sd[ab]8
# mdadm --assemble /dev/md6 /dev/sd[ab]9
# mdadm --assemble /dev/md7 /dev/sd[ab]10
# mdadm --assemble /dev/md8 /dev/sd[ab]11
# mdadm --assemble /dev/md9 /dev/sd[ab]12
# mdadm --assemble /dev/md10 /dev/sd[ab]13
add them to /etc/mdadm/mdadm.conf:
# /usr/share/mdadm/mkconf | diff --ed /etc/mdadm/mdadm.conf -
... and apply those changes

mount our CD image and configure into apt

install:
  sudo libvirt-bin libvirt-doc virtinst virt-viewer qemu-kvm ed nvi
reconfigure network using br0

convert sflug from Xen to qemu-kvm and configure to autostart

change line in /etc/modules from:
loop
to:
loop max_loop=256
reboot vicki, confirm all comes up as expected

time permitting, work on IMPI

remainder can be handled remotely
# 

checklist/outline of things to bring onsite:

procedure "details" - the "short" list /outline (specifics, etc.)

vicki "upgraded" to Debian GNU/Linux 6.0.4 amd64

balug VM upgraded to Debian GNU/Linux 6.0.4

sflug VM upgrade to Debian GNU/Linux 6.0.4

//I show my comments on lines starting with //
//I don't show all output of all commands (mostly reduced for brevity/clarity)
//from sflug (guest), we have relatively simplistic filesystem and swap setup,
//one "physical" (virtual from host, but effectively physical to guest) hard drive
//with partition for root (/) filesystem, and swap
# sfdisk -uS -l

Disk /dev/sda: 1435 cylinders, 255 heads, 63 sectors/track
Units = sectors of 512 bytes, counting from 0

   Device Boot    Start       End   #sectors  Id  System
/dev/sda1            63  20980889   20980827  83  Linux
/dev/sda2      20980890  22041179    1060290  82  Linux swap / Solaris
/dev/sda3             0         -          0   0  Empty
/dev/sda4             0         -          0   0  Empty
# 
//and to confirm a bit - ignoring virtual filesystems, but our other filesystem(s) and swap:
# mount -t notmpfs,noproc,nodevpts,nosysfs,nousbfs
/dev/sda1 on / type ext3 (rw,errors=remount-ro)
# awk '{if($2=="/")print;};' /etc/fstab
UUID=792a4837-91bc-4b36-bce7-a5a4208845bf / ext3 errors=remount-ro 0 1 #/dev/sda1
# blkid /dev/sda1
/dev/sda1: UUID="792a4837-91bc-4b36-bce7-a5a4208845bf" SEC_TYPE="ext2" TYPE="ext3"
# cat /proc/swaps
Filename                                Type            Size    Used    Priority
/dev/sda2                               partition       530136  0       -1
# awk '{if($3=="swap")print;};' /etc/fstab
UUID=e326206f-55a6-4fbb-8c81-62f8dfb30753 none swap sw 0 0 #/dev/sda2 LABEL=swap
# blkid /dev/sda2
/dev/sda2: TYPE="swap" LABEL="swap" UUID="e326206f-55a6-4fbb-8c81-62f8dfb30753"
# 
//so, quite simple there and no surprises.  And, what does that look like from the host?
# hostname
vicki
# virsh list --all
 Id Name                 State
----------------------------------
  2 sflug                running
 26 balug                running

# virsh dumpxml sflug | fgrep '<source dev='
      <source dev='/dev/vg-sflug/sda'/>
# 
//so, the storage for the sflug VM is on /dev/vg-sflug/sda in VG /dev/vg-sflug,
//how much space used and free (and PE size) for that VG?
# vgdisplay /dev/vg-sflug
  PE Size               4.00 MiB
  Total PE              6904
  Alloc PE / Size       2941 / 11.49 GiB
  Free  PE / Size       3963 / 15.48 GiB
# 
//so, we can see that a fair bit less than half of the VG is used,
//thus we know we've got ample room to complete our backup,
//and also for our snapshot LV to be present;
//snapshot LV doesn't have to consume same space as LV it's a snapshot of,
//as it essentially only tracks changed data,
//and since we'll do this backup relatively quickly,
//and the sflug host won't be changing lots of data nor quickly at that time anyway,
//we don't need much space for the snapshot.
//So, let's do a good test run of our procedure:
//we peek at LV names in the VG, to avoid any accidental name collision:
# ls /dev/vg-sflug
sda  sflug-disk-BACKUP  sflug-disk-BACKUP_2010-04-04
# 
//we examine size of our source LV:
# lvdisplay /dev/vg-sflug/sda
  Current LE             2816
# 
//we create our target LV,
//we'll use -lenny suffix,
//as that's codename of the "old" Debian GNU/Linux 5.0.10 release we're upgrading from,
//and less likely to cause confusion (e.g. backup, which backup, when, of what?)
# lvcreate -l 2816 -n sda-lenny vg-sflug
  Logical volume "sda-lenny" created
# 
//we check on free PEs in our VG - we'll use them all for our snapshot (probably excessive),
//but we only need them for a quite short while anyway,
//so not an issue to be a bit excessive here for a quite short while anyway.
//we reexamine our free PEs in the VG:
# vgdisplay /dev/vg-sflug
  Free  PE / Size       1147 / 4.48 GiB
# 
//we use them all for our snapshot LV:
# lvcreate -l 1147 -n sda-snapshot -s /dev/vg-sflug/sda
  Logical volume "sda-snapshot" created
//I include use of time below, just to see how long it takes (dd also happens to provide similar information)
//and also, especially when operating as superuser (root), we want to be sure our commands are correct,
//Unix, and mostly likewise Linux, tends to presume one knows what one is doing,
//and probably doubly (or more) so for superuser,
//so carefully triple-check the command(s) before viciously striking the <RETURN> key
//(says me ;-) - has saved my butt on more than one occasion)
//here we want to be particularly careful with where we're writing our data to,
//what we remove,
//and also that we in fact pick our correct data source
# time dd if=/dev/vg-sflug/sda-snapshot of=/dev/vg-sflug/sda-lenny bs=4194304
2816+0 records in
2816+0 records out
11811160064 bytes (12 GB) copied, 424.946 s, 27.8 MB/s

real    7m4.957s
user    0m0.016s
sys     0m35.142s
//a bit over 7 minutes - from a point-in-time snapshot, recoverable,
//had we done that instead with source of the live rw mounted sda (well, containing sda1),
//what we'd have had instead in our target data would have been an inconsistent mess with no guarantees of recoverability
//and out of curiosity, just before removing it,
//I peek at our snapshot to see how much space it has actually used:
# lvdisplay /dev/vg-sflug/sda-snapshot
  Current LE             2816
  COW-table LE           1147
  Allocated to snapshot  0.16%
//as we can see, almost none (0.16%)
//and removing our snapshot:
# lvremove -f /dev/vg-sflug/sda-snapshot
  Logical volume "sda-snapshot" successfully removed
# 
//removing the snapshot is important - if it's left around ("too long") and runs out of room, I/O on the LVs stops
//(that can then be corrected by extending the snapshot LV, or removing it)
//pretty much the same LV snapshot techniques are often used for live database (DB) backups, e.g.:
//put the relevant database files/devices in "backup mode",
//(done within the DB, and in many cases involves more than one which must be done together as a set),
//snapshot them (as set), take database (or those files/devices) out of backup mode, start backing up the snapshots,
//proceed to next set, etc.,
//and also release each snapshot after it (or its set) has completed being backed up.
//so, as exercise, we'll check the recoverability of our backup,
//since it's just one journaled filesystem (and swap, that we don't care about the swap data),
//fairly simple to test (we won't do this test when we do our backup "for real" - we know it works,
//we're just going to demonstrate it here.
//And to note first, we're going to bend our UUID "rule" slightly,
//as here we're going to give host peek into the guest's UUIDs - but that's not an issue in this particular case,
//as we only let it see one instance of this UUID,
//as the other duplicate UUIDs are within the partitions of /dev/vg-sflug/sda,
//which we won't be having the host access (certainly at least not at the same time!)
//from our partition information earlier from the guest (we could also examine it from the host),
//we know how to construct a loop device to access just the first partition within our LV copy:
# losetup -o `expr 512 '*' 63` --sizelimit `expr 512 '*' 20980827` -f --show /dev/vg-sflug/sda-lenny
/dev/loop0
# 
//does it match what we expect?
# blkid /dev/loop0
/dev/loop0: UUID="792a4837-91bc-4b36-bce7-a5a4208845bf" TYPE="ext3"
# 
//yes ... journaled filesystem, so, manner in which we copied it should be quite recoverable,
//but rather than just try to mount it (which may replay journal and do relatively light checks),
//let's do heavier set of checks, and see what, if anything it complains about ... complaints should be
//relatively negligible and few (mostly about state not being set to clean, and perhaps a bit of journal
//to replay - but that should be about it).
# fsck.ext3 -f /dev/loop0
e2fsck 1.41.12 (17-May-2010)
/dev/loop0: recovering journal
Clearing orphaned inode 278547 (uid=0, gid=1, mode=0100700, size=452)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information

/dev/loop0: ***** FILE SYSTEM WAS MODIFIED *****
/dev/loop0: 28306/1310720 files (2.1% non-contiguous), 294444/2621440 blocks
# 
//and as we'd expect, quite "clean" (recoverable),
//some bit(s) like Clearing orphaned inode or similar, may be expected,
//e.g. if snapshot happens, and there are unlinked open files at the time,
//then those files no longer exist in any directory,
//but their inode (and storage, if any) hasn't yet been freed,
//while that's normal for Unix/Linux if a PID still has the files open,
//that's no longer relevant after the fact when we're cleaning up a backup of the filesystem,
//so that bit if housekeeping on the filesystem is still to be done (which would
//normally happen rather quickly on live mounted filesystem after last PID having the unlinked
//file open, closed the file (explicitly, or via other action such as exit)).
//we could do more with that filesystem (e.g. testing, etc.) if we wished at this point,
//but that more than suffices for our testing here, so time to free up that loop device:
# losetup -d /dev/loop0
# 
installation "failed" around step 4.4.5 in:
http://www.debian.org/releases/stable/i386/release-notes/ch-upgrading.en.html
namely reboot failed - kernel couldn't find root filesystem,
upon investigation, found that update procedure hadn't added initrd
lines in /boot/grub/menu.lst for the new kernel, I added those and was
then able to proceed
# hostname; pwd -P; cat bkup; echo; cat bkup.out
vicki
/root/upgrades/5.0_lenny_to_6.0_squeeze/sflug
#!/bin/sh
set -e
umask 077
cd /
at 1740 << \__EOT__
set -e
umask 077
cd /root/upgrades/5.0_lenny_to_6.0_squeeze/sflug/
exec >> bkup.out 2>&1
echo start `date -Iseconds`
[ -b /dev/vg-sflug/sda-lenny ]
lvcreate -l 1147 -n sda-snapshot -s /dev/vg-sflug/sda && {
time dd if=/dev/vg-sflug/sda-snapshot of=/dev/vg-sflug/sda-lenny bs=4194304 || 2>&1 echo dd returned $?
lvremove -f /dev/vg-sflug/sda-snapshot
}
echo end `date -Iseconds`
__EOT__

start 2012-03-14T17:40:00-0700
  Logical volume "sda-snapshot" created
2816+0 records in
2816+0 records out
11811160064 bytes (12 GB) copied, 421.079 s, 28.0 MB/s
0.01user 34.83system 7:01.09elapsed 8%CPU (0avgtext+0avgdata 19248maxresident)k
23068824inputs+23068680outputs (1major+1242minor)pagefaults 0swaps
  Logical volume "sda-snapshot" successfully removed
end 2012-03-14T17:47:04-0700
#