User Tools

Site Tools


system:vicki_debian_lenny_to_squeeze

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
system:vicki_debian_lenny_to_squeeze [2012-03-13T06:13:54+0000]
michael_paoli added a bunch of stuff detailing example of (and comments upon) our pre-upgrade backup, including example
system:vicki_debian_lenny_to_squeeze [2013-02-17T20:00:48+0000] (current)
michael_paoli typo/wordo fix
Line 543: Line 543:
  
 </​file>​ </​file>​
- 
 ===== vicki host install/"​upgrade"​ procedure outline + select details: ===== ===== vicki host install/"​upgrade"​ procedure outline + select details: =====
 <​file>​ <​file>​
Line 645: Line 644:
       o /boot       o /boot
       o vicki-boot       o vicki-boot
-    o Finish partitioning ​nad write changes to disk+    o Finish partitioning ​and write changes to disk
     (continue without swap)     (continue without swap)
 o Install base system o Install base system
Line 846: Line 845:
 ===== sflug VM upgrade to Debian GNU/Linux 6.0.4 ===== ===== sflug VM upgrade to Debian GNU/Linux 6.0.4 =====
   * sflug VM log bits made available here: [[http://​www.sf-lug.com/​log.txt]]   * sflug VM log bits made available here: [[http://​www.sf-lug.com/​log.txt]]
-  * upgrade to be done [[http://​linuxmafia.com/​pipermail/​sf-lug/​2012q1/​009242.html|soon]] +  * upgrade to be done [[http://​linuxmafia.com/​pipermail/​sf-lug/​2012q1/​009242.html|soon]] ... [[http://​linuxmafia.com/​pipermail/​sf-lug/​2012q1/​009246.html|"​done"​]] 
-  * will be upgrading, mostly per:+  * will be upgrading/was upgraded, mostly per:
     * **backup(s)** - since [www.]sf-lug.com is (generally) "​down"​ when sflug is down, we'll mostly try to minimize the host's downtime. ​ So, for backup, backing up "​everything"​ is good,  but as this is a VM, and its storage on the hosting system is under LVM, we have some nice options for fast live backups - backups from which, state is at least "​recoverable",​ namely:     * **backup(s)** - since [www.]sf-lug.com is (generally) "​down"​ when sflug is down, we'll mostly try to minimize the host's downtime. ​ So, for backup, backing up "​everything"​ is good,  but as this is a VM, and its storage on the hosting system is under LVM, we have some nice options for fast live backups - backups from which, state is at least "​recoverable",​ namely:
       * from host (vicki), sflug'​s storage is an LVM LV,       * from host (vicki), sflug'​s storage is an LVM LV,
Line 858: Line 857:
         * from the host there are separate and unique LVs, and for the most part host wouldn'​t go poking within to look at, e.g. partitions within or other constructs within, thus host normally wouldn'​t see any of those further lower level UUIDs within,         * from the host there are separate and unique LVs, and for the most part host wouldn'​t go poking within to look at, e.g. partitions within or other constructs within, thus host normally wouldn'​t see any of those further lower level UUIDs within,
         * we won't present both "​original"​ and "​copy"​ to guest (sflug) or any other single VM at the same time - if we did, that's where problems would occur, as host would see two hard drives, with identical UUIDs, and thus problems would be likely to occur,         * we won't present both "​original"​ and "​copy"​ to guest (sflug) or any other single VM at the same time - if we did, that's where problems would occur, as host would see two hard drives, with identical UUIDs, and thus problems would be likely to occur,
-        * also, these backups are "​temporary"​ and for convenience - "just in case" upgrade goes horribly wrong or something like that - gives us quick convenient way to get back to "​precisely"​ where we were before we started the upgrade (specifically to the point of the moment the LV snapshot was taken), we probably won't need to use this backup at all, and will dispose of it (and it'​s ​UUIDs - wipe the data), fairly shortly after we're confident all'​s ​gone quite well with the upgrade and we've no interest in potentially reverting to the pre-upgrade snapshot backup, +        * also, these backups are "​temporary"​ and for convenience - "just in case" upgrade goes horribly wrong or something like that - gives us quick convenient way to get back to "​precisely"​ where we were before we started the upgrade (specifically to the point of the moment the LV snapshot was taken), we probably won't need to use this backup at all, and will dispose of it (and its UUIDs - wipe the data), fairly shortly after we're confident all has gone quite well with the upgrade and we've no interest in potentially reverting to the pre-upgrade snapshot backup, 
-        * note also this is separate/​independent of our more (semi-?​)regular backups, which tend to be filesystem based and avoid most all the UUID issues (and also tend to generally be much more space efficient) - in this particular pre-upgrade backup scenario, we're going for recoverability and speed over space efficiency of stored (or transmitted) backup. +        * note also this is separate/​independent of our more (semi-?​)regular backups, which tend to be filesystem based and avoid most all the UUID issues (and also tend to generally be much more space efficient) - in this particular pre-upgrade backup scenario, we're going for recoverability and speed and uptime ​over space efficiency of stored (or transmitted) backup. 
-      * doing the backup - here's part of what doing the "​proposed"​ LV snapshot based backup looks like, in this case I do a "​real"​ pre-run (plus some additional checks) - when we're much close to our upgrade time, will do another nice fresh backup (except some of that pre-work will already be done, so we can skip repeating some bits - we can also skip some of the test/​verification bits done here for demonstration purposes)+      * doing the backup - here's part of what doing the "​proposed"​ LV snapshot based backup looks like, in this case I do a "​real"​ pre-run (plus some additional checks) - when we're much closer ​to our upgrade time, will do another nice fresh backup (except some of that pre-work will already be done, so we can skip repeating some bits - we can also skip some of the test/​verification bits done here for demonstration purposes)
 <​file>​ <​file>​
 //I show my comments on lines starting with // //I show my comments on lines starting with //
Line 1006: Line 1005:
 //yes ... journaled filesystem, so, manner in which we copied it should be quite recoverable,​ //yes ... journaled filesystem, so, manner in which we copied it should be quite recoverable,​
 //but rather than just try to mount it (which may replay journal and do relatively light checks), //but rather than just try to mount it (which may replay journal and do relatively light checks),
-//lets do heavier set of checks, and see what, if anything it complains about ... complaints should be+//let'​s ​do heavier set of checks, and see what, if anything it complains about ... complaints should be
 //​relatively negligible and few (mostly about state not being set to clean, and perhaps a bit of journal //​relatively negligible and few (mostly about state not being set to clean, and perhaps a bit of journal
 //to replay - but that should be about it). //to replay - but that should be about it).
Line 1031: Line 1030:
 //so that bit if housekeeping on the filesystem is still to be done (which would //so that bit if housekeeping on the filesystem is still to be done (which would
 //normally happen rather quickly on live mounted filesystem after last PID having the unlinked //normally happen rather quickly on live mounted filesystem after last PID having the unlinked
-//file open, closed the file (explicitly,​ or via other action such as exit).+//file open, closed the file (explicitly,​ or via other action such as exit)).
 //we could do more with that filesystem (e.g. testing, etc.) if we wished at this point, //we could do more with that filesystem (e.g. testing, etc.) if we wished at this point,
 //but that more than suffices for our testing here, so time to free up that loop device: //but that more than suffices for our testing here, so time to free up that loop device:
Line 1050: Line 1049:
     * **relevant Debian documentation**:​     * **relevant Debian documentation**:​
       * [[http://​www.debian.org/​releases/​stable/​i386/​release-notes/​|Release Notes for Debian GNU/Linux 6.0 (squeeze), 32-bit PC]]       * [[http://​www.debian.org/​releases/​stable/​i386/​release-notes/​|Release Notes for Debian GNU/Linux 6.0 (squeeze), 32-bit PC]]
-        * **especially:​ [[http://​www.debian.org/​releases/​stable/​i386/​release-notes/​ch-upgrading.en.html|Upgrades from Debian 5.0 (lenny)]]**will do pre-upgrade steps in advance as feasible +        * **especially:​ [[http://​www.debian.org/​releases/​stable/​i386/​release-notes/​ch-upgrading.en.html|Upgrades from Debian 5.0 (lenny)]]** 
-      * and also if/​as/​where/​relevant:​ [[http://​www.debian.org/​releases/​stable/​i386/​|Debian GNU/Linux Installation Guide]] (installation instructions for the Debian GNU/Linux 6.0 system (codename “squeeze”),​ for the 32-bit PC (“i386”) architecture)+          * will do pre-upgrade steps in advance as feasible, have done the following:​ 
 +            * 4.1.1\\ output of:\\ # dpkg_--get-selections '​*'​\\ saved to:\\ /​root/​upgrades/​5.0_lenny_to_6.0_squeeze/​dpkg_--get-selections\\ from host (vicki), saved sflug VM configuration for reference, "just in case":​\\ # virsh dumpxml sflug > /​root/​upgrades/​5.0_lenny_to_6.0_squeeze/​sflug/​sflug.xml\\ will do fresh snapshot backup of balug VM (virtual) disk before upgrade 
 +            * 4.1.6 checked package splashy not installed 
 +            * 4.2 - should be good on no 3rd party packages, etc., system has been updated to the latest point release of lenny (5.0.10). 
 +            * 4.2.1 checked good: No packages are scheduled to be installed, removed, or upgraded. 
 +            * 4.2.2 good (/​etc/​apt/​preferences not present) 
 +            * 4.2.3:\\ looks good: # dpkg --audit\\ dpkg -l looks quite good, used: # dpkg -l | grep -v '^ii '\\ dpkg --get-selections '​*'​ also quite clean, examined: # dpkg --get-selections '​*'​ | awk '​{print $2;}' | sort | uniq -c\\ also clean: # aptitude search "​~ahold"​ 
 +            * 4.2.4 proposed-updates not in /​etc/​apt/​sources.list 
 +            * 4.2.5 - presumably not an issue (doesn'​t appear to be an issue) 
 +          * **done** (main upgrade proper sequence, and immediately preceding steps): 
 +            * did LV snapshot backup, from host (vicki) (target created earlier; also, this was set up to execute via at(1) at 5:40 P.M. on the day of the upgrade - 20 minutes prior to start of our upgrade maintenance window; and in earlier test runs, completed in under 10 minutes; checked that it completed okay, script and output on viki host under /​root/​upgrades/​5.0_lenny_to_6.0_squeeze/​sflug/​):​ 
 +              * # lvcreate -l 1147 -n sda-snapshot -s /​dev/​vg-sflug/​sda 
 +              * # time dd if=/​dev/​vg-sflug/​sda-snapshot of=/​dev/​vg-sflug/​sda-lenny bs=4194304 
 +              * # lvremove -f /​dev/​vg-sflug/​sda-snapshot 
 +              * and here's what that script that was executed looks like, and its output: 
 +<​file>​ 
 +# hostname; pwd -P; cat bkup; echo; cat bkup.out 
 +vicki 
 +/​root/​upgrades/​5.0_lenny_to_6.0_squeeze/​sflug 
 +#!/bin/sh 
 +set -e 
 +umask 077 
 +cd / 
 +at 1740 << \__EOT__ 
 +set -e 
 +umask 077 
 +cd /​root/​upgrades/​5.0_lenny_to_6.0_squeeze/​sflug/​ 
 +exec >> bkup.out 2>&​1 
 +echo start `date -Iseconds` 
 +[ -b /​dev/​vg-sflug/​sda-lenny ] 
 +lvcreate -l 1147 -n sda-snapshot -s /​dev/​vg-sflug/​sda && { 
 +time dd if=/​dev/​vg-sflug/​sda-snapshot of=/​dev/​vg-sflug/​sda-lenny bs=4194304 || 2>&1 echo dd returned $? 
 +lvremove -f /​dev/​vg-sflug/​sda-snapshot 
 +
 +echo end `date -Iseconds` 
 +__EOT__ 
 + 
 +start 2012-03-14T17:​40:​00-0700 
 +  Logical volume "​sda-snapshot"​ created 
 +2816+0 records in 
 +2816+0 records out 
 +11811160064 bytes (12 GB) copied, 421.079 s, 28.0 MB/s 
 +0.01user 34.83system 7:​01.09elapsed 8%CPU (0avgtext+0avgdata 19248maxresident)k 
 +23068824inputs+23068680outputs (1major+1242minor)pagefaults 0swaps 
 +  Logical volume "​sda-snapshot"​ successfully removed 
 +end 2012-03-14T17:​47:​04-0700 
 +#  
 +</​file>​ 
 +            * present Debian GNU/Linux 6.0.4 "​Squeeze"​ - Official i386 CD Binary-1 20120128-12:​53 (or latest 6.0.x) from host to sflug guest,\\ we do this as a SCSI "​disk"​ (though SCSI CD-ROM would be even more preferred), as we can't hot remove virtual IDE devices from guest (though we can change virtual media ... but sometimes we want to present multiple ISO images concurrently,​ and it's preferable to be able to remove the devices when they'​re no longer needed or wanted), so we present it from host vicki thusly:\\ # virsh attach-disk sflug /​var/​local/​pub/​mirrored/​cdimage.debian.org/​debian-cd/​6.0.4/​i386/​iso-cd/​debian-6.0.4-i386-CD-1.iso sdc --mode readonly\\ Note that that sometimes causes a slight booboo in the guest'​s configuration,​ which we'd fix via:\\ # virsh edit sflug\\ namely we'd want to change the driver name from file to qemu in the section for our added disk - otherwise the guest will fail to restart once (virtually) powered off.  In our particular case this time around, virsh appeared to not introduce that issue (it used driver name phy) - probably "good enough",​ so didn't bother to "​correct"/​alter that at time of upgrade; we do however, still need to test full (virutal) power down and subsequent reboot of guest 
 +            * on guest, scan SCSI devices, create mount point (if not present), update /etc/fstab, and mount it: 
 +              * # (for tmp in /​sys/​class/​scsi_host/​host*/​scan;​ do echo '- - -' > "​$tmp";​ done) 
 +              * # (umask 022 && mkdir /​media/​cdrom9) 
 +              * Even though we told the host to present the device to guest as sdc, it showed up on guest as sdb, so we went with that: 
 +              * add to /​etc/​fstab:​\\ /dev/sdb /​media/​cdrom9 udf,iso9660 ro,​nosuid,​nodev 0 0 
 +              * # mount /​media/​cdrom9 
 +            * 4.3 - continue per documentation\\ add new cdrom ISO to /​etc/​apt/​sources.list:​\\ # apt-cdrom -m -d=/​media/​cdrom add\\ and make other /​etc/​apt/​sources.list updates 
 +            * looking now at the above, and checking shell history, had actually intended to use /​media/​cdrom9 for the above apt-cdrom command, but didn't catch that mis-entry at the time ... so, ... apt and friends "of course"​ then wanted to use /​media/​cdrom instead of /​media/​cdrom9 ... at the time, just made symbolic link to cover that, and then continued from there: 
 +            * # ln -s /​media/​cdrom9 /​media/​cdrom 
 +            * ... 
 +            * 4.4.5 **before** first reboot, inspected /​boot/​grub/​menu.lst,​ and corrected/​edited if/as relevant, probably also increase timeout if a "too short" default timeout got placed in there - /​boot/​grub/​menu.lst actually looked fine - unlike the glitch we encountered when upgrading the balug VM, where the new kernel entries were missing their initrd entries. 
 +            * inspected legacy grub, and grub2 configurations - grub looked fine, tweaked grub2 - mostly via /​etc/​default/​grub and mostly modeling after quite similar from the balug VM guest'​s configuration with grub2, tested and legacy grub boot looked fine, chainload of grub2 worked and looked fine, booted via grub2 chainload, used the indicated procedure to then convert to update to grub2 (including MBR) and remove (but not purge) legacy grub (all handled pretty much automagically for us by running the one indicated program that did that for us - we just had to "​tell"​ it (via presented options) where grub2 was to "​install"​ (write its MBR) - the rest what highly automagic; tested reboot with direct grub2 without any "​user"​(/​admin) input being used on console (graphics or terminal) to ensure all came up fine multi-user without any input needed (to ensure unattended (re)boots should generally quite work well and as expected). 
 +            * ... 
 +            * we essentially completed through step 4.6.3, "​skipped"​ (deferred) 4.7 (the part before 4.7.1), and completed step 4.7.1, and then deferred subsequent steps (4.8 and beyond) for later subsequent "​clean-up"​ and the like, having essentially made it through all the particularly critical and important portions of the main sequence of the upgrade proper. 
 +            * we also checked both web server and DNS server both on the host and up and running and operational as expected. 
 +          * **and done(!)** - these bits were completed later, but not later than 2012-03-17:​ 
 +            * went through and completed our various clean-up tasks: 
 +              * reviewed configurations - in the interests of time (and uptime), some of the configurations where we had local modifications,​ we just used our "​old"​ configurations,​ and didn't examine or merge/​integrate our earlier changes with the newer default configurations;​ completed going through those carefully and adjusting and cleaning up as appropriate 
 +              * init / rc stuff - we didn't earlier get the advantages of the newer faster start-up system - some legacy/​dependency bit prevented that from being activated; reviewed that and got that all squared away 
 +              * 4.7 (the part before 4.7.1) 
 +              * steps starting at and beyond step 4.8 
 +              * saved script output from most of the upgrade process - reviewed that for any potentially security sensitive bits, and as there weren'​t any significant security issues with public disclosure, made that data available (at least "for a while"​) at: [[http://​www.sf-lug.com/​upgrade-squeeze.tar.bz2]] 
 +              * high-level (human written (with some machine assist) and intended for human consumption) log file for the system is available at:\\ [[http://​www.sf-lug.com/​log.txt]] - for this system upgrade bit, look/search for the line that starts with:\\ 2012-03-14\\ (at the time of this writing that's near the end of that file) ... and got that file quite caught up. 
 +            * did full virtual power reboot test (did reboot tests 2012-03-14, but not with full virtual power down and back up again with guest), ensured that guest comes up fine with no need for any console input being provided 
 +      * and (documentation bits - didn't really need to refer to or use this particular document with/for our upgrade) ​also if/​as/​where/​relevant:​ [[http://​www.debian.org/​releases/​stable/​i386/​|Debian GNU/Linux Installation Guide]] (installation instructions for the Debian GNU/Linux 6.0 system (codename “squeeze”),​ for the 32-bit PC (“i386”) architecture)
system/vicki_debian_lenny_to_squeeze.1331619234.txt.bz2 · Last modified: 2012-03-13T06:13:54+0000 by michael_paoli