HOWTO – Kubuntu 9.04, RAID-10, LVM2, and XFS…


Time to rebuild the Beast…

The poor thing had been in sparse use since it started shutting down (really, just POOF! –  it may as well be unplugged) randomly.  I didn’t pursue it for months, because I am fundamentally lazy at home.

(I think I mentioned this before.)

But with a trip coming up, and me needing a laptop for it, and my wife saying I could have hers if I got her Windows-based internet movie playing experience working on another computer, and the release of Ubuntu 9.04, well, let’s just say the planets finally aligned.

So I burned several CD’s – Xubuntu Live 9.04 (32-bit), MythBuntu 9.04 (64-bit), Kubuntu 9.04 Live (32-bit), and Kubuntu 9.04 Alternate Install (64-bit).  I rebuilt my targeted media PC first as Kubuntu.  I got a 1 TB SATA drive, put it in my beast computer, and used scp to back up everything in the house to it (execpt the Vista laptop).  It power dropped several times before I finally got it all (I hope).  Then I popped it into my media PC (not Microsoft – I call it that because I have it hooked up to the VGA port on my flat-screen TV).  I then loaded the Live CD of Kubuntu 32-bit and will use it as a backup in case my beastie dies while I am away.  This way, my wife is not stuck unable to watch her Japanese TV program downloads.

You have no idea how important that is to maintaining a happy family.  Seriously.

Anyway, enough boring crap.

I first installed the seven SATA drives I had (four pulled from the media PC, one was already installed, and two were sitting in a drawer), each identical 80 GB Hitachis, and left them powered up overnight to find any serious drive errors.  I got seek errors on the one I suspected of being bad, and tossed it.  Trust me – it was bad.

(That may explain why it was sitting in a drawer…)

The other six have stayed quite civilized.  Maybe they got the hint.

After trial and error, I used a combination of the Live CD to google and hand-build the file system, and the alternate 64-bit CD to install.

Why hand-build?  I guess that’s just how I roll…  And it gave me total control over how I built it.

Playing around with the drives, I found I could reliably pop the power just by running “hdparm -Tt /dev/sda”, so off I went to get a new power supply.  I found a 650W PS that more than makes up for my failing 450W PS, and let me clean up my cable mess as well.  Out with the old, in with the new, and everything is smooth as silk.

Back to googling, I found a collection of sites that allowed me to piece together what I think, and hope, is a very solid compromise between performance and reliability.  Space is not too much of an issue, since only one VM will be running on this system, and we are not huge downloaders.  As long as it has more space than the laptop (160 GB), it is fine.

System specs:

  • Athlon FX-53 (the old obsolete server-board-based one with 959 pins or something).
  • 2 GB of registered memory, I forget how fast.
  • An old NVidia AGP 7600 GT card (I think).
  • Four SATA ports onboard (two controllers, no hardware RAID enabled).
  • One four port add-in PCI SATA controller (RAID disabled).
  • No special BIOS tweaks.
  • Six SATA drives, 80 GB each, /dev/sda, /dev/sdb, /dev/sdc, /dev/sdd, /dev/sde, /dev/sdf.
  • Fans.  Lots of fans.
  • 64-bit Kubuntu 9.04, alternate install CD.  Supports VMware-Server 2, and can run 64-bit and 32-bit virtual machine guests.

From the Live CD:

Opening Konsole and Konqueror:

sudo -i

Partitioning the drives:

cfdisk /dev/sda

  • sda1 primary 82 MB type FD (Linux RAID), bootable – this will be the RAID-1 /boot partition of six drives (ext3)
  • sda2 primary 404 MB type FD – this will be the swap partion on RAID-10 and LVM
  • sda3 primary 82 GB type FD – this will be the OS partition on RAID-10 and XFS

sfdisk -d /dev/sda | sfdisk /dev/sdb

sfdisk -d /dev/sda | sfdisk /dev/sdc

sfdisk -d /dev/sda | sfdisk /dev/sdd

sfdisk -d /dev/sda | sfdisk /dev/sde

sfdisk -d /dev/sda | sfdisk /dev/sdf

REBOOT (power cycle), run Live CD again, same apps opened:

sudo -i
apt-get install mdadm lvm2 (the live CD does not get RAID and LVM on its own – so install them)

RAID-1 and RAID-10 (all active, no spares):
Link = http://www.howtoforge.org/install-ubuntu-with-software-raid-10

  • boot partition: mdadm -v -C /dev/md0 -c 256 -n 6 -l 1 /dev/sd[abcdef]1 – RAID1 so LILO can boot it, all drives for max redundancy.
  • swap partition: mdadm -v -C /dev/md1 -c 256 -n 6 -l 10 -p f6 /dev/sd[abcdef]2
  • os partition: mdadm -v -C /dev/md2 -c 256 -n 6 -l 10 -p f2 /dev/sd[abcdef]3

cat /proc/mdstats to see RAID sets:

md2 : active raid10 sda3[0] sdf3[5] sde3[4] sdd3[3] sdc3[2] sdb3[1]
239817216 blocks 256K chunks 2 far-copies [6/6] [UUUUUU]
[=================>...] resync = 86.1% (206697408/239817216) finish=12.6min speed=43546K/sec

md1 : active raid10 sda2[0] sdf2[5] sde2[4] sdd2[3] sdc2[2] sdb2[1]
393472 blocks 6 near-copies [6/6] [UUUUUU]

md0 : active raid10 sda1[0] sdf1[5] sde1[4] sdd1[3] sdc1[2] sdb1[1]
79872 blocks 256K chunks 6 far-copies [6/6] [UUUUUU]

unused devices:

Next, set up LVM:
Link = http://www.linuxdynasty.org/lvm2-how-to.html

Physical Volumes:

  • pvcreate /dev/md0
  • pvcreate /dev/md1
  • pvcreate /dev/md2

Volume Groups:
create, with useful names:

  • vgcreate boot-vg /dev/md0
  • vgcreate swap-vg /dev/md1
  • vgcreate os-vg /dev/md2

activate:

  • vgchange -a y boot-vg
  • vgchange -a y swap-vg
  • vgchange -a y os-vg

pvdisplay and pvscan to see physical volumes:

PV /dev/md2 VG os-vg lvm2 [228.71 GB / 4.00 MB free]
PV /dev/md1 VG swap-vg lvm2 [384.00 MB / 0 free]
PV /dev/md0 VG boot-vg lvm2 [76.00 MB / 0 free]
Total: 3 [229.16 GB] / in use: 3 [229.16 GB] / in no VG: 0 [0 ]

vgdisplay and vgscan to see volume groups:

Reading all physical volumes. This may take a while…
Found volume group “os-vg” using metadata type lvm2
Found volume group “swap-vg” using metadata type lvm2
Found volume group “boot-vg” using metadata type lvm2

Logical Volumes, create with useful names:

  • lvcreate -L 76M -n boot-lv boot-vg
  • lvcreate -L 384M -n swap-lv swap-vg
  • lvcreate -L 10G -n root-lv os-vg
  • lvcreate -L 2G -n var-lv os-vg
  • lvcreate -L 3G -n temp-lv os-vg
  • lvcreate -L 213.7G -n home-lv os-vg

lvdisplay and lvscan to see logical volumes:

ACTIVE ‘/dev/os-vg/root-lv’ [10.00 GB] inherit
ACTIVE ‘/dev/os-vg/var-lv’ [2.00 GB] inherit
ACTIVE ‘/dev/os-vg/temp-lv’ [3.00 GB] inherit
ACTIVE ‘/dev/os-vg/home-lv’ [213.70 GB] inherit
ACTIVE ‘/dev/swap-vg/swap-lv’ [384.00 MB] inherit
ACTIVE ‘/dev/boot-vg/boot-lv’ [76.00 MB] inherit

So far, partitioning, RAID-10, and LVM are done. Format using swap, ext3 (boot) and XFS:

  • mkfs.ext3 /dev/boot-vg/boot-lv
  • mkswap /dev/swap-vg/swap-lv

XFS Links:
http://www.csamuel.org/2008/03/23/btrfs-013-and-xfs-benchmarks
http://oss.oracle.com/projects/btrfs/dist/documentation/benchmark.html
http://everything2.com/index.pl?node_id=1479435

http://www.issociate.de/board/post/472270/New_XFS_benchmarks_using_David_Chinner%27s_recommendations_for_XFS-basedoptimizations..html

  • mkfs.xfs -f -d agcount=1 -i attr=2 -l lazy-count=1,size=128m,version=2 /dev/os-vg/root-lv
  • mkfs.xfs -f -d agcount=1 -i attr=2 -l lazy-count=1,size=128m,version=2 /dev/os-vg/var-lv
  • mkfs.xfs -f -d agcount=1 -i attr=2 -l lazy-count=1,size=128m,version=2 /dev/os-vg/temp-lv
  • mkfs.xfs -f -d agcount=1 -i attr=2 -l lazy-count=1,size=128m,version=2 /dev/os-vg/root-lv
  • mkfs.xfs -f -d agcount=1 -i attr=2 -l lazy-count=1,size=128m,version=2 /dev/os-vg/home-lv

df -h to see:

/dev/mapper/os–vg-root–lv
9.9G 4.1M 9.9G 1% /target
/dev/mapper/os–vg-var–lv
1.9G 4.1M 1.9G 1% /target/var
/dev/mapper/os–vg-temp–lv
2.9G 4.1M 2.9G 1% /target/temp
/dev/mapper/os–vg-home–lv
214G 4.1M 214G 1% /target/home
/dev/mapper/boot–vg-boot–lv
74M 5.6M 65M 8% /target/boot

Install using the 64-bit Alternate Install CD for Kubuntu 9.04.  Use the ext3 partition for /boot, the XFS partitions for /, /var, /tmp, and /home, and use the swap partition.  Do not format anything  – it will then only demand to format the swap partition.  I hand-formated to get additional control over how XFS was formatted.

I always separate /home to survive any OS rebuilds I might have to do, or a distro change.  I also separate out /var (so if it fills up, it does not fill up the root space), and /tmp (VMware stores lots of stuff there when snapshotting virtual machines, so make the room).

It installed LILO with the large-memory option on /dev/md0, and reran LILO successfully.  There was some misinformation out there that LILO would boot from a RAID-10 volume.  Yeah, only if it looks exactly like a RAID-1 mirror.  Whoops.

Mount options for XFS I put into /etc/fstab after successfully booting:

-o noatime,nodiratime,logbsize=256k,logbufs=8

Conclusion:  Well, it hasn’t thrown up yet.  I guess that is good.  It seems plenty fast, but I have not really exercised it.  I do not think I will mess with the kernel for a while – I need it to be very stable while I am away, which is exactly when it is most likely to break.

Next, I will put VMware-Server 2 on it and install a 32-bit XP VM for my wife to use with her JNet TV streaming addiction.  I am assuming it won’t work with Firefox and Linux, but I will try that also, to be sure.  She switched to that after Pandora TV changed for the worse.

Should it all work out, I will putty her Windows settings over (bookmarks really), finish up her XP VM, and finally get around to fixing her Vista laptop with a prescription-case extra-strength dose of Xubuntu.

I can hardly wait.

6 Responses

  1. Good news, thanks for information I love this site :)

  2. Do you have a detailed description of your installation of lilo on your Kubuntu RAID, please?
    I’ve scripted your RAID-assembly procedure, so that now takes just 2 minutes, but I cannot for the life of me get lilo to install. I restart on the Alternate-build CD, using the “Repair” option on the menu, filling-in the demanded details. The best result so far (via apt-get install lilo) complains that it can’t install because three variables (LC_ALL, LC_CTYPE and LC_MESSAGES) aren’t defined ~ but they aren’t defined in a non-RAID build either!
    This is all on Ubuntu 10.04.1, but I wouldn’t have thought that this would make much difference?
    And thanks for releasing this document ~ it’s clarified things beautifully, making it so much more frustrating that the last step defies solution!
    Regards
    Peter Overfield

    • Hi Peter. Sorry, the LILO install was plain-vanilla-boring. So uneventful that there was nothing to really document. I tried to find something helpful out there for you, but nothing really came up. Sorry.
      Anyway, I am glad you liked the article, and I am interested in hearing how you solve your problem. Good luck!

  3. Boring, eh? Uneventful! I could do with a bit of that!!
    Anyway, on progress…
    The LC_ variables are all associated with the LANG variable, and setting them all to that value appears to be appropriate. It doesn’t stop the Warnings, however, but they are just warnings and the process does now appear to conclude. It just doesn’t work, yet. Booting of the “results” just halts at a blank screen. I think my next need is to force a Linux boot-sector into the MBRs, since they’re virgin drives that have never been contaminated with Micro$oft. Bootactv is widely promoted as a good option, but finding a copy has proved fruitless. It’s reported to be included in a package called “pfdisk”, but the Ubuntu repositories know nothing about that, either. But as an intermediate step, might I prevail on you to release the contents of your lilo.conf file, at least as far as pertains to the Linux RAID? That would give me some pointers, and reduce the unknown variables to be guessed or deduced.
    Incidentally, the scripts to implement your process now appear to be reliable. If you’d like a copy, perhaps even to make it available through your web presence, I’d be delighted to send it on.
    Kind Regards
    Peter

  4. Well, it now works. Just don’t ask me HOW. I can tell you WHAT I’ve done, but you’ll have to decide WHY.
    Here goes.
    I’d already downloaded reams of stuff on raid, lilo, mkinitrd, lilo.conf, etc. and confused myself witless reading it all. So much contradictory advice! It was helped (a bit) when I stumbled across mkinitramfs, which apparently has quietly replaced mkinitrd.
    Initial attempts to run a downloaded lilo objected that it didn’t understand “bterm”. Eventually resolved by discovering and downloading bogl-bterm. Other messages indicating massive failure were resolved by manually mounting all the various LVs ~ the “repair broken system” option from the Alternate CD only reliably mounts the LV one’s selected to run the command-line system in.
    Various experiments with boot=/dev/mapper/bootVG-bootLV (the fstab entry), /dev/bootVG/bootLV (the path that actually displayed the contents of /boot), /dev/md0 (suggested in README.raid1) gave errors like “map file must be in /boot” (it was ~ I could see it there!) or “Not a RAID install: ‘raid-extra-boot=’ not allowed” (but I’d just BUILT the raid arrays!!) were eventually resolved with boot=/dev/sda1 (also offered in README.raid1). Similarly root=/dev/mapper/systemVG-rootLV didn’t work, but /dev/systemVG/rootLV did.
    Then, after using the serendipitous mkinitramfs to make an initrd.img, lilo insisted that “No images have been defined” ~ even though it was 9.43MB long. After some further progress it was delighted to advise me that “Image name, label or alias is too long: ‘vmlinuz-2.6.32-24-generic'”.
    You get the idea. To cut a three-week-long story short, lilo.conf thus:
    ####################################
    boot=/dev/sda1
    compact
    large-memory

    image=/boot/vmlinuz
    root=/dev/systemVG/rootLV
    read-only
    initrd=/boot/initrd.img-2.6.32-24-generic
    ####################################
    works, when coupled with ln -s /boot/vmlinuz-2.6.32-24-generic /boot/vmlinuz.
    It would be interesting to see how that compares with your own lilo.conf. Please.
    So now I can offer a set of four scripts to do the entire job:
    a) Set of configuration data, so that operator can define exactly what RAID assembly he wants
    b) Partitioning executable
    c) RAID-building executable
    d) LILO-installing executable
    Oh, yes, and e) Instruction on how to use them.
    If you’d like to have them, just let me know how to get them to you.
    Kind Regards
    Peter Overfield

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: