Interesting Smartcard Behavior…

I ma no expert on smartcards (we use Common Access Cards, or CAC, at work).  Recently, I wqaas having issues with ActiveClient reading my card, which told me I had mistyped my pin twice.  Once more, and it would lock out my card.  So I went and asked a coworker how long the timeout is.  He told me there is no timeout – the card count is reset with a successful login.

Worried, I went to my Linux box and logged into webmail with my card, successfully.  Afterwards, ActiveClient on the other machine saw the card fine.

I had a chance yesterday to retest this behavior when another coworker was having similar issues with his card.  He had tried at two different machines to log in with his smartcard, and each time, it was unable to read his card (I did not see the exact error message, but it was not an incorrect PIN entry).  So I offered to let him try on my Linux box (I am the only one using Linux as a workstation), and he was quite surprised to be able to log into webmail from there.  Afterwards, his card worked fine on ActiveClient Windows machines, which was a relief to him, since he had assumed a trip to the ID card office (and a long wait) were in store for him.

I surmise that something on his card got a little scrambled and the Windows ActiveClient could not read what it needed.  The PCSC client, however, not only worked without flaw, it also seemed to clear whatever was bugging ActiveClient.

So, if this happens again, I hopefully will be able to spare someone else a trip to the ID card office.

Stateless VMware ESXi 3.5 on an HP c7000 Blade Server…

NOTE:  This is only an overview.  Due to the detailed nature of this project, I will break it up over several more-focused articles over time for easier reference.

Well, despite my more negative impression of this year’s VMworld conference, it still really paid off.  There I learned about stateless ESX deployment.  Using this information, I was able to build in my lab, after a couple months of trial and error, a highly robust VMware environment, fully managed and licensed, using the midwife scripts I modified for this effort.  And configuration is hands-free.

Here are the system components:

  • SERVER – HP c7000 Blade Enclosure with sixteen Bl465c blades, two 4 GB FC modules, and four VC Enet modules
  • Each blade has two dual-core AMD CPUs, 16 GB RAM, two 72 GB SAS drives (hardware RAID-1), two embedded gig NICs, and a mezzanine card with two more gig NICs/iSCSI initiators and two FC HBAs
  • NETWORK – Cisco 6509 with two SUP 720 cards, two 48 port LC Gig-E fiber cards, and four 48 port gig copper cards
  • MANAGMENT – Dell 1850 with two 146 GB SAS drives (hardware RAID-1) for management and boot services
  • STORAGE – Scavenged proof-of-concept totally ghetto Dell Optiplex desktop with four internal 1.5 TB SATA drives (software RAID-10 formatted with tuned XFS) providing 3 TB of NFS shared storage
  • Scavenged HP IP-KVM box for OOB-management of the two Dells

Here are the steps I took:

  1. First I had to update all the firmware on the blade server.  This includes the two OA cards for the Onboard Administrator, the Virtual Connect software, the iLO2 software for each blade, the BIOS on each blade, and the Power Management Controller firmware.  There is a particular order this is done in, and it is not easy, but it really needs to be done.  The fixes that come with these updates are often vital to success.  Overall, I spent a week researching and updating.  I set all the blades to boot via PXE.
  2. Next, I built the storage server.  I really had no choice – nothing was available but a Dell Optiplex desktop.  It had four internal SATA ports available, and room for four 1GB RAM modules.  It also had a single dual-core Intel CPU and PCI slots for more NICs, and a PCI-Express mini-slot as well.  I had to order parts, and it took a little while, but once done, it had a total of four gig NICs (one embedded, two PCI, one PCI-Express), four 1.5 TB SATA drives, and 4 GB RAM.  I loaded it with 64-bit Ubuntu-9.04, hand-carved the partitions and RAID-10 setup, formatted the 3 TB volume with XFS, tuned as best I knew how, and then put it on the 2.6.31 kernel (I later updated it to 2.6.31.5).  There were no BIOS or other firmware updates needed.
  3. I then built the management server on the Dell 1850.  It only has one power supply (I cannot find a second one), but it does have 8 GB RAM and two dual-core CPUs.  I loaded 64-bit Ubuntu-9.04 on it afte installing two 146 GB SAS drives in a RAID-1 mirror (hardware-based).  I also updated the BIOS and other firmware on it.
  4. Having these components in place, I studied the blade server to see what I could get away with, and ultimately decided to use each NIC on a blade server to support a set of traffic types, and balanced the likelyhood of traffic demands across them.  For example, Vmotion traffic, while it may be intense, should be relatively infrequent, so it shares a V-Net with another type of traffic that is low-bandwidth (the alternate management  network).  Altogether, I ended up with a primary management network on up V-Net, Vmotion and the alternate on another V-Net, storage traffic (NFS and iSCSI) on a third V-Net, and VM traffic on its own V-Net.  Each V-Net maps to the its own NIC on a blade, the same NIC on each blade.

The physical network design:

For the V-Nets, the management network went on NIC 1 as an untagged VLAN.  It has to be untagged, because when it boots up, it needs to get a DHCP address and talk to the boot server for its image.  Since it comes up untagged, it will not be able to talk out to the DHCP/PXE server if the V-Net is set to pass through tags.  The other V-Nets support tagged VLANs to further separate traffic.  Each V-Net has four links to the Cisco 6509, except for the storage V-Net, which has eight.  Two links form an LACP bundle from the active side (VC-Enet module in Bay 1), and two make up an LACP bundle (or etherchannel) from the module in Bay 2, which is the offline side.  This is repeated for the other networks across the other modules in Bays 5 and 6.  Bays 3 and 4 house the Fiber Channel modules, which I am not using.  Everything is on its own individual private 10.x.x.x network as well, except for the VM traffic net, which will contain the virtual machine traffic.

The storage design:

Like I said, a really ghetto NFS server.  It does not have enough drives, so even though it would be overkill for a home PC, it will not cut it in this situation.  I expect it to run out of steam after only a few VMs are added, but it does tie everything together and provides the shared storage component needed for HA, Vmotion, and DRS.  I am working on an afforable and acceptable solution, rack-mounted, with more gig NICs and up to 24 hot-swap drives – more spindles should offer more thoughput.  I bonded the NICs together into a single LACP link, untagged back the the Cisco, on the NFS storage VLAN.  Once working, I stripped out all unneeded packages for a very minimal 64-bit Ubuntu server.  It boots in seconds, and has no GUI.  Unfortuately, I did not get into the weeds enough to align the partitions/volumes/etc.  I just forgot to do that.  I will have to figure that out next time I get a storage box in.

The management server:

It is also on a very minimal 64-bit Ubuntu-9.04 install.  Ithas four NICs, but I only use two (the other two are only 100 MB).  The two gig NICs are also bonded into one LACP link back to the Cisco, untagged.  The server is running a stripped down 2.6.31 kernel, and has VMware Server 2.0.x installed for the vCenter Server (running on a Windows 2003 server virtual machine).  On the Ubuntu host server, I have installed and configured DHCP, TFTP, and gPXE.  I also extracted the boot guts from the ESXi 3.5.0 Update 4 ISO and set up the tftpboot directory so that each blade will get the image installed.  On the vCenter Server virtual machine, I installed the Microsoft PowerShell tool (which installed ActiveState PERL), and the VMware PowerCLI tool.  I also downloaded the midwife scripts and installed Notepad++ for easy editing.  The vCenter Server VM is on a private 10.x.x.x net for isolated management, but this gets in the way of the Update Manager plugin, so I still have some work to do later to get around this.

Really key things I learned from this:

  1. The blade server VC-Enet modules are NOT layer-2 switches.  They may look and feel that way in some aspects, but they, by design, actually present themselves to network devices as server ports (NICs), not as more network devices.  Learn about them – RTFM.  It makes a difference.  For instance, it may be useful to know that the right side bay modules are placed in standby by default, and the left-side are active – they are linked via an internal 10Gig connection.  I know of another lab with the same hardware that could not figure out why they could not connect the blade modules to the network if all the modules were enabled, so they solved it by disabling all but Bay-1, instead of learning about the features and really getting the most out of it.
  2. Beware old 64-bit CPUs.  Just because it lets you load a cool 64-bit OS on it does NOT mean it will let you load a cool 64-bit virtual machine on it.  If it does not have virtualization instruction sets in its CPU(s), you will run into failure.  I found this out the hard way, after trying to get the RCLI appliance (64-bit) from VMware in order to manage the ESXi hosts.  I am glad I failed, because it forced me to try the PowerCLI/PowerShell tools.  Without those tools, I seriously doubt I could have gotten this project working.
  3. Learn PowerShell.  The PowerCLI scripts extend it for VMware management, but there are plenty of cool tricks you can do using the base PowerShell scripts as well.  I am no fan of Microsoft, so it is not often I express satisfaction with one of their products.  Remember where you were on this day, ‘cuz it could be a while before it happens again.
  4. Name resolution is pretty important.  HA wants it in a real bad way.  Point your hosts to a DNS server, or give them identical hosts files (a little ghetto, but a good failsafe for a static environment).  I did both.
  5. Remember those Enet modules?  Remember all that cool LACP stuff I mentioned?  Rememeber RTFM?  Do it, or you will miss the clue that while the E-net modules like to play with LACP, only one link per V-Net is set active to avoid loops.  So if, on your active V-Net, you have two LACP links, each for a different tagged VLAN, and your NFS devices won’t talk to anyone, you will know that it is because it saw your iSCSI V-Net first, so it set your NFS link offline.  Meaning, the iSCSI link on Bay-1 and it’s offline twin on Bay-2 both have to fail before your NFS link on Bay-1 will come up.  Play it safe – one LACP link per V-Net per bay.  Tag over multiple VLANs on the link instead. The E-net modules only see the LACP links, and do not care if they support different VLANs – only one is set active at a time.
  6. Be careful with spanning tree (this can be said for everything related to networking).    Use portfast on your interfaces to the E-net modules, and be careful with spanning tree guards on the Cisco side.  In testing, I would find that by pulling one of the pairs in a link, it would isolate the VLAN instead of carrying on as if nothing had happened.  Turns out a guard on the interface was disabling the link to avoid potential loops.  Once I disabled that, the port-channel link functioned as desired.
  7. Doesn’t it suck to get everything working, and then not have a clean way to import in VMs?  I mean, now that you built it, how do you get stuff into it?  I ended up restructuring my NFS server and installing Samba as well.  This is because when importing a VM from the GUI (say, by right-clicking on a resource pool), the “Other Virtual Machine” option is the only one that fits.  However, it then looks for a UNC path (Windows share-style) to the .vmx file.  I could browse the datastore and do it that way, but for VMs not on the NFS datastore already, I needed to provide a means for other labs to drop in their VMs.  Samba worked.  Now they can drop in their VMs on the NFS server via Samba, and the vCenter Server can import the VMs from the same place.

Currently, we are restructuring phycial paths between labs for better management.  It is part of an overall overhaul of the labs in my building.  Once done, my next step is to start building framework services, such as repository proxy servers, WSUS servers, DHCP/DNS/file/print, RADIUS/S-LDAP/AD, etc., etc.  I also need to wrap in a management service framework as well that extends to all the labs so everyone has an at-a-glance picture of what is happening to the network and the virtual environment.  One last issue I am fighting is that I am unable to complete importing VMs I made on ESX 3.5 U2 earlier this year.  It keeps failing to open the .vmdk files.  I will have to pin that down first.

The end result?

  1. If I run the midwife service on the vCenter server and reboot a blade, it is reloaded and reconfigured within minutes.
  2. If I upgrade to beefier blades, I pop them in and let them build.
  3. If I update to a newer release of ESXi (say, update 5 or 6), I extract from the ISO to the tftpboot directory and reboot the blades.  The old configs get applied on the new updated OS.
  4. All configs are identical – extremely important for cluster harmony.  No typos.
  5. If someone alters a config and “breaks” something, I reboot it and it gets the original config applied back.
  6. If I make a change to the config, I change it in the script once, not on each blade individually.  This also allows for immediate opportunity to DOCUMENT YOUR CHANGES AS YOU GO.  Which is just a little bit important.

As stated before, this is an overview.  I will add more detailed articles later, which will include scripts and pictures as appropriate.  I am at home now and do not have access to my documentation, but once I get them, I will post some goodies that hopefully help someone else out.  To include myself.

Lost Password – Windows XP Virtual Machine…

Ok, had to reset it.  Here is what I did:

  1. Searches led me to this site:   http://www.petri.co.il/forgot_administrator_password.htm
  2. Trial and error led me to this tool:  Offline NT Password & Registry Editor, Bootdisk / CD
  3. I downloaded the cd080802.iso file.
  4. As I was unable to boot from the iso file itself in the VMware Server 2 web console (don’t ask me why), I was forced to burn this to a CD and boot the VM off of it (it boot s so fast, I had to set it to boot to the BIOS first and make the CD drive the first boot device).
  5. I followed the defaults offered up by the boot CD and reset (blanked) my password.
  6. Removed the CD and booted the VM, got right into my account just fine.
  7. Created a new password.

I won’t soon forget this password now.  What a pain.  Hope this helps – I looked at John the Ripper and a couple others, but this tool really did the trick.

Dual-boot Laptop – Vista and Kubuntu 9.04…

I started last night.  First, I decided to use the 32-bit LiveCD installer.  I booted off the CD after shutting down Microsoft Windows Vista Home Premium, and soon was at the GUI (I chose the first option; to test before installing).  Once there, i opened up a konsole session, ran “sudo -i” to get root, and installed gparted – “apt get install gparted”.  After it installed (to RAM of course), I ran it to see what I could do.

NOTHING.

I could not resize the 140 GB partition Windows called a “C drive”, because I forgot to defragment it first.  Crap.  So I booted back into Windows, Safe Mode.  I found the defrag tool under the System Accessories, but it would not run.  I tried from the command prompt as well.  I rebooted, into SAFE Mode With Console, and it still wouldn’t work.  I finally just rebooted into Vista normally – then it worked.  It gave no status other than a flickering hard drive light and a spinning cue that meant it was not finished.  Eventually, it did finish.  It claimed to have been doing it on a schedule, and the last defrag was back on the 5th of May, yet it took over two hours to complete.  Guess what?  It made all the difference in the world.  I suspect it wasn’t really defragmenting after all.

Once I rebooted into Kubuntu Live CD and reran gparted, I was able to resize it.  The first attempt failed – I cut it too close to the bare minimum space i could shrink the drive.  I decided to split it 50-50, giving about 70 GB for each side, and then it worked.  This took another hour, but I had 70 GB or free space.  I went into cfdisk and manually made a 10 GB bootable partition for root, a 3 GB for /var, a 2 GB for /tmp, a 2 GB for swap, and the rest for /home.  i then rebooted into Windows.

Windows behaved as expected, like it had been punched int the mouth, but didn’t know by whom.  It rescan itself, determined that everything was still ok, and rebooted again.  This reboot came up fine.  Satisfied I had not broken Vista, I rebooted a final time back into the Live CD.

I went ahead and formatted everything with XFS except the swap partition:

  • mkfs.xfs -f -d agcount=1 -i attr=2 -l lazy-count=1,size=128m,version=2 /dev/sda3
  • mkfs.xfs -f -d agcount=1 -i attr=2 -l lazy-count=1,size=128m,version=2 /dev/sda5
  • mkfs.xfs -f -d agcount=1 -i attr=2 -l lazy-count=1,size=128m,version=2 /dev/sda6
  • etc…

I then made the swap partition and then installed, choosing to manually select my partitions and not to format them.   I went to bed, abd when I woke up and checked in the morning, it was done.  I had been unable to get wireless to work (no proprietary drivers needed, just would not work) on the Live CD, so I had connected it up via network cable.  Once I booted into the new system, I saw that it had a GRUB entry for Windows (it works).  After logging into KDE, I was able to set up a working wireless connection with no real drama.  I also modified my /etc/fstab to mount the XFS partitions with the following options:

noatime,nodiratime,logbsize=256k,logbufs=8

I edited /etc/X11/xorg.conf and added in the section to reenable the CTRL-ALT-BACKSPACE zap for X:

Section "ServerFlags"
      Option          "DontZap"               "false"
EndSection

I installed the medibuntu repositories, the kubuntu-restricted package, the sun-java6 package, the non-free flash package, the libdvdcss and libdvdread packages, lots of TTF fonts, the MSTTF core fonts, skype2, firefox, thunderbird, and the packages needed for a DoD smartcard.

Links:

Medibuntu

DoD CAC

Thunderbird setup with AKO

Kubuntu-restricted and Sun-JRE6

Xorg no-zap

Results:  It boots and shuts down much faster than Vista.  It is a Compaq lapto, Pentium Core-Duo, 1 GB RAM, uses the ath5k driver for wireless, has an integrated Intel graphics adapter (maybe 800 fps max on glxgears), and a 160 GB SATA drive.  It has sound, a mic, speakers, a DVD writer, some USB ports, and a network jack.  Overall, not too bad for what I need it to do.  But it is a little shaky and unstable from time to time, so I have shut off the compositing effects and unloaded some troublesome widgets (RSS news widget especially seemed flaky).  But the suspend and hibernate functions work great, and the webcam i bought (Logitech) worked right off the bat with skype.  So did my smartcaard reader.  I also installed the Acrobat Reader from the Adobe website – with it, I added the coolkey security device and am able to sign fillable PDF files with my card.  DVDs also play (region-free, of course).

So, these are my ramblings on the notebook.  I dual-booted because my wife insisted I keep Vista, just in case the Linux machine she is on dumps.  But she is getting more comfortable without Vista already – I can tell.

VMware-Server 2.0 Won’t Start a VM…

The title says it all.  I had taken a snapshot earlier of the VM.  Later, my machine locked for some reason last night, and I had to reset it hard.  Even the magic keys didn’t save me.  Everything seemed to come up ok, but this morning when I tried to start the VM, it failed with the error:

Cannot open the disk ‘/home/vmguests/WinXP/WinXP-000001.vmdk’ or one of the snapshot disks it depends on.  Reason:  Failed to lock the file.

I found that by deleting all the .lck folders in the VM directory (as root), I was able to start it normally.

Impressions of Kubuntu 9.04 and VMware-Server 2.0.1…

So far, RAID-10/LVM/XFS is working quite well with Kubuntu 9.04.  Jaunty picks up hardware effortlessly.  I plugged in a USB thumb drive, and a little notification pops up.

Ok.

I plug in my camera, and it sees it fine,no muss, no fuss.

Better.

I plug in my webcam – no notification, it just works.

Sweeeet.

I plug in my HP printer, and I have to dig around to see that it was added as quietly and politely as you please, ready to print.

Awesome.

I ran out of things to plug in.  Kubuntu 8.04 (the previous version I was using) didn’t boot nearly as quickly, took longer to load the desktop after login, and was good about detecting devices, mostly, but needed polish and charm.

9.04 has it in spades.  I am really quite impressed with the hardware cababilities of it.  There are some programs, like adept, I am missing, but the learning curve for the newer stuff is really more like a learning bump.

Update:  It even loaded the sensors package to track temperatures.  Wow.

I am running 64-bit now, and flash and java work fine.  It took me a while to find the right libjavaplugin and link it into the Firefox plugins folder, but flash 10 worked fine and installed easily.

VMware-Server is a different story.  The 64-bit is slow, flaky, and cranky.  It times out all the time, it resets often, and it just stalls doing stuff.  I now have a VM ready for loading, but it took all day to fight it into doing so.  And I found no reliable cure, to include swapping out the java jre version used for a later version.  I am really dissapointed with the 2.0.1 release in terms of ease of install, performance, and reliability.  Oh well, at least it installed without needing a special patch or script.

Update:  After a huge fight, I got a new Windows XP VM made.  Using the command ‘watch “du -s –si /home/vmguests/WinXP” ‘, I was able to get a sense of the speed of the file system when I was creating the virtual disk files.  I chose to make one large file at once for each of the two disks; C drive (15 GB), and E drive (48 GB).  With the watch command updating every two seconds, I was able to see that the RAID-10 XFS filesystem was handling about 100 Mbps as the disk files were created.

Once I had made theVM, loading it was uneventful.  Just a regular Windows XP professional install, like any other.  The vmware-server played nice mostly after that and has continued to do so.  I have only had to log out once due to unresponsiveness, and have not had to restart the server services.  The VM is quite fast, and allows my wife to see her video streams in Media Player 11 with only minor stuttering of the video.   Audio is fine.

I really like the USB visibilty of vmware-server.  The VM picked up the printer as if it were directly connected, and once I loaded the drivers for it, I was printing from the VM like normal.  All of my USB devices can be presented to the VM, which is an area I had problems with in the past with the 1.x versions of vmware-server.

Anyway, my wife is set up with her login and has a shortcut to RDP to the Windows XP VM, where she can login and watch her JNet streams.

HOWTO – Tunnelling RDP over SSH, Windows to Windows

For when you have a need to access one Windows box from another using RDP (like from XP to Server 2003, perhaps), and you want to do it securely – tunnel it over SSH.

Link for using PuTTY

Link for using OpenSSH for Windows (the link to the SSHSecureShellClient may not work, so you might have to dig around on Google, LIKE THIS)

These are Open-Source tools (not sure about the SSHSecureShellClient – if it isn’t and that bothers you, use PuTTY), so no nag-ware or cripple-ware to deal with.  I have not yet used either of these, but will soon start using the PuTTY option.  Alternative solutions are nice, however.

Thanks to farmdwg for the OpenSSH link, and to the unknown author of the PuTTY link.

Follow

Get every new post delivered to your Inbox.