Mar 29, 2012

Bauer-Power SAN v2.1

I’ve made my last and final improvement to my Ubuntu iSCSI SAN setup that I like to dub the Bauer-Power SAN. In my previous post about this setup I use iSCSI Enterprise Target (IET) as the iSCSI target software, and although it works for most things it is not VMWare certified and cannot be used in Windows clustering. I therefore had to make the change from IET to SCST. You can follow my last to posts on how to set up your SAN and they are:

Bauer-Power SAN v2


How To Install SCST on Ubuntu

The physical hardware is the same with the exception of my disk partitioning which is now setup as such:

Device Mount Point Format Size
/dev/sda1 / ext4 13GB
/dev/sda2 None swap 2GB
/dev/sdb1 /data xfs 18TB



I had to increase the size of the OS partition because after downloading the SCST and Kernel source files I ran out of space. No big deal though. To setup your iSCSI SAN using my method, use the two above articles and replace the IET section with the SCST setup and you should be good to go.

If you have any questions or comments, hit me up below! Tags: ,,

Mar 28, 2012

How To Install SCST on Ubuntu

This post has been a long time coming. I have yet to find a definitive post on installing SCST on Ubuntu Linux. At least, I haven’t found a how-to guide on installing SCST on Ubuntu to setup a simple iSCSI Target SAN. There are posts about setting it up with packages to make virtual tape libraries, and there is one post on how to set it up with Infiniband, but the guy who wrote it was clearly full of shit. First of all, he mentioned a repository where one can install SCST by running sudo apt-get install linux-scst. Well it installs, but doesn’t work. No, the only real way to do this is by compiling it from source because not only do you have to install SCST but you have to compile a custom kernel with the SCST modules in it.

So why SCST and not iSCSI Enterprise Target (IET)? Well, although IET is easier to install and setup, it does not support advanced features like SCSI-3 Persistent Reservations which is required by VMWare and Microsoft Failover Clustering. If you want to build an Open Source iSCSI SAN that supports these features then you need SCST. Neither Openfiler (Unless you want to fork over some cash for their advanced iSCSI plugin) or FreeNAS supports advanced iSCSI features.
SCST A Generic SCSI Target Subsystem for Linux - Google Chrome_2012-03-28_11-37-36
So how to you install it? I thought you would never ask! Please note that I build this on Ubuntu 10.10. You will need to change the kernel number to the number of the kernel you are using. For instance Where I say cd linux-2.6.35, you might be using 2.6.38 or later. Run the ls command in /usr/src to find your kernel version number after you finish section three below.

First install some prerequisites:
#sudo apt-get update
#sudo apt-get install fakeroot kernel-wedge build-essential makedumpfile kernel-package libncurses5 libncurses5-dev subversion
#sudo apt-get upgrade
After that reboot:
#sudo shutdown –r now
When it comes back up run the following so you can compile your custom kernel:
#sudo apt-get build-dep --no-install-recommends linux-image-$(uname -r)
Now it’s time to patch the kernel and compile. Run the following:
#cd /usr/src
#sudo apt-get source linux-image-$(uname -r)
#cd linux-2.6.35

#sudo cp -vi /boot/config-`uname -r` .config
#cd /root
#sudo svn co scst
#sudo cp /root/scst/iscsi-scst/kernel/patches/put_page_callback-2.6.35.patch /usr/src/linux-2.6.35
#sudo cp /root/scst/scst/kernel/scst_exec_req_fifo-2.6.35.patch /usr/src/linux-2.6.35
#cd /usr/src/linux-2.6.35

#sudo patch -p1 < put_page_callback-2.6.35.patch
#sudo patch -p1 < scst_exec_req_fifo-2.6.35.patch

#sudo make && make modules (Note, this takes a REALLY long time)
#sudo make modules_install && make install
#sudo update-initramfs –u
Now reboot to boot up using the new custom kernel with SCST modules installed:
#sudo shutdown –r now
After it comes back up, it’s time to install SCST. Change into /root/scst and run the following:
#make scst scst_install iscsi iscsi_install scstadm scstadm_install
Now lets create a generic config file called /etc/scst.conf
#sudo nano /etc/scst.conf
Paste the following in the config:
HANDLER vdisk_fileio {
DEVICE disk01 {
filename /data/LUN0
nv_cache 1

enabled 1

TARGET iqn.2012-03.bauer-power:iscsi.lun0 {
enabled 1
rel_tgt_id 1

LUN 0 disk01

EDIT: If you want to use thin provisioning, please see my latest post on how to specify thin provisioning for SCST.

The above example creates a LUN called LUN0 which a file created in /data. It can either be a physical disk like /dev/sdb1, or a file. I prefer files for thin provisioning. It also masks the LUN to only two ESXi hosts in the group ESXi, and it’s masked by initiator. This file can be edited either manually or using the scstadmin command, the latter being the best because it doesn’t require a restart of the scst daemon.

After you create the /etc/scst.conf file run the following:
#modprobe scst
#modprobe scst_vdisk
#modprobe iscsi-scst
#scstadmin -set_drv_attr iscsi -attributes enabled=1
#scstadmin -config /etc/scst.conf

#update-rc.d scst defaults
#/etc/init.d/scst restart
You are now ready to serve up some iSCSI goodness! If you have any questions about this setup, hit me up in the comments!

Mar 23, 2012

Bauer-Power iSCSI SAN v2

I posted the other day that I decided to give up on the network RAID configuration I did with my first iSCSI SAN setup using Ubuntu, iSCSI Enterprise Target (IET), ZFS and GlusterFS. The problem was in the data integrity after the LUN files were replicated to the passive node. If you take out the complication of GlusterFS though, the Ubuntu/IET setup is really easy and rock solid. The only issue with IET is that it currently doesn’t support SCSI-3 features like persistent reservations which VMWare likes, and Windows 2008 R2 server requires for failover clustering. I haven’t had an issue with VMWare, but I cannot get failover clustering in Windows 2008 R2 to work with it. The good news is that according to this thread, SCSI-3 should be available in the next version of IET.

Anyway, I decided to rebuild both nodes into stand-alone SAN’s. By themselves they are redundant enough for my liking. I have redundant NICs, the hard drives are in a RAID, I have redundant power supplies. The only singe points of failure on the nodes are the motherboard and the RAID controller really. To mitigate that risk I am purchasing an additional motherboard and a RAID card to have on standby.

The setup for the individual nodes is actually way easier, so I thought I would post this on how I have mine setup so you can do the same. I will be referring to the original post on some things to save time, and because I setup the stand-alones slightly different. Mainly I decided not to use ZFS in my setup because I’m not using the compression or dedupe features. I decided to go with XFS because you can format the storage partition at install time and be done with it. I didn’t go with EXT4 because it has a 16TB partition limit. Also, you can tweak performance of your XFS partition. I found an interesting guide on how to do that here: (Tweak XFS for RAID Performance)

For instance, after some Google searching I found that the default stripe size used on a 3Ware 9750-4i card is 256K, which makes my sunit 512. Since there are 12 drives in RAID 6 that makes my swidth 5120 (10 disks x sunit), therefore to optimally run XFS on my storage I formatted it using the following command:

#mkfs.xfs -d sunit=512,swidth=5120 /dev/sdb1

If you want to go with ZFS, you follow the setup instructions from the original post.


So lets get to the meat, here is an overview of my partition setup:

Device Mount Point Format Size
/dev/sda1 / ext4 9GB
/dev/sda2 None swap 1GB
/dev/sdb1 /data xfs 18TB


If you look at the original post you see that the storage partition lost some space. That’s because I rebuilt the RAID array using RAID 6. The reasoning behind that was by a recommendation from Mike McAfferty, the CEO of M5 Hosting in San Diego. He said that with that much storage, I should have RAID 6 so the array can sustain two drive failures. He said with that much data, the chance of a second drive failing when rebuilding the RAID after a failure is higher. I couldn’t argue with that logic at all. Done!

After Ubuntu is installed, and your storage is partitioned and configured, you need to run the following to install IET, and a few other goodies:

#sudo apt-get install snmpd ifenslave iscsitarget sysstat

I install snmpd so I can monitor the SANs with Zenoss, ifenslave so I can team my NICs and systat so I can monitor I/O performance. I teamed my NICs the same as the original post.

After everything is installed you can setup your storage LUNs using the DD command. Change into /data, and  if you want a thin provisioned LUN here is an example of the command you would use for a 1TB LUN:

#sudo dd if=/dev/zero of=LUN0 bs=1 count=0 seek=1T

If you wanted a 1TB thick provisioned LUN run the following:

#sudo sudo dd if=/dev/zero of=LUN0 bs=1024 count=1T seek=1T

Those commands will create an empty file called LUN0 that our servers will use for storage. You can name that file whatever you want.

Next we configure IET. Change into /etc/iet and edit the ietd.conf file with your favorite editor. You can delete all the crap in the original file. Assuming you created your LUN file with the same name above, you should enter the following in iet.conf:

Target iqn.2012-03.BAUER-POWER:iscsi.LUN0
Lun 0 Path=/data/LUN0,Type=fileio,ScsiSN=random-0001
Alias LUN0

You can add CHAP authentication here too if you want, but I’ll let you Google that. I don’t use CHAP because my iSCSI network is separate, and has no access from outside the network. I do however lock down connections to LUNs by IP addresses. To lock down by IP open the initiators.allow file in /etc/iet, delete all the junk in there and add the following:

Target iqn.2012-03.BAUER-POWER:iscsi.LUN0

That restricts access to LUN0 to only the server with the IP address of After those are configured, restart the IET service by running the following:

#service iscsitarget restart

That’s it, now you are ready to store some data!

If this write up doesn’t make sense, read the original post and fill in the blanks. It’s basically the same setup without ZFS, Heartbeat and GlusterFS. This setup just works, and without the complexity of GlusterFS, I think it’s safer and simpler. This setup also still gives you the option for thin provisioning which is nice if you want to over commit your storage. You can still do the dedupe and compression stuff if you decide to go with ZFS as well, just make sure you have the proper hardware for it.

This setup gives me 18TB of storage for about $6,200. If you have any questions or comments about this setup, let me know in the comments!

Mar 21, 2012

GlusterFS is Not Ready For SAN Storage

Well today is a sad day in Bauer-Power land. Today I have to tuck tail and say that something that I designed and thought was rock solid is not as solid as I would have liked. I am referring to the Bauer-Power SAN that I originally wrote about back in August of last year. In that article I talk about building an iSCSI SAN using Ubuntu, IET, ZFS, Heartbeat and GlusterFS. It all seemed to be going well until a month ago during a data center move.

In my initial tests, I was able to successfully fail the nodes over without issue. However, I think my test wasn't very accurate because the amount of data that I tested was really small. Now that I have been running this thing for 7 months, and have accumulated almost 7TB of data, I am finding that there are some data integrity issues with GlusterFS. What happened was we powered off both nodes in order to do a fork lift move. When we powered them back on, for some reason the data on the primary node wasn't being served by IET, so we failed over to the secondary node to get things rolling. Everything appeared fine for a few weeks, but when I finally brought the primary back up and the data had been out of sync for a while, the GlusterFS healing process corrupted a few things.

Luckily I didn't keep anything mission critical on the cluster, so my rebuild process isn't that big a deal. The issue also doesn't seem to be with Ubuntu or IET. By itself, that is running rock solid. I just don't think mirroring the nodes is safe.

On a separate note, IET isn't certified with VMWare, although I haven't had any issues with it. It does, however, have a serious issue with Microsoft failover clustering. Microsoft will not use IET disks for clustering. Period. Since this issue came up, I am going to setup each node separately, and will purchase a motherboard and an additional RAID card to have on standby in the event of a hardware failure. The drives, NICs  and power supplies are all redundant on their own otherwise.

That being said, I think I will rebuild the primary node with SCST instead of IET because that is VMWare certified and should work with Microsoft failover clustering. If any of you built one of these SAN's, I appologise, but I don't think the clustered setup is ready for prime time. If you haven't done so, I would re-configure the nodes as standalone storage devices, and purchase a backup set of the non-redundant hardware. tags:        


Mar 16, 2012

Another Ode To Epic Meal Time

This video has been a long time coming. If you don't remember, back in November I filmed myself making a ridiculous meat loaf filled with bacon, mac and cheese and McDonalds sausage biscuits with egg and cheese. It was so damn good, and so damn bad for my heart! It was my original Ode to Epic Meal Time, one of the most outragsous cooking shows on Youtube.

Well, my co-workers Frank and Rachel were simply inspired and decided to one-up me by throwing a party and making their own versions. They made two loafs though, one healthy (Bull shit) using turkey, chicken nuggets, french fries, turkey bacon and some other junk; the other was ground beef, real bacon, cheese burgers, onion rings and other junk as well. That means a double dose of epicosity!

Check it out!


Did your arteries clog up while watching? Did it sound good to you? Let me know in the comments! tags:                 

Mar 9, 2012

How To Install Ice Cream Sandwich on HTC Evo 4G

Damn that title makes me hungry… Seriously, doesn’t that make you crave an actual Ice Cream Sandwich? Anyway, enough about that, you know I’m talking about the latest and greatest Android operating system, Ice Cream Sandwich (Android 4.0). It’s not officially available for the HTC Evo 4G, so I am going to give you the next best thing. It’s a custom ROM built on Gingerbread (Version 2.3.7) with a really awesome Ice Cream Sandwich launcher called Syndroid 4.0.

The ROM is based on Cyanogen Mod 7.1, which is built on the Gingerbread Android platform. Even if it isn’t the “real” Ice Cream Sandwich, it is still better than the stock ROM that HTC gives you, plus it doesn’t have all the bloatware that comes with the stock ROM (Amazon MP3 and Blockbuster? Really?). The best thing in my opinion though is the email application which allows you to finally select multiple emails and mark them all as read! Hells yes!

Here’s what you need to install this custom Ice Cream Sandwich themed ROM:

  • You’re EVO must be rooted
  • You need to install Rom Manager from Clockwork Mod (It’s free)
  • You need to download the custom ROM using one of these links: (ICS ROM FTP or ICS ROM Torrent)

First off, if your Evo isn’t rooted, I recommend using Unrevoked3, it’s probably the easiest way to root your Evo. If you don’t know how to perform the rooting procedure, you can Google it.

  • Once your phone is rooted, you can install Rom Manager from the Android Market.
  • Next, download the ISC ROM zip file from the links above and save it to your phone’s SD card.
  • In ROM Manager select the option to install ROM from SD card, and browse to the ICS zip file.
  • Check all the boxes to wipe your cache, and backup your existing ROM, then follow the prompts.

Pretty soon your phone will reboot and go through the setup. Easy peasy, lemon squeezy! Want to see that it will look like? Check out this quick video I made here:


Looks cool? Thinking about doing the same? Have questions? Please sound off in the comments!

Mar 8, 2012

Patrick Stewart Doesn't Tweet, But Does Love The Internet

I found a recent interview with Sir Patrick Stewart, who if you didn't know, played Captain Jean-Luc Picard on Star Trek: The Next Generation for several seasons, and in a number of movie spin-offs. He also played Professor Xavier in the X-men movie series. I should also say that if you did not know who he is, then you need to turn in your geek card right now, and should lay down and friggin' die!

Anyway, back on topic, I found an interview he did on PBS's Digital Nation where he talks about how he loves the internet, and how he uses it in every dday life. I thought I would share it with you. here you go!

I'm sure you feel the same way about technology as he does. I know I certainly do! I guess it makes his view more poignant because it comes from Captain Friggin' Picard! What do you think about what he has to say about the internet and technology? Let me know in the comments! tags:        


Mar 7, 2012

How To Create A Secure Password [Infographic]

I have another infographic for you today. This one is from the folks at ZoneAlarm. It’s funny that I found their infographic, and haven’t heard from them in quite a while. Before Microsoft started including a software firewall in Windows XP Service Pack 2, the place where I started my IT career used ZoneAlarm’s firewall on all of our workstations and it worked pretty good.

Anyway, their infographic on how to avoid bad passwords, and how to create secure passwords is pretty good as well, and follows pretty much everything I’ve been taught on the subject. I decided to send it around my company because I’m sure half of the users are using some form of their kids name, or another dictionary word. Check it out:

[Via ZoneAlarm]

Mar 6, 2012

Numbers Behind Identity Theft [Infographic]

Sorry I haven't been that active on the blog lately. I've been preoccupied with other things at work. Namely I get to rebuild my company's network from the ground up which is something not many admins get to do. I mean, you come into a company and inherit someone elses network, and most of the time the way they did things is not how you would have done it. Sometimes it's nice to be able to start over, and do it the way you feel it should be done.

Anyway, I thought I would post something for you, and these infographics are handy that way. Since I recently did a Tech Chop on protecting your data using encryption, I thought this would be a good follow up post on some numbers associate with identity theft to drive home how important it is to protect your data. Check it out!

Identity Crisis: The Numbers Behind Identity Theft
[Via] tags:     


Mar 1, 2012

Ways You Can Encrypt Pretty Much Everything

New month, new Tech Chop folks! I kind of waited until the last minute for this one, so it didn’t turn out quite as good as I wanted. I also have to start being more conscious of what shirt I’m wearing because I had a hell of the time Chroma keying parts of it out this time. I wore a Punisher shirt that I got from Universal Studios last year, and didn’t realize there were faint traces of green on it. Anyway, if my shirt looks sparkly, that’s why.

Enough about that, in this episode I show you all how to protect your files using encryption. I talk about protecting individual files, full hard drives, email, instant messaging, and even VoIP phone calls. It’s pretty good if you are concerned about the security of your data, and don’t want “The Man” sticking his nose where it doesn’t belong.

Here it is!


The software mentioned in this episode is:

Did you enjoy this episode? Do you use encryption? Do you know of any other tools that one can use to protect data? What do you use? Let us know in the comments!

Twitter Delicious Facebook Digg Stumbleupon Favorites More

Design by Free WordPress Themes | Bloggerized by Lasantha - Premium Blogger Themes | stopping spam