Setting Up A HAProxy Failover Cluster

So it looks like I am going to get some time to write about my setup after all. If you read my post yesterday, I recently setup a failover load balancing cluster for my company’s public web servers. My previous company did something similar with two F5 Big-IP appliances for about $90,000 or so. I however did it completely for free using Ubuntu Linux and HAProxy!

I mentioned in my post yesterday how to install HAProxy on Ubuntu 11.04, and what you need to do to get it to start correctly at boot up. Today I am going to tell you how to add a second HAProxy server, and configure failover using Keepalived.

• Stand up two identical Ubuntu servers and install HAProxy using my instructions here: (How to Install HAProxy)
• Configure your /etc/haproxy/haproxy.cfg file on Node1, and copy to to Node2 using scp sudo scp /etc/haproxy/haproxy.cfg root@node2:/etc/haproxy/       
• Install Keepalived on both servers  sudo apt-get install keepalived   
• On both servers edit the /etc/sysctl.conf with your favorite text editor (i.e. sudo nano /etc/sysctl.conf) and add the following at the bottom of the file so HAProxy can bind with our virtual IP address(es).  net.ipv4.ip_nonlocal_bind=1
  
• Run sudo sysctl –p
• Run sudo nano /etc/keepalived/keepalived.conf to create your Keepalived configuration, and paste in the following, make sure to change the IP address to match your VIP (Virtual/shared IP address):  

  
  vrrp_script chk_haproxy {           # Requires keepalived-1.1.13
          script "killall -0 haproxy"     # cheaper than pidof
          interval 2                      # check every 2 seconds
          weight 2                        # add 2 points of prio if OK
  }
  
  vrrp_instance VI_1 {
          interface eth0
          state MASTER
          virtual_router_id 51
          priority 101                    # 101 on master, 100 on backup
          virtual_ipaddress {
              192.168.0.99
          }
          track_script {
              chk_haproxy
          }
  }
  
  
  
  

• Do the same on Node2, but change the priority to 100 to make Node2 the slave.
• Run the following to start Keepalived:   sudo /etc/init.d/keepalived start   
• Run the following on Node1 to check to make sure it is listening to your virtual IP address (VIP/Shared IP):  ip addr sh eth0   
• Start Keepalived on Node2, and run the command above to make sure Node2 is NOT listening to the VIP.
• If it’s not running already, start HAProxy on both servers by running the following:
  
  sudo /etc/init.d/haproxy start
  
• To test, you can run a constant ping on the VIP and reboot Node1, you should only see one dropped packet if done correctly.

Pretty cool right? One thing to node is that you can have your HAProxy cluster server load balancing to more than one server farm, with multiple IP’s. Just add the multiple VIPs to your /etc/keepalived/keepalived.conf file. For instance, lets say you have to IP’s you want to act as VIPs for two sets of web servers. One VIP is going to be 192.168.0.98 and the other is going to be 192.168.0.99 then add the two IP’s to your config in the virtual IP section like so:

virtual_ipaddress {
            192.168.0.98

             192.168.0.99

        }

If you have to add more IP’s later, you can do so and simply restart the Keepalived service. As far as how to create an HAProxy config file, I am going to let you look that up on your own. There are many different settings you can configure for HAProxy including both HTTP modes and TCP modes. I will tell you that I am using HTTP mode for port 80 HTTP traffic, and TCP mode for HTTPS/SSL traffic.

What do you use for load balancing? An appliance like Netscaler, or F5? Maybe an open source option like Pound that provides SSL offload? Let me know what you use and why in the comments.

Via [HowtoForge]

del.icio.us Tags: how to,setup,haproxy,failover,cluster,load balance,keepalived,linux

Related articles

• Client-side Load Balancing Web 2.0 Apps is Voodoo (blogs.sitepoint.com)
• Have the latest HAProxy as a Ubuntu Service (gubatron.com)

Read more »

How To Install HAProxy on Ubuntu 11.04

Happy Monday everyone! I have a pretty good one to share with your today. Over the weekend I stood up an open source load balancer running on Ubuntu Linux and running a free tool called HAProxy. I was originally going to use another load balancing platform called Pound that also does SSL offload, but I decided against it because it would require me changing the way my company’s firewall is configured to ensure that the sites that need to be secured, stayed secured.

I decided to go with HAProxy not just because it is simple in that it can load balance anything, including SSL encrypted sites, but because it doesn’t natively do SSL offload on its own, and lets the web servers handle that. With HAProxy I was able to stand it up in our existing environment without having to change a damn thing on the firewall.

So why the write up on how to install it? I mean, it’s available in the repositories right? Yes it is, but after installing it there are a couple of things you need to do to make sure it starts up when you reboot, especially on Ubuntu 11.04.

So here it is, how to install HAProxy:

• Run sudo apt-get update
• Run sudo apt-get install haproxy
• After the install, run sudo nano /etc/default/haproxy and change ENABLED=0 to ENABLED=1 and save it

Normally, you can stop here, but for some reason in Ubuntu the /etc/init.d/haproxy script tries to start before networking and fails. To fix that run the following:

• sudo update-rc.d -f haproxy remove
• sudo update-rc.d -f networking remove
• sudo update-rc.d haproxy start 37 2 3 4 5 . stop 20 0 1 6 .
• sudo update-rc.d networking start 34 2 3 4 5 .

What that does is changes the order of the startup scripts so that Networking has a chance to start before HAProxy. After that you can reboot all you want and HAProxy will start right back up.

I am going to try to find time to write more about my HAProxy setup sometime this week (Maybe tomorrow). It’s pretty cool, and it is allowing me to load balance four webservers each with a mix of SSL HTTPS and standard port 80 HTTP traffic.

Via [Ubuntu Bugs]

del.icio.us Tags: how to,install,haproxy,ubuntu,11.04,won't start,boot

 

Related articles

• Have the latest HAProxy as a Ubuntu Service (gubatron.com)
• How to solve: No wireless network in Ubuntu 11.04 : Centrino Wireless-N 1000 (computerandu.wordpress.com)
• SSL|Detailed analysis of SSL Certificate (techtimely.wordpress.com)

Read more »

Recovering Deleted Offline Files in Windows 7

I haven’t had to write too much about desktop related issues recently. I’ve been mainly involved with higher level systems for the past few years, but in my new gig since I am the only IT person I have to do everything including Desktop support.

Well I got hit up by a pretty frantic user yesterday. Apparently for the last two weeks they were working on a bunch of documents in their offline file cache and didn’t realize it. Something happened and all of a sudden their cache got wiped out and everything they were working on disappeared. Holy crap right?

I went to look at their sync center in Windows 7 to see if I could browse their offline files, but they were gone. I had never really dealt with a situation like that, so of course I hit up Google. It turns out that Windows 7 saves your offline file cache to C:\Windows\CSC\v2.0.6. If you try to go in their you will get an access denied notice. It doesn’t matter if your are an administrator or not. The only way to take ownership of this particular file is to run the takeown command from command line:

takeown /f c:\windows\csc /r /a

After that you can get in, but guess what, his cache was completely empty. Flipping awesome! I right clicked on CSC and looked at previous versions, sure enough there were versions from the day before, but when I tried to restore the previous versions I would get access denied errors! Damn it! I even tried modifying the permissions in the previous version folder to take ownership, and couldn’t do that!

After a while of Googling around and seeing nothing, I decided to shoot for the stars and try my old free buddy Recuva. Guess what? Recuva was able to restore about 5000 documents from the users cache!

After scanning the hard drive for missing documents, I sorted the results by file path. I restored everything I saw in C:\Windows\CSC to a network share. The user is happy, I’m happy, the issue is resolved.

Have you run into this before? Were you able to recover the files? How did you do it? Is there an easier way? Let me know in the comments.

del.icio.us Tags: how to,recover,restore,deleted,offline files,cache,csc,windows 7,access denied

 

 

Related articles

• Offline files in Windows ? (testingunderground.wordpress.com)
• How To Recover Lost Files In Dropbox (makeuseof.com)
• Folder Redirection and Offline Files Long Login times (edugeek.net)

Read more »

How To Create a HA Active/Passive Openfiler Cluster

Greetings fellow geeks! If you have been following along with me, I am thinking about changing SAN technologies in the office. The reason being is cost savings, pure and simple. Right now we have a NetApp FAS2020 which goes for $24,000. We are starting to run low on disk space, and instead of forking over another $24,000 I felt I could get something that does what we need for less.

Enter Openfiler ESA. For those that don’t know, Openfiler is a free and open source NAS/SAN operating system that runs on Linux. It can act as a simple file server, or if you want you can configure it as an iSCSI, or even a Fibre channel SAN. Not only that, but what sets it apart from other free NAS operating systems like FreeNAS is that you can actually configure Openfiler in a HA Active/Passive cluster for redundancy. Hell, you can even configure it for offsite replication for disaster recovery!

So here is my plan, I want to get two SuperMicro storage chassis with hardware RAID controllers. I will configure each chassis with hardware RAID 5. Then I will create a software network mirror of the two nodes using the HA setup for Openfiler using DRBD, Corosync and Pacemaker.

Here is a drawing of my plan to give you a visual:

I am not going to write out the full how-to because it is kind of a lengthy process. I will however direct you to the how-to I used in my test environment that worked out really well here: (Cluster Openfiler)

I will point out some things that I felt the author left out, or didn’t explain though:

• Your Meta partition only needs to be 512MB, however your Data partition needs to be the full size of the storage you want to use for your SAN. I created my partitions using cfdisk. They must be logical partitions. Also, your data partition needs to be an LVM which is type 8e in cfdisk/fdisk. It’s important that you don’t create these partitions during the installation!
• In the section where you create your LVM filter, the file you want to edit is /etc/lvm/lvm.conf

Other than that, I think the how to was pretty informative, and well written. If you follow it exactly, you should have your HA setup in no time!

Have you ever setup an Openfiler cluster? How do you like the performance? If you’ve setup an Openfiler cluster, have you ever used another SAN technology? how do you think it compares? Let me know your opinion in the comments!

del.icio.us Tags: how to,configure,ha,failover,cluster,openfiler

 

Related articles

• Combine All Your Hard Drives Into One Seamless Storage Container With LVM [Linux] (makeuseof.com)
• How do you handle your Linux based File Server? (edugeek.net)
• LVM snapshots with duplicity encrypted backup (kc0rjx.wordpress.com)

Read more »

Can't Add Partitions in Openfiler 2.99

Greetings all! I hope you had a good 3 day weekend. I certianly did. I got to spend some time with the family, and some time researching some low cost storage solutions. I didn't have to research them of course, but I love technology so I like doing that sort of thing in my spare time.

Anyhoo, the reason I was looking into low cost storage solutions is because my company currently has a NetApp FAS2020 SAN. It's basically the lowest end NetApp one can buy. Well, my company has managed to just about use up all the storage on it, and they have budgeted for another FAS2020 purchase this year. Our vendor TIG has quoted us at $24K or something. Not bad right?

Well, my company operates on a shoe string, and besides the storage there are other things I want to buy. If we blow all our money on a high priced SAN like NetApp, that leaves me no money to get anything else I need. One thing I learned from one of my previous managers was that in a company with litle cash flow, the way you get your money for IT is in cost savings. Therefore I decided that instead of buying another NetApp, I could make my own SAN using SuperMicro Chassis's and Openfiler.

For about $15,500 I can build a redundant 6TB SAN using 24 600GB SAS drives. For a little over $6,000 I can make a redundant 12TB SAN with SATA drives. It all comes down to performance right? For those of you who didn't know, with Openfiler, not only can you create iSCSI targets, you can also cluster two nodes together in an HA Active/Passive cluster. Pretty cool right?

Anyway, now to get back on topic. One of the things I've been doing this weekend is playing with Openfiler. It's been a while since I've messed with it so I thought I would load it in some VM's a screw around. One of the things I noticed was after I booted up my VM, I had about 100GB of free space. I wanted to create an iSCSI target with that 100GB. For some reason I couldn't!

Every time I went to create a new partition the page would refresh and nothing would happen. I finally figured out what I needed to do. I'm not sure if the Openfiler people know about this bug or not, but if you leave the default cylinder numbers it will never work. What I found out was I had to change the beginning cylinder number to be 60 cylinders more that what was displayed.

For example, here is what was displayed by default:

I added 60 to the starting cylinder:

And after that I was able to create the partition! Now 60 really is the magic number. I tried adding 10 at a time until it finally worked, and it didn't work until I added 60.

Besides that one little thing, everything else worked as expected with Openfiler 2.99. Have you experienced this in Openfiler? Is it a bug? Do you know why it works that way? Let us know the comments.

del.icio.us tags: san, under $20,000, openfiler, bug, can't add partition, can't add volume, supermicro

Related articles, courtesy of Zemanta:

• 2TB iSCSI limit
• In an iSCSI context, what do LUNs mean?
• Openfiler version 2.99

Read more »

Windows Powershell 1.0 Missing From Add/Remove Programs!

I ran into an interesting thing at work. I wanted to perform an in-place upgrade of a Windows 2003 R2 domain controller to Windows 2008. When I ran the upgrade utility it said that before I could continue that I have to uninstall Windows Powershell. Easy enough right? Well, not in this case.

I went into add/remove programs and neither Windows Powershell was in there, and neither was the update associated with it which is KB926139. WTF? How am I supposed to remove it if it's not there to remove? I hate this kind of stuff, and this is why most people don't go through with in place upgrades. The only reason I'm doing it is it's the quickest way for my to get my environment up to a 2008 functional level in Active Directory without have to do a whole lot of FSMO role changes, and what not.

Anyhoo, I was finally able to remove it, but in order to do it I had to trick Windows into thinking it wasn't there by doing the following:

• In the registry I exported the following key then deleted it:

  HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\PowerShell

• After that, I downloaded KB926139 and manually installed it
• In add/remove programs I found the KB926139-V2 Update, and uninstalled Powershell

After that I was ready for my in-place upgrade to Windows 2008. Were you in a similar situation? Did you find a different way to fix it? Let me know in the comments.

del.icio.us tags: how to, uninstall, windows powershell, missing, add remove programs,

Related articles, courtesy of Zemanta:

• Windows PowerShell 2.0 Core Help - May 2011 Update
• Automate the installation of Active Directory tools with PowerShell
• Managing Registry via Powershell

Read more »

I'm Switching To Open Source Virtualization

You read that right my friends. Even though I am VMware certified, and I love their products the fact is that the new company I work for is on a shoe string budget and can't afford the expensive cost of VMware. Now we do have a 3 node ESX cluster using VMware advanced edition, but that is the limit my company wants to spend which is to say it costs a lot just for that.

That cluster is in our colo and hosts my company's production websites which are our money makers, so of course they were willing to spend money on that, but my home office environment? That is a different story. It is completely done with whatever the hell the previous admin could put together to make things work. Don't believe me? Here is a picture of the server room I inherited:

Yes, those are all just stacked on each other without rails, and yes the top four boxes are desktop computer running as servers. I am not making this shit up! Not only that but if you can't tell, the bottom servers are old HP DL360 G3's! Those came out in 2004! WTF?

Well, I can certainly do better I think even with a limited budget. What I want to do is buy two low end SuperMicro servers with a crap load of RAM, and an 8TB iSCSI NAS. Then on the two servers I want to install a really cool open source virtualization system built on Debian Etch and uses KVM and OpenVZ virtualization technologies. It is called Proxmox VE. Unlike other free virtualization servers like ESXi, it supports clustering and live migration! That means I can start with the two servers, and as the business grows, I can just add more servers to the cluster! Mmmm, smells like scalability!

Also, if I even need to do maintenance I can migrate my VMs over to the other node while I take the other one offline. Smells like 3 nines of uptime baby! Also it supports backups of full VMs all in one easy to use web based console!

Here is a video I grabbed off the Proxmox wiki showing how easy it is to setup a two node cluster:

 

 

I know SuperMicro servers are really cheap, but they can hold way more RAM than the DL360 G3's which only hold 4GB of RAM each. Plus the 8TB NAS I want to get runs on SATA drives, but then so do the desktop servers above. My plan is to setup the two servers, with the iSCSI and virtualize all the other servers in the room and get rid of it all.

I think with the size of my company a two node Proxmox VE cluster will keep us running at the home office fore quite some time. What do you think? Are you going to give Proxmox VE a try? Do you use a different alternative to VMware in your environment? Let us know in the comments.

del.icio.us tags: free, open source, alternative to, vmware, esx, proxmox ve, cluster, live migration, vmotion

Read more »