May 24, 2018

A faster and easier way to make LUN files for your SCST SAN

I've been writing a lot lately about SCST iSCSI SANs again. It's been a few years since I've had a chance to configure one of these from scratch, and a lot has changed since 2012 when I first started using these.

In the past I've always used dd to create LUN files for use with SCST. For thin provisioned LUNs I would run something like the following:
sudo dd if=/dev/zero of=lun1 bs=1 count=0 seek=1T
For thick provisioned LUNs I would run this instead:
 sudo sudo dd if=/dev/zero of=lun1 bs=1024 count=1T seek=1T
Well, I found two utilities that do the same thing, but they are way faster and the syntax is way easier! One is called fallocate and the other is called truncate!

To create a thick provisioned LUN, you would use fallocate to create your file by running:
sudo fallocate -l 1T lun1
To create a thin provisioned LUN, you would use truncate to create your file by running:
truncate -s 1T lun1
So simple right? Why am I just learning about this now!?

May 23, 2018

How to specify a thin provisioned LUN in SCST

The other day I wrote about how to install SCST 3.4.0 on Ubuntu 18.04. If you are not familiar with SCST, it is basically a SAN target software that you can run on Linux so you can build your own low cost SAN storage. I've been using it for years, and just recently I've started to learn a few new things about it.

For instance, I used to think that for thin provisioning, all you had to do was to create a thin provisioned disk file to present as a LUN. To do that you just run the following:
sudo dd if=/dev/zero of=lun1 bs=1 count=0 seek=1T
The above creates a thinly provisioned 1TB LUN file called lun1. Simple right?

Well, this is great and all, but if you want to use features like TRIM or UNMAP to reclaim disk space, you also need to tell SCST that this LUN file is a thin provisioned LUN. To do that, you need to add the thin_provisioned parameter to the device section of your /etc/scst.conf file. See below for an example:

 HANDLER vdisk_fileio {  
     DEVICE lun1 {  
         filename /data/lun1  
         nv_cache 1  
         thin_provisioned 1  
     }  
 }  
 TARGET_DRIVER iscsi {  
     enabled 1  
     TARGET iqn.2018-05.bauer-power.net:iscsi.lun1 {  
         enabled 1  
         rel_tgt_id 1  
         GROUP VMWARE {  
             LUN 0 lun1  
             INITIATOR iqn.2018-05.com.vmware1:8bfdfcd0  
         }  
     }         
 }  

After making this change, you can either restart the scst daemon, or reboot your SAN. If you can't reboot the SAN you will have to actually remove the LUN on the fly to make this change. To do that you have to do the following:
  • sudo scstadmin -rem_lun 0 -driver iscsi -target iqn.2018-05.bauer-power.net:iscsi.lun1 -group VMWARE
  • sudo scstadmin -close_dev lun1 -handler vdisk_fileio
  • sudo scstadmin -open_dev lun1 -handler vdisk_fileio -attributes filename=/data/lun1 thin_provisioned=1 
  • sudo scstadmin -add_lun 0 -driver iscsi -target iqn.2018-05.bauer-power.net:iscsi.lun1 -group VMWARE -device lun1
Obviously, you need to change the lun names, file names and target names to match your environment. Special thanks to Gilbert Standen from the Scst-devel mailing list for the above steps on making this change on the fly! Check out his blog here: (brandydandyoracle)

There are a lot of parameters you can add to your config file as well. Here's a list from SCST's Source Forge page:

  - filename - contains path and file name of the backend file.  
  - blocksize - contains block size used by this virtual device.  
  - write_through - contains status of write back caching of this virtual  
   device.  
  - read_only - contains read only status of this virtual device.  
  - o_direct - contains O_DIRECT status of this virtual device.  
  - nv_cache - contains NV_CACHE status of this virtual device.  
  - thin_provisioned - contains thin provisioning status of this virtual  
   device.  
  - removable - contains removable status of this virtual device.  
  - rotational - contains rotational status of this virtual device.  
  - size_mb - contains size of this virtual device in MB.  
  - t10_dev_id - contains and allows to set T10 vendor specific  
   identifier for Device Identification VPD page (0x83) of INQUIRY data.  
   By default VDISK handler always generates t10_dev_id for every new  
   created device at creation time based on the device name and  
   scst_vdisk_ID scst_vdisk.ko module parameter (see below).  
  - usn - contains the virtual device's serial number of INQUIRY data. It  
   is created at the device creation time based on the device name and  
   scst_vdisk_ID scst_vdisk.ko module parameter (see below).  
  - type - contains SCSI type of this virtual device.  
  - resync_size - write only attribute, which makes vdisk_fileio to  
   rescan size of the backend file. It is useful if you changed it, for  
   instance, if you resized it.  

Pretty cool right? Let us know what you think in the comments!

May 18, 2018

How to install SCST 3.4.0 in Ubuntu 18.04

Well crap. The other day I talked about how I re-configured one of my Bauer-Power iSCSI SANs using tgt. It was an easy setup, but once I started using it I noticed that tgt performed like shit. CPU's were spiking like crazy on the SAN itself, and when I was backing stuff up I couldn't access the drive on the backup server. It would get completely unresponsive!

I decided I had to go back to SCST. Luckily installing it is way easier than it used to be. To install version 3.4.0 now just do the following:
  • Create an empty working directory
 rm -rf ~/scst-build  
 mkdir ~/scst-build  
 cd ~/scst-build  
  • Install dependencies
 sudo apt install git devscripts equivs dkms 
 git clone -b ubuntu-3.4.x https://github.com/ubuntu-pkg/scst.git  
 cd scst  
 sudo mk-build-deps -i -r  
  • Build the package
 dpkg-buildpackage -b -uc  
  • Pre-install, create two directories (For some reason the deb packages don't do it...)
 sudo mkdir -p /var/lib/scst/pr  
 sudo mkdir -p /var/lib/scst/vdev_mode_pages  
  • Install
 sudo dpkg -i ../scst-dkms_*deb  
 sudo dpkg -i ../iscsi-scst_*.deb  
 sudo dpkg -i ../scstadmin_*deb  

Now you just have to configure your LUN using the instructions in my tgt post, and configure your /etc/scst.conf file using my old SCST post. Once those are done restart the scst service.

 sudo service scst restart  

Of course, if you don't want to mess with all of the above stuff, you could just download my pre-packaged scst 3.4.0 deb files for Ubuntu 18.04 and run my install script...

 cd ~  
 wget https://mail.bauer-power.net/drop/scst/scst-3.4.0-Ubuntu.tgz  
 tar -xzvf scst-3.4.0-Ubuntu.tgz  
 cd scst*  
 sudo chmod +x install.sh  
 sudo ./install.sh  

Now just setup your LUN files, create your /etc/scst.conf file and run the following commands:

  • modprobe scst 
  • modprobe scst_vdisk 
  • modprobe iscsi-scst 
  • iscsi-scstd 
  • scstadmin -set_drv_attr iscsi -attributes enabled=1 
  • scstadmin -config /etc/scst.conf
  • update-rc.d scst defaults
  • /etc/init.d/scst restart

Boom! Now you are off to the races!

May 17, 2018

How to Re-IP An OSSEC Agent

At my day job we use OSSEC for host based intrusion detection. It works great! It does all sorts of things from verifying registry integrity, checking files for changes, reading security logs etc., and sends email alerts for anything out of the ordinary.

Well, we're in the process of migrating servers from on-premise to Azure, so that means that some of our servers are getting new IP addresses. Googling around, I didn't find a good way to re-IP the agents except to remove them, and re-add them. I didn't want to do that.

It turns out, there is an easier way. All you have to do is edit /var/ossec/etc/client.keys with your favorite text editor and modify the IP address of the client you want to change. If you don't want to deal with this in the future, you can replace the IP address with 'any' so that OSSEC will accept connections from that client as long as the hostname and the client key match.

After you make your change, restart the OSSEC daemon on your OSSEC server:
sudo service ossec restart
Re-run /var/ossec/bin/manage_agents and extract the key again for the agent you want to update. Then on the client, open OSSEC Agent Manager as an administrator, click Manage > Stop OSSEC, re-paste the key, click Save, then restart OSSEC by clicking Manage > Start OSSEC.

Boom! Done! You should now be able to connect using the new IP address or 'any'.

May 16, 2018

Bauer-Power SAN 3.0

NOTE: Please read my post about installing SCST on Ubuntu 18.04 first...

Many moons ago I wrote about how to configure an Ubuntu Linux based iSCSI SAN. The first iteration used iSCSITarget as the iSCSI solution. The problem with that is that it didn't support SCSI-3 Persistent Reservations. That means it wouldn't work for Windows failover clustering, and you would probably see issues if you were trying to use it in VMWare, XenServer or Hyper-V.

The second iteration used SCST as the iSCSI solution, and that did work pretty well, but you had to compile it from source and the config file was kind of a pain in the ass. Still though, it did support SCSI-3 Persistent Reservations, and was VMWare ready. It's the solution I've been using sing 2012 and it's worked out pretty well.

Well the other day I decided to rebuild one of the original units I setup from scratch. The first two units I did this setup on were SuperMicro SC826TQ's with 4 NICs, 2 quad core CPUs and 4GB of RAM, 3Ware 9750-4i RAID Controller, and twelve 2TB SATA Drives. This sucker gave me about 18TB of usable backup storage after I configured the 12 disks in RAID 6.

This time I used Ubuntu 18.04 server because unlike the first time I did this, the latest versions of Ubuntu have native drivers for 3Ware controllers. On top of that, the latest versions of Ubuntu have the iSCSI software I wanted to use in the repositories... More on that later.

I partitioned my disk as follows:

Device Mount Point Format Size
/dev/sda1 N/A bios/boot 1MB
/dev/sda2 / ext4 10GB
/dev/sda3 N/A swap 4GB
/dev/sda4 /data xfs 18TB

After Ubuntu was installed I needed to setup my network team. Ubuntu 18.04 uses Netplan for network configuration now, which means that NIC bonding or teaming is built in. In order to setup bonding or teaming you just need to modify your /etc/netplan/50-cloud-init.yaml file. Here is an example of how I setup my file to team the four NICs I had, as well as use MTU 9000 for jumbo frames:


network:
    version: 2
    ethernets:
        enp6s0:
            dhcp4: no
            dhcp6: no
            mtu: 9000
        enp7s0:
            dhcp4: no
            dhcp6: no
            mtu: 9000
        enp1s0f0:
            dhcp4: no
            dhcp6: no
            mtu: 9000
        enp1s0f1:
            dhcp4: no
            dhcp6: no
            mtu: 9000
    bonds:
        bond0:
            interfaces: [enp6s0, enp7s0, enp1s0f0, enp1s0f1]
            mtu: 9000
            addresses: [100.100.10.15/24]
            gateway4: 100.100.10.1
            parameters:
                mode: balance-rr
            nameservers:
                addresses: [8.8.8.8, 8.8.4.4]


It's important to note that Netplan is picky about indentation. You must have everything properly indented or you will get errors. If you copy the above config, and modify it for your server, you should be fine though.

After setting up my bonded network, I installed my software. I opted to use tgt this time. If you are unfamiliar with it, it's apparently a re-write of iscsitarget, but it supports SCSI-3 Persistent Reservations. I tested it myself using a Windows Failover Cluster Validation test:



Boom! We're in business!

To install tgt simply run the following:
sudo apt-get install tgt
 After installing you will want to create a LUN file in /data. To create a thin provisioned disk run the following:
sudo dd if=/dev/zero of=/data/lun1 bs=1 count=0 seek=1T
This creates a 1TB thinly provisioned file in /data called lun1 that you can present to iSCSI initiators as a disk. If you want to create a thick provisioned disk simply run:
sudo dd if=/dev/zero of=/data/lun1 bs=1024 count=1T seek=1T
Once you have your LUN file, you will want to create a config file for your LUN. You can create separate config files for each LUN you want to make in /etc/tgt/conf.d. Just append .conf at the end of the file name and tgt will see it when the service restarts. For our purposes, I created one called lun1.conf and added the following:

<target iqn.2018-05.bauer-power.net:iscsi.lun1>
        backing-store /data/lun1
        write-cache off
        vendor_id www.Bauer-Power.net
        initiator-address 100.100.10.148
</target>


The above creates an iSCSI target and restricts access to it to only 100.100.10.148. You can also use initiator-name to restrict access to particular iSCSI initiators, or you can use incominguser to specify chap authentication. You can also use a combination of all three if you want. Restricting by IP works for me though.

I also opted to disable write-cache because with it enabled I noticed that tgt was pegging my RAM. On top of that, my RAID controller handles write-cache on it's own, so it actually helped my performance to disable it.

All of this being said, you can find lots of configuration options here: (tgt Config Options)

After you have your file created, all you have to do is restart the tgt daemon and you're ready to serve up your iSCSI LUN!
sudo service tgt restart
After you restart, you can see your active LUNs by running:
sudo tgtadm --op show --mode target
You can also create LUNs on the fly without restarting tgt. This is handy if you need to add a LUN and you don't want to mess up connections to LUNs you've already created. To do that, create your LUN file like you did before. Obviously, name it something new like lun2.

Next,make sure to note what LUNs you already have running by running this command:
sudo tgtadm --op show --mode target
Target 1 = tid 1, Target 2 = tid 2 and so on and so forth. If you only have one target, then your next target will be tid 2. Assuming that, and assuming your new LUN file is called lun2 you would run:

sudo tgtadm --lld iscsi --op new --mode target --tid 2 -T iqn.2018-05.bauer-power.net:iscsi.lun2
sudo tgtadm --lld iscsi --op new --mode logicalunit --tid 2 --lun 1 -b /data/lun2
sudo tgtadm --lld iscsi --op bind --mode target --tid 2 --initiator-address 100.100.10.148

This will create a target, and will be available only to 100.100.10.148. If you wanted to allow other IP's re-run that last line for each IP address you want to allow.

Now if you want to have this LUN persist after a reboot, you can either manually create a conf file in /etc/tgt/conf.d/ or you can run the following to automatically create one for you:
tgt-admin --dump | sudo tee /etc/tgt/conf.d/lun2.conf
The only issue with the above is that it dumps all running target information in you new file. You will have to go in there and remove the other targets. In this case, it's just better to manually create the config file... but that's just me. Also, that is not a typo... tgt-admin is a different tool than tgtadm... Weird right?

Anyway, this setup is way easier than SCST ever was. I'm looking forward to replacing all of my SCST SANs with tgt in the upcoming months.

It's important to note that using the above hardware is not going to give you high performance. It's suitable for backup storage, and that's about it. If you want to run VMs or databases, I'd recommend getting 10GBe switches for use in iSCSI. You can get one fairly cheap here (10GBe switches). If you get 10GB switches, you will need a 10GB NIC as well. You can get one here (10GB NICs). Finally you will need faster disks. You can get 15K RPM SAS disks here (15K RPM SAS).

What do you think about this setup? Are you going to try it out? Let us know in the comments!



May 10, 2018

Script to Clone Azure Network Security Groups (NSGs) in PowerShell

This script is a life saver! I am working on a big project to migrate to Azure, and one of the tedious parts of the project is setting up Network Security Groups, or NSGs. My company uses many granular rules, so setting these up the first time is time consuming. The idea of manually setting them up in other regions is down right daunting!

Well, not anymore! I found this series of commands from Virtual Geek that lets you do it easily in PowerShell!

First, you need the Azure PowerShell module if you don't already have it. After that, run the following:
$TemplateNSGRules =  Get-AzureRmNetworkSecurityGroup -Name '<Original NSG>' -ResourceGroupName '<Resource Group of Original NSG>' | Get-AzureRmNetworkSecurityRuleConfig
This creates a variable called TemplateNSGRules that we will use in step three. Next create your new NSG by running the following:

$NSG = New-AzureRmNetworkSecurityGroup -ResourceGroupName '<Destination Resource Group>' -Location '<Region Where You Want The New NSG>' -Name '<Name of New NSG>'
If you have already created an NSG in the portal, you would use this instead:

 $NSG = Get-AzureRmNetworkSecurityGroup -Name '<Name of New NSG>' -ResourceGroupName '<Destination Resource Group>'
Once you have executed one of the previous two commands, you will have a new variable called NSG that we will run a foreach loop on to import our rules from the original NSG:

 foreach ($rule in $TemplateNSGRules) {
    $NSG | Add-AzureRmNetworkSecurityRuleConfig -Name $rule.Name -Direction $rule.Direction -Priority $rule.Priority -Access $rule.Access -SourceAddressPrefix $rule.SourceAddressPrefix -SourcePortRange $rule.SourcePortRange -DestinationAddressPrefix $rule.DestinationAddressPrefix -DestinationPortRange $rule.DestinationPortRange -Protocol $rule.Protocol # -Description $rule.Description
    $NSG | Set-AzureRmNetworkSecurityGroup
}
Boom! That's it! Now you have an exact clone of your original NSG in just a few minutes! Make sure you replace the items I used in < > to fit your environment!

Did this help you out? Let us know in the comments!

May 7, 2018

Like apt-get for Windows! Meet Chocolatey!

I'm surprised I haven't written about this already. I've known about it for several years now, so I thought I would have wrote about it before now... I guess I was wrong.

Anyway, I started thinking about Chocolatey again today when I was asked to come up with a way to easily handle third party application patches. There are tools out there that do it, but Chocolatey is free and it works pretty much the same way that apt-get does in Ubuntu. That means, you can script it and automate it!

If you are unfamiliar with Chocolatey, this is a description from their page:
Chocolatey is a package manager for Windows (like apt-get or yum but for Windows). It was designed to be a decentralized framework for quickly installing applications and tools that you need. It is built on the NuGet infrastructure currently using PowerShell as its focus for delivering packages from the distros to your door, err computer. 
Chocolatey is a single, unified interface designed to easily work with all aspects of managing Windows software (installers, zip archives, runtime binaries, internal and 3rd party software) using a packaging framework that understands both versioning and dependency requirements. Chocolatey packages encapsulate everything required to manage a particular piece of software into one deployment artifact by wrapping installers, executables, zips, and scripts into a compiled package file. Chocolatey packages can be used independently, but also integrate with configuration managers like SCCM, Puppet, and Chef. Chocolatey is trusted by businesses all over the world to manage their software deployments on Windows. You’ve never had so much fun managing software!
If you want to use it for 3rd party software updates, you can install Chocolatey, then just run a scheduled task that runs the following command:
C:\choco update all -y
It's important to note that Chocolatey will only update software that you've installed with Chocolatey. So if you already have Adobe Reader, Java, Flash etc. You will first need to run the install commands for these applications with Chocolatey before you can start getting updates. You don't have to uninstall and re-install though which is nice.

For instance, I already had 7zip installed, but now I want to make sure I get updates for it with Chocolatey, so I ran the following to install the latest version of 7zip:
C:\choco install 7zip -y
You can find a full list of their packages here:  https://chocolatey.org/packages

What do you use to keep your third party software up to date? Let us know in the comments!

May 4, 2018

STOP: 0x0000007B BSOD After Restoring UrBackup Image to XenServer VM

Sorry I haven't been writing very much lately. I've been completely slammed at my day job. I'm juggling many different projects, trying to chase down consultants, putting out fires, training new hires and guys who just got promoted, etc etc.

One of the projects I'm working on is setting up UrBackup for full image backups as well as file level backups. We've been using CrashPlan for years, but that only really give us file level backup capabilities. The other day we had a backplane on one of our SAN units take a shit, and we lost connection to our storage for a bit. Luckily everything came back up fine, but I got to thinking what an epic pain it would be to rebuild some of our servers with just the file backups.

So after originally dismissing UrBackup a little while back, I decided to take another look at it. It turns out it is pretty bad ass! I was able to take an image backup of one of our VMWare VMs and restore it to a blank VM in about 20 minutes!

So it obviously worked great with a VMWare VM, but we also use XenServer pretty heavily in our environment. I wanted to test a restore on that as well. That didn't go so well.

You see I was backing up a Windows 2008 R2 VM, and when I went to restore it to a blank Windows 2008 R2 VM in XenServer I got this blue screen of death message!

STOP: 0x0000007B

Oh hell, what is that about?

Anyway, Googling it I found some forums where people say to run the following command in the XenServer terminal:

xe vm-param-set uuid=<UUID of the VM> platform:device_id=0001

Pro tip, that is bullshit. It didn't work at all.

You know what did work? Creating the blank VM using the Windows XP SP3 (32-bit) template!



Once I did that, and ran the restore again, the VM booted up just fine!

I don't know what is up with that template, but it's the one size fits all, never fails template. Plus, it doesn't matter if you are running a 64 bit OS or not!

I once wrote about issues with Ubunu in XenServer and the fix for that was to use a Windows XP template too!

Anyway, if you run into this issue. Try giving the Windows XP template a shot. You can thank me later!

If you need more than 4GB of RAM for your VM, you could also try the Windows 2003 64 bit template. It should work too.



Twitter Delicious Facebook Digg Stumbleupon Favorites More