ZFS + GlusterFS on Linux

Time is almost ripe for start using the native ZFS port on Linux (http://zfsonlinux.org/), and to increase the performances, reliability and space usage of our affordable distributed opensource storage solution.

Installing ZFS on Debian/Ubuntu is straightforward: you need first to build the SPL (Solaris Porting Layer) and after ZFS itself.

Download the latest SPL package, unpack it and build:

sudo apt-get install build-essential gawk alien fakeroot linux-headers-$(uname -r)
./configure
make deb
dpkg -i *.deb

Download the latest ZFS package, unpack it and build:

sudo apt-get install zlib1g-dev uuid-dev libblkid-dev libselinux-dev parted lsscsi
./configure
make deb
dpkg -i *.deb

ZFS is now ready to be used.

Let’s create our first pool with:

zpool create tank raidz [devices]

Devices can be partitions, UUIDs or entire disk. Is often a very good practice to use an entire disk using the disks id as found on /dev/disk/by-id/* (it should be advisable to not mix up existing drives with one of an existing volume).

sudo zpool status tank
df -h

Check pool status. :)

Now, the interesting features:

zfs set compression=on tank
zfs set dedup=on tank

Et voilà, your space usage is highly optimized compressing and deduplicating data.

Now it’s the GlusterFS turn, download the latest version and install it:

dpkg -i glusterfs_3.2.2-1_amd64.deb
update-rc.d glusterd defaults
/etc/init.d/glusterd start

Only on one node do the peer probing for every peer:

gluster peer probe ip.add.re.ss

Only on one node, create and start the volume:

gluster volume create gtank replica 2 transport tcp ip.add.re.ss1:/tank ip.add.re.ss2:/tank
gluster volume start gtank

On every node, mount it with:

mount -t glusterfs localhost:/gtank /gtank

Now some live example:

root@debz-1:/gtank# df -h
File system           Dim. Usati Disp. Uso% Montato su
[..]
tank                   10G  1,8G  8,2G  18% /tank
localhost:/gtank       10G  1,8G  8,2G  18% /gtank

GlusterFS replica 2 on 2 servers, anything unexpected here.

root@debz-1:/gtank# ls -lh
totale 1,8G
-rw-r--r-- 1 scorp scorp  253K  1 gen  1980 Fattura Garanzia redcoon dns-323.pdf
-rw-r--r-- 1 scorp scorp   68K  1 gen  1980 fattura scontrino eeepc 900a.pdf
-rw-r--r-- 1 scorp scorp   55K  1 gen  1980 Fattura Scontrino WD HD Caviar Green Videofantasy.pdf
-rw-r--r-- 1 scorp scorp  681M 23 ago 19.29 lubuntu-11.04-desktop-i386.iso
-rw-r--r-- 1 scorp scorp 1021K  1 gen  1980 Scontrino xbox 360.pdf
-rw-r--r-- 1 root  root    98M 27 ago 12.00 test2_random.dat
-rw-r--r-- 1 root  root    98M 27 ago 11.59 test_random_copy.dat
-rw-r--r-- 1 root  root    98M 27 ago 11.55 test_random.dat
-rw-r--r-- 1 root  root   293M 27 ago 12.06 test_random-with-zero-hole.dat
-rw-r--r-- 1 root  root    98M 27 ago 11.53 test_zero.dat
-rw-r--r-- 1 root  root   674M 18 ago 16.43 ubuntu-11.04-server-amd64.iso

Here I’ve copied some PDFs, 2 ISO, and some dd generated file: test_zero.dat is a dd if=/dev/zero test_random.dat and test2_random.dat are 2 different iteration with dd if=/dev/urandom test_random_copy.dat is a cp of test_random.dat test_random-with-zero-hole.dat is the result of cat test_random.dat + test_random.dat + test2_random.dat

The following are the real disk usage:

root@debz-1:/gtank# du -sh *
259K	Fattura Garanzia redcoon dns-323.pdf
69K	fattura scontrino eeepc 900a.pdf
55K	Fattura Scontrino WD HD Caviar Green Videofantasy.pdf
675M	lubuntu-11.04-desktop-i386.iso
1,1M	Scontrino xbox 360.pdf
98M	test2_random.dat
98M	test_random_copy.dat
98M	test_random.dat
196M	test_random-with-zero-hole.dat
512	test_zero.dat
667M	ubuntu-11.04-server-amd64.iso

As you can see, compression is doing its work with ISO and test_zero.dat, but isn’t effective with PDFs and random data (do you remember that if you zip an already zip file the total size will increase?).

And what about the dedup? You should check it with:

root@debz-1:/gtank# zpool list
NAME   SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
tank  9,94G  1,60G  8,34G    16%  1.20x  ONLINE  -

so there are ~ 200 MB of available deduped space.

Other “batteries included” functions with ZFS are:

• Decrease disk I/O bottleneck using fast SSDs as caches with L2ARC
• Copy-on-write transactions: no need for fsck after hard reboot, data is always consistent on disk.
• Online Repair: ZFS store a checksum for every data block, and can notify data alternation on avery access or during a scheduled online scrub operation

Let me know if you have some good usage tips to submit!