A Guide to Install and Use ZFS on CentOS 7

March 29, 2016 | By
| 5 Replies More

ZFS, short form of Zettabyte Filesystem is an advanced and highly scalable filesystem. It was originally developed by Sun Microsystems and is now part of the OpenZFS project. With so many filesystems available on Linux, it is quite natural to ask what is special about ZFS.  Unlike other filesystems, it is not just a filesystem but a logical volume manager as well.  Some of the features of ZFS that make it popular are:

  • Data Integrity -  data consistency and integrity are ensured through copy-on-write and checksum techniques
  • Pooling of storage space - available storage drives can be put together into a single pool called zpool
  • Software RAID - Setting up a raidz array is as simple as issuing a single command.
  • Inbuilt volume manager - ZFS acts as a volume manager as well.
  • Snaphots, clones, compression - these are some of the advanced features that ZFS provides.

ZFS  is a 128-bit filesystem and has the capacity to store 256 zetta bytes!  In this guide, we will be learning how to install, setup and also to use some important ZFS commands on a CentOS 7 server.

NOTE: The installation part is specific to CentOS server while the commands are common on any Linux system

Terminology

Before we move on, let us understand some of the terminologies that are commonly used in ZFS.

Pool

Logical grouping of storage drives. It is the basic building block of ZFS and it is from here that storage space gets allocated for datasets.

Datasets

The components of ZFS filesystem namely filesystem, clones, snapshots and volumes are referred to as datasets.

Mirror

A virtual device storing identical data copies on two or more disks. In situations where one disk fails, same data is available on other disks of that mirror.

Resilvering

Process of copying data from one disk to another in the event of restoring a device.

Scrub

Scrub is used for consistency check in ZFS like how fsck is used in other filesystems

Installing ZFS

In order to install ZFS on CentOS, we need to first setup the EPEL  repository for supporting packages and then the ZFS repository to install the required ZFS packages.

Note: Please prefix sudo to all the commands if you are not the root user. 

yum localinstall --nogpgcheck http://epel.mirror.net.in/epel/7/x86_64/e/epel-release-7-5.noarch.rpm

yum localinstall --nogpgcheck http://archive.zfsonlinux.org/epel/zfs-release.el7.noarch.rpm

Now install the kernel development and zfs packages. Kernel development packages are needed as ZFS is built as a module and inserted into the kernel.

yum install kernel-devel zfs

Verify if the zfs module is inserted into the kernel using 'lsmod' command and if not, insert it manually using 'modprobe' command.

[root@li1467-130 ~]# lsmod |grep zfs

[root@li1467-130 ~]# modprobe zfs

[root@li1467-130 ~]# lsmod |grep zfs
zfs 2790271 0
zunicode 331170 1 zfs
zavl 15236 1 zfs
zcommon 55411 1 zfs
znvpair 89086 2 zfs,zcommon
spl 92029 3 zfs,zcommon,znvpair

Let us check if we are able to use the zfs commands:

[root@li1467-130 ~]# zfs list
no datasets available

Administration

ZFS has two main utilities, zpool and zfs.  While zpool deals with creation and maintenance of pools using disks zfs utility is responsible for creation and maintenance of datasets.

zpool utility

Creating and destroying pools

First verify the disks available for you to create a storage pool.

[root@li1467-130 ~]# ls -l /dev/sd*
brw-rw---- 1 root disk 8,  0  Mar 16 08:12 /dev/sda
brw-rw---- 1 root disk 8, 16 Mar 16 08:12 /dev/sdb
brw-rw---- 1 root disk 8, 32 Mar 16 08:12 /dev/sdc
brw-rw---- 1 root disk 8, 48 Mar 16 08:12 /dev/sdd
brw-rw---- 1 root disk 8, 64 Mar 16 08:12 /dev/sde
brw-rw---- 1 root disk 8, 80 Mar 16 08:12 /dev/sdf

Create a pool from a set of drives.

zpool create <option> <pool name. <drive 1> <drive 2> .... <drive n>

[root@li1467-130 ~]# zpool create -f zfspool sdc sdd sde sdf

'zpool status' command displays the status of the available pools

[root@li1467-130 ~]# zpool status
pool: zfspool
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
zfspool ONLINE 0 0 0
sdc ONLINE 0 0 0
sdd ONLINE 0 0 0
sde ONLINE 0 0 0
sdf ONLINE 0 0 0

errors: No known data errors

Verify if the pool creation was successful.

[root@li1467-130 ~]# df -h
Filesystem    Size   Used      Avail  Use%   Mounted on
/dev/sda      19G    1.4G        17G      8%      /
devtmpfs    488M        0      488M      0%     /dev
tmpfs          497M        0      497M      0%    /dev/shm
tmpfs          497M    50M     447M     11%   /run
tmpfs          497M         0     497M      0%   /sys/fs/cgroup
tmpfs          100M         0     100M      0%   /run/user/0
zfspool         3.7G         0       3.7G      0%  /zfspool

As you can see, zpool has created a pool by name 'zfspool' of size 3.7 GB and has also mounted it in /zfspool.

To destroy a pool, use the 'zpool destroy' command

zpool destroy <pool name>

[root@li1467-130 ~]# zpool destroy zfspool
[root@li1467-130 ~]# zpool status
no pools available

Let us now try creating a simple mirror pool.

zpool create <option> <pool name> mirror <drive 1> <drive 2>... <drive n>

We can also create multiple mirrors at the same time by repeating the mirror keyword followed by the drives.

[root@li1467-130 ~]# zpool create -f mpool mirror sdc sdd mirror sde sdf
[root@li1467-130 ~]# zpool status
pool: mpool
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
mpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
sdc ONLINE 0 0 0
sdd ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
sde ONLINE 0 0 0
sdf ONLINE 0 0 0

errors: No known data errors

In the above example, we have created mirror pools each with two disks.

Similarly, we can create a raidz pool.

[root@li1467-130 ~]# zpool create -f rpool raidz sdc sdd sde sdf
[root@li1467-130 ~]# zpool status
pool: rpool
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
sdc ONLINE 0 0 0
sdd ONLINE 0 0 0
sde ONLINE 0 0 0
sdf ONLINE 0 0 0

errors: No known data errors

Managing devices in ZFS pools

Once a pool is created, it is possible to add or remove hot spares and cache  devices from the pool, attach or detach devices from mirrored pools and replace devices. But non-redundant and raidz devices cannot be removed from a pool.  We will see how to perform some of these operations in this section.

I'm first creating a pool called 'testpool' consisting of two devices, sdc and sdd.  Another device sde will then be added to this.

[root@li1467-130 ~]# zpool create -f testpool sdc sdd

[root@li1467-130 ~]# zpool add testpool sde
[root@li1467-130 ~]# zpool status
pool: testpool
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
testpool ONLINE 0 0 0
sdc ONLINE 0 0 0
sdd ONLINE 0 0 0
sde ONLINE 0 0 0

errors: No known data errors

As mentioned earlier, I cannot remove this newly added device as it is not a redundant or raidz pool.

[root@li1467-130 ~]# zpool remove testpool sde
cannot remove sde: only inactive hot spares, cache, top-level, or log devices can be removed

But I can add a spare disk to this pool and remove it.

[root@li1467-130 ~]# zpool add testpool spare sdf
[root@li1467-130 ~]# zpool status
pool: testpool
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
testpool ONLINE 0 0 0
sdc ONLINE 0 0 0
sdd ONLINE 0 0 0
sde ONLINE 0 0 0
spares
sdf AVAIL

errors: No known data errors
[root@li1467-130 ~]# zpool remove testpool sdf
[root@li1467-130 ~]# zpool status
pool: testpool
state: ONLINE
scan: none requested
config:

NAME        STATE       READ  WRITE   CKSUM
testpool    ONLINE       0          0               0
sdc            ONLINE       0           0               0
sdd            ONLINE      0            0               0
sde            ONLINE      0            0               0

errors: No known data errors

Similarly, we can use attach command to attach disks to a mirrored or non-mirrored pool and detach command to detach disks from a mirrored pool.

zpool attach <options> <pool name> <device> <new device>

zpool detach <pool name> <device>

When a device fails or gets corrupted, we can replace it using the 'replace' command.

zpool replace <options> <pool name> <device> <new device>

We will test this by forcefully corrupting a device in a mirrored configuration.

[root@li1467-130 ~]# zpool create -f testpool mirror sdd sde

This creates a mirror pool consisting of disks sdd and sde. Now, let us deliberately corrupt sdd drive by writing zeroes into it.

[root@li1467-130 ~]# dd if=/dev/zero of=/dev/sdd
dd: writing to ‘/dev/sdd’: No space left on device
2048001+0 records in
2048000+0 records out
1048576000 bytes (1.0 GB) copied, 22.4804 s, 46.6 MB/s

We will use the 'scrub' command to detect this corruption.

[root@li1467-130 ~]# zpool scrub testpool
[root@li1467-130 ~]# zpool status
pool: testpool
state: ONLINE
status: One or more devices could not be used because the label is missing or
invalid. Sufficient replicas exist for the pool to continue
functioning in a degraded state.
action: Replace the device using 'zpool replace'.
see: http://zfsonlinux.org/msg/ZFS-8000-4J
scan: scrub repaired 0 in 0h0m with 0 errors on Fri Mar 18 09:59:40 2016
config:

NAME STATE READ WRITE CKSUM
testpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
sdd UNAVAIL 0 0 0 corrupted data
sde ONLINE 0 0 0

errors: No known data errors

We will now replace sdd with sdc.

[root@li1467-130 ~]# zpool replace testpool sdd sdc; zpool status
pool: testpool
state: ONLINE
scan: resilvered 83.5K in 0h0m with 0 errors on Fri Mar 18 10:05:17 2016
config:

NAME STATE READ WRITE CKSUM
testpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
replacing-0 UNAVAIL 0 0 0
sdd UNAVAIL 0 0 0 corrupted data
sdc ONLINE 0 0 0
sde ONLINE 0 0 0

errors: No known data errors

[root@li1467-130 ~]# zpool status
pool: testpool
state: ONLINE
scan: resilvered 74.5K in 0h0m with 0 errors on Fri Mar 18 10:00:36 2016
config:

NAME STATE READ WRITE CKSUM
testpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
sdc ONLINE 0 0 0
sde ONLINE 0 0 0

errors: No known data errors

Migration of pools

We can migrate storage pools between different hosts using export and import commands. For this, the disks used in the pool should be available from both the systems.

[root@li1467-130 ~]# zpool export testpool
[root@li1467-130 ~]# zpool status
no pools available

The command 'zpool import' lists all the pools that are available for importing. Execute this command from the system where you want to import the pool.

[root@li1467-131 ~]# zpool import
pool: testpool
id: 3823664125009563520
state: ONLINE
action: The pool can be imported using its name or numeric identifier.
config:

testpool ONLINE
sdc ONLINE
sdd ONLINE
sde ONLINE

Now import the required pool

[root@li1467-131 ~]# zpool import testpool
[root@li1467-131 ~]# zpool status
pool: testpool
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
testpool ONLINE 0 0 0
sdc ONLINE 0 0 0
sdd ONLINE 0 0 0
sde ONLINE 0 0 0

errors: No known data errors

iostat

One can verify the io statistics of the pool devices using the iostat command.

[root@li1467-130 ~]# zpool iostat -v testpool
capacity          operations                        bandwidth
pool          alloc      free            read     write             read   write
----------    -----     -----            -----     -----                -----   -----
testpool    1.80M  2.86G        22            27               470K  417K
sdc             598K   975M           8              9               200K  139K
sdd             636K  975M            7              9                135K  139K
sde             610K   975M           6              9                 135K 139K
----------   -----     -----           -----          -----               -----  -----

zfs utility

We will now move on to the zfs utility.  Here we will take a look at how to create, destroy datasets, filesystem compression, quotas and snapshots.

Creating and destroying filesystem

ZFS filesystem can be created using the zfs create command

zfs create <filesystem>

 

[root@li1467-130 ~]# zfs create testpool/students
[root@li1467-130 ~]# zfs create testpool/professors
[root@li1467-130 ~]# df -h
Filesystem                    Size             Used          Avail          Use%          Mounted on
/dev/sda                       19G              1.4G          17G             8%                     /
devtmpfs                   488M                  0      488M             0%                    /dev
tmpfs                          497M                  0       497M            0%                   /dev/shm
tmpfs                          497M            50M       447M           11%                  /run
tmpfs                          497M                 0        497M            0%                /sys/fs/cgroup
testpool                       2.8G                  0         2.8G            0%               /testpool
tmpfs                          100M                  0        100M            0%             /run/user/0
testpool/students     2.8G                   0         2.8G             0%            /testpool/students
testpool/professors  2.8G                   0         2.8G             0%           /testpool/professors

From the above output, observe that though there is no mount point given at the time of filesystem creation, mountpoint is created using the same path relationship as that of the pool.

zfs create allows  using -o with it using which we can specify options like mountpoint, compression, quota, exec etc.

One can list the available filesystem using zfs list:

[root@li1467-130 ~]# zfs list
NAME                           USED     AVAIL     REFER    MOUNTPOINT
testpool                         100M       2.67G       19K         /testpool
testpool/professors        31K     1024M   20.5K        /testpool/professors
testpool/students        1.57M     98.4M   1.57M      /testpool/students

We can destroy a filesystem using the destroy option

zfs destroy <filesystem>

Compression

We will now understand how compression works in ZFS. Before we start using compression, we need to enable it using 'set compression'

zfs set <compression=value> <filesystem|volume|snapshot>

Once this is done, compression and decompression happens on the filesystem on the fly transparently.

In our example, I will be enabling compression on the students directory using lz4 compression algorithm.

[root@li1467-130 ~]# zfs set compression=lz4 testpool/students

I will now copy a file of size 15M into this filesystem and check the size once it is copied.

[root@li1467-130 /]# cd /var/log
[root@li1467-130 log]# du -h secure
15M secure

[root@li1467-130 ~]# cp /var/log/secure /testpool/students/

[root@li1467-130 students]# df -h .
Filesystem               Size     Used   Avail    Use%      Mounted on
testpool/students   100M   1.7M   99M        2%      /testpool/students

Notice that the size used in the filesystem is only 1.7M while the file size was 15M. We can check the compression ratio as well..

[root@li1467-130 ~]# zfs get compressratio testpool
NAME      PROPERTY         VALUE            SOURCE
testpool    compressratio     9.03x                     -

 Quotas and reservation

Let me explain quotas with a real life example. Suppose we have a requirement in a university to limit the disk space used by the filesystem for professors and students. Let us assume that we need to allocate 100MB for students and 1GB for professors. We can make use of 'quotas' in ZFS to fulfill this requirement. Quotas ensure that the amount of disk space used by a filesystem doesn't exceed the limits set. Reservation helps in actually allocating and guaranteeing that the required amount of disk space is available for the filesystem.

zfs set quota=<value> <filesystem|volume|snapshot>

zfs set reservation=<value> <filesystem|volume|snapshot>

[root@li1467-130 ~]# zfs set quota=100M testpool/students
[root@li1467-130 ~]# zfs set reservation=100M testpool/students
[root@li1467-130 ~]# zfs list
NAME                          USED      AVAIL    REFER    MOUNTPOINT
testpool                        100M       2.67G       19K        /testpool
testpool/professors      19K       2.67G        19K       /testpool/professors
testpool/students      1.57M       98.4M    1.57M    /testpool/students

[root@li1467-130 ~]# zfs set quota=1G testpool/professors
[root@li1467-130 ~]# zfs list
NAME                           USED     AVAIL    REFER    MOUNTPOINT
testpool                         100M     2.67G       19K          /testpool
testpool/professors       19K    1024M       19K         /testpool/professors
testpool/students       1.57M    98.4M    1.57M       /testpool/students

In the above example, we have allocated 100MB for students and 1GB for professors. Observe the 'AVAIL' column in 'zfs list'.  Initially they had the size of 2.67GB each and after setting the quota, the values have changed accordingly.

Snapshots

Snapshots are read-only copy of the ZFS filesystem at a given point in time. They do not consume any extra space in zfs pool. We can either roll back to the same state at a later stage or extract only a single or a set of files as per the user requirement.

I will now create some directories and a file under '/testpool/professors' from our previous example and then take a snapshot of this filesystem.

[root@li1467-130 ~]# cd /testpool/professors/

[root@li1467-130 professors]# mkdir maths physics chemistry

[root@li1467-130 professors]# cat > qpaper.txt
Question paper for the year 2016-17
[root@li1467-130 professors]# ls -la
total 4
drwxr-xr-x  5  root root    6   Mar 19 10:34 .
drwxr-xr-x  4  root root    4   Mar 19 09:59 ..
drwxr-xr-x  2  root root    2   Mar 19 10:33 chemistry
drwxr-xr-x  2  root root    2   Mar 19 10:32 maths
drwxr-xr-x  2  root root    2   Mar 19 10:32 physics
-rw-r--r--     1  root root  36   Mar 19 10:35 qpaper.txt

To take a snapshot, use the following syntax:

zfs snapshot <filesystem|volume@<snap>>

[root@li1467-130 professors]# zfs snapshot testpool/professors@03-2016
[root@li1467-130 professors]# zfs list -t snapshot
NAME                                             USED         AVAIL     REFER     MOUNTPOINT
testpool/professors@03-2016       0                -                20.5K          -

I'll now delete the file that was created and extract it from the snapshots

[root@li1467-130 professors]# rm -rf qpaper.txt
[root@li1467-130 professors]# ls
chemistry maths physics
[root@li1467-130 professors]# cd .zfs
[root@li1467-130 .zfs]# cd snapshot/03-2016/
[root@li1467-130 03-2016]# ls
chemistry maths physics qpaper.txt

[root@li1467-130 03-2016]# cp -a qpaper.txt /testpool/professors/
[root@li1467-130 03-2016]# cd /testpool/professors/
[root@li1467-130 professors]# ls
chemistry maths physics qpaper.txt

The deleted file is back in its place.

We can list all the available snapshots using zfs list:

[root@li1467-130 ~]# zfs list -t snapshot
NAME                                             USED     AVAIL    REFER    MOUNTPOINT
testpool/professors@03-2016    10.5K       -              20.5K       -

Finally, let's destroy the snapshot using the zfs destroy command:

zfs destroy <filesystem|volume@<snap>>

[root@li1467-130 ~]# zfs destroy testpool/professors@03-2016

[root@li1467-130 ~]# zfs list -t snapshot
no datasets available

Conclusion

In this article, you have learnt how to install ZFS on CentOS 7 and use some basic and important commands from zpool and zfs utilities. This is not a comprehensive list. ZFS has much more capabilities and you can explore them further from its official page.

Filed Under : LINUX HOWTO, OPEN SOURCE TOOLS

Tagged With :

Free Linux Ebook to Download

Comments (5)

Trackback URL | Comments RSS Feed

  1. Floyd Briscoe says:

    Excellent tutorial, very precise methodology and testing. Thank you.

  2. Joe says:

    Excellent intro. Thanks!

    I thought after "export" the host can still use the exported pool but apparently it's no longer available. If I setup a JBOD using zfs and want to share it with many hosts, should I use NFS or some better way?

  3. huang says:

    It's very nice!!! Thank you!!!

Leave a Reply

All comments are subject to moderation.