How to Setup Red Hat Ceph Storage on CentOS 7

Ceph is an open-source software platform that stores data on a single distributed computer cluster. When you are planning to build a cloud, then on top of the requirements you have to decide on how to implement your storage. Open Source CEPH is one of HAT mature technology based on object-store system, called RADOS, with a set of gateway APIs that present the data in block, file, and object modes. As a result of its open-source nature, this portable storage platform may be installed and used in public or private clouds. The topology of a Ceph cluster is designed around replication and information distribution, which are intrinsic and provide data integrity. It is designed to be fault-tolerant, and can run on commodity hardware, but can also be run on a number of more advanced systems with the right setup.

Ceph can be installed on any Linux distribution but it requires the recent kernel and other up-to-date libraries in order to be properly executed. But, here in this tutorial we will be using CentOS-7.0 with minimal installation packages on it.

System Resources


OS: CentOS Linux 7 (Core)
DISK: 20


OS: CentOS Linux 7 (Core)
DISK: 20

Pre-Installation Setup

There are few steps that we need to perform on each of our node before the CEPH storage setup. So first thing is to make sure that each node is configured with its networking setup with FQDN that is reachable to other nodes.

Configure Hosts

To setup the hosts entry on each node let's open the default hosts configuration file as shown below.

# vi /etc/hosts ceph-storage ceph-node

Install VMware Tools
While working on the VMware virtual environment, it's recommended that you have installed its open VM tools. You can install using below command.

#yum install -y open-vm-tools

Firewall Setup

If you are working on a restrictive environment where your local firewall in enabled then make sure that the number of following ports are allowed from in your CEPH storge admin node and client nodes.

You must open ports 80, 2003, and 4505-4506 on your Admin Calamari node and port 80 to CEPH admin or Calamari node for inbound so that clients in your network can access the Calamari web user interface.

You can start and enable firewall in centos 7 with the given below command.

#systemctl start firewalld
#systemctl enable firewalld

To allow the mentioned ports in the Admin Calamari node run the following commands.

#firewall-cmd --zone=public --add-port=80/tcp --permanent
#firewall-cmd --zone=public --add-port=2003/tcp --permanent
#firewall-cmd --zone=public --add-port=4505-4506/tcp --permanent
#firewall-cmd --reload

On the CEPH Monitor nodes you have to allow the following ports in the firewall.

#firewall-cmd --zone=public --add-port=6789/tcp --permanent

Then allow the following list of default ports for talking to clients and monitors and for sending data to other OSDs.

#firewall-cmd --zone=public --add-port=6800-7300/tcp --permanent

It quite fair that you should disable firewall and SELinux settings if you are working in a non-production environment, so we are going to disable the firewall and SELinux in our test environment.

#systemctl stop firewalld
#systemctl disable firewalld

System Update

Now update your system and then give it a reboot to implement the required changes.

#yum update
#shutdown -r 0

Setup CEPH User

Now we will create a separate sudo user that will be used for installing the ceph-deploy utility on each node and allow that user to have passwordless access on each node because it needs to install software and configuration files without prompting for passwords on CEPH nodes.

To create new user with its separate home directory run the below command on the ceph-storage host.

[root@ceph-storage ~]# useradd -d /home/ceph -m ceph
[root@ceph-storage ~]# passwd ceph

Each user created on the nodes must have sudo rights, you can assign the sudo rights to the user using running the following command as shown.

[root@ceph-storage ~]# echo "ceph ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/ceph
ceph ALL = (root) NOPASSWD:ALL

[root@ceph-storage ~]# sudo chmod 0440 /etc/sudoers.d/ceph

Setup SSH-Key

Now we will generate SSH keys on the admin ceph node and then copy that key to each Ceph cluster nodes.

Let's run the following command on the ceph-node to copy its ssh key on the ceph-storage.

[root@ceph-node ~]# ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/
The key fingerprint is:
5b:*:*:*:*:*:*:*:*:*:c9 root@ceph-node
The key's randomart image is:
+--[ RSA 2048]----+
[root@ceph-node ~]# ssh-copy-id ceph@ceph-storage
SSH key

Configure PID Count

To configure the PID count value, we will make use of the following commands to check the default kernel value. By default it's a small maximum number of threads that is '32768'.

So will configure this value to a higher number of threads by editing the system conf file as shown in the image.

Change PID Value

Setup Your Administration Node Server

With all the networking setup and verified, now we will install ceph-deploy using the user ceph. So, check the hosts entry by opening its file.

#vim /etc/hosts

Now to add its repository run the below command.

#rpm -Uhv
Adding EPEL

OR create a new file and update the CEPH repository parameters but do not forget to mention your current release and distribution.

[root@ceph-storage ~]# vi /etc/yum.repos.d/ceph.repo
name=Ceph noarch packages

After this update your system and install the ceph deploy package.

Installing CEPH-Deploy Package

To update the system with the latest ceph repository and other packages, we will run the following command along with ceph-deploy installation command.

#yum update -y && yum install ceph-deploy -y


Setup the cluster

Create a new directory and move into it on the admin ceph-node to collect all output files and logs by using the following commands.

#mkdir ~/ceph-cluster
#cd ~/ceph-cluster
#ceph-deploy new storage
setup ceph cluster

Upon successful execution of above command you can see it creating its configuration files.
Now to configure the default configuration file of CEPH, open it using any editor and place the following two lines under its global parameters that reflects your public network.

#vim ceph.conf
osd pool default size = 1
public network =

Installing CEPH

We are now going to install CEPH on each of the node associated with our CEPH cluster. To do so we make use of the following command to install CEPH on our both nodes that is ceph-storage and ceph-node as shown below.

#ceph-deploy install ceph-node ceph-storage
installing ceph

This will takes some time while processing all its required repositories and installing the required packages.

Once the ceph installation process is complete on both nodes we will proceed to create monitor and gather keys by running the following command on the same node.

#ceph-deploy mon create-initial
CEPH Initial Monitor

Setup OSDs and OSD Daemons

Now we will setup disk storage, to do so first run the below command to List all of your usable disks by using the below command.

#ceph-deploy disk list ceph-storage

In result will get the list of your disks used on the storage nodes that you will use for creating the OSD. Let's run the following command that consists of your disks names as shown below.

#ceph-deploy disk zap storage:sda
#ceph-deploy disk zap storage:sdb

Now to finalize the OSD setup let's run the below commands to setup the journaling disk along with data disk.

#ceph-deploy osd prepare storage:sdb:/dev/sda
#ceph-deploy osd activate storage:/dev/sdb1:/dev/sda1

You will have to repeat the same command on all the nodes while it will clean everything present on the disk. Afterwards to have a functioning cluster, we need to copy the different keys and configuration files from the admin ceph-node to all the associated nodes by using the following command.

#ceph-deploy admin ceph-node ceph-storage

Testing CEPH

We have almost completed the CEPH cluster setup, let's run the below command to check the status of the running ceph by using the below command on the admin ceph-node.

#ceph status
#ceph health

So, if you did not get any error message at ceph status, that means you have successfully setup your ceph storage cluster on CentOS 7.


In this detailed article, we learned about the CEPH storage clustering setup using the two virtual Machines with CentOS 7 OS installed on them that can be used as a backup or as your local storage that can be used to process other virtual machines by creating its pools. We hope you have got this article helpful. Do share your experiences when you try this at your end.

2 Comments... add one

  1. Thanks for the guide.
    Can you provide some guidance on the actual hardware setup.
    Any Vlan of the nodes in the cluster, etc, etc
    I have access to two HP DL380 and two fully populated Promise VTrak E310s.



Leave a Comment