How To Create Ceph Storage Cluster on Ubuntu 16.04

Ceph is one of the most exciting open source storage technologies to come out in recent years. Scalable to exabytes, and extendable to multiple datacenters, the Ceph developers have made it easy for System Administrators and Infrastructure Architects to deploy their software. This article will offer a step-by-step guide on how to create a basic Ceph stroage cluster. This has been tested on Ubuntu 16.04. Take note that unless otherwise stated, all commands are executed as root. Also note that when this document mentions "all Ceph nodes", this includes the Admin node as well.

General Setup

In our example, we will create a basic three-node Ceph cluster, each with two OSDs. We will use the hostname convention "Storage-x", where "x" is a number from 1 to 3, used to refer to specific nodes. We will use an external computer (could be your own computer or laptop) as the ceph Admin Node.

Network Setup

Each node will be on the same private network, with a gateway through which The Internet can be accessed.
The Admin node should be on the same network as well, but does not need to be available to the network all the time. Consequently, your work computer may be the Admin node, and may use a VPN to connect to the Ceph nodes' network.

Ceph uses TCP ports 6789 for Ceph Monitor nodes and ports 6800-7100 for Ceph OSDs to the public zone. For example on iptables :

sudo iptables -A INPUT -i {iface} -p tcp -s {ip-address}/{netmask} --dport 6789 -j ACCEPT

Prepare Nodes

Each storage node needs to be synchronized, so we will install ntp on them, and make sure that we can access them via SSH through the Network.

sudo apt install ntp ssh

Make sure that each Ceph node's hostname is resolvable from all Ceph nodes. On each Ceph node, edit the /etc/hosts file and add the following:

[Admin Node IP Address] admin-node
[Storage-1 IP Address] Storage-1
[Storage-2 IP Address] Storage-2
[Storage-3 IP Address] Storage-3

Substitute each node's IP address accordingly.
To test if resolution works, do the following:

ping admin-node
ping Storage-1
ping Storage-2
ping Storage-3

Make sure that the admin-node hostname resolves to the Admin Node's external network IP Address, not the loopback IP Address (

On each Ceph node (meaning the Admin node and all Storage nodes), we will add the Ceph Ubuntu package repository to apt, then update the local cache with the new repository's contents:

wget -q -O- '' | apt-key add -
echo deb{ceph-stable-release}/ $(lsb_release -sc) main | tee /etc/apt/sources.list.d/ceph.list
apt-get update

Create a cephadmin user on all Ceph nodes. This user will be used to install and administer Ceph on the whole cluster of nodes, so make sure to maximize security for the credentials to this user.

ssh-keygen -t rsa
useradd cephadmin

Set Up Password-less SSH

The Ceph install scripts and tools from the Admin node will need to be able to access all of the members of the cluster passwordlessly.
On the admin node, switch to the cephadmin user and create a SSH key:

Copy the ssh key you generated over to the three Storage nodes:

ssh-copy-id Storage-1
ssh-copy-id Storage-2
ssh-copy-id Storage-3

As the cephadmin user on your Admin node, test if the passwordless ssh into the Storage nodes now properly work:

ssh Storage-1
ssh Storage-2
ssh Storage-3

Set Up Password-less Sudo

Now that passwordless access via SSH is set up, configure passwordless sudo for the cephadmin user on all of the Ceph nodes:


You should see the following:

# This file MUST be edited with the 'visudo' command as root.
# Please consider adding local content in /etc/sudoers.d/ instead of
# directly modifying this file.
# See the man page for details on how to write a sudoers file.
Defaults env_reset
Defaults mail_badpass
Defaults secure_path="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

# Host alias specification

# User alias specification

# Cmnd alias specification

# User privilege specification

# Members of the admin group may gain root privileges
%admin ALL=(ALL) ALL

# Allow members of group sudo to execute any command

# See sudoers(5) for more information on "#include" directives:

#includedir /etc/sudoers.d

Add the following line at the bottom:


This should now allow passwordless sudo for the cephadmin user on all Ceph nodes.

Install ceph-deploy

Ceph-deploy is a tool created by the Ceph developers to facilitate quick deployments of Ceph clusters by scripting the individual steps needed to deploy a node. We will take advantage of this by installing the tool on our Admin node.

apt install ceph-deploy

On the Admin node, switch to the cephadmin user we created in Part 1 of the tutorial, switch to the home directory, and create a subdirectory that will be used to contain all of the files needed for deployment and management of our Ceph cluster:

su cephadmin
mkdir my-first-ceph-cluster
cd my-first-ceph-cluster

At this point, all of the deployment commands to executed are to be done only from within /home/cephadmin/my-first-ceph-cluster. Administration commands are to be done from within the same directory as well.

Ceph Configuration

Assign all Storage Nodes as Monitor Nodes:

ceph-deploy new Storage-1 Storage-2 Storage-3

Within your working directory, you should now see that files have been generated by ceph-deploy, including keyring files and the Ceph config file.

Add the following line to the ceph.conf file:

osd pool default size = 2

Since we only have two OSDs per Storage Node in our cluster, this will allow Ceph to be satisfied with having only a single extra copy of every data we store in it.
Add the following line to ceph.conf as well:

public network = {ip-address}/{netmask}

Where you will replace the network part with the actual values. For example, if your storage nodes are on the network, that should be the value. Do not be confused when it mentions "public network"; it only refers to a network external to the Ceph cluster. Internal replication networks are not covered by this tutorial.

Deploy Ceph

The software and config files will now be installed and copied over to the Ceph Nodes.

ceph-deploy install admin-node Storage-1 Storage-2 Storage-3

This will install all base Ceph packages to the nodes.

Install and Configure Ceph Monitor software to the Storage Nodes:

ceph-deploy mon create-initial

Ceph OSDs

While it is possible to use directories as OSDs, it is not recommended in a production setup. Assuming that the Ceph OSDs to be used on each of the Storage Nodes are /dev/sda and /dev/sdb, we have Ceph prepare the disks for use.

WARNING: The following command will destroy existing data on the specified OSDs, so care should be taken that there are no mistakes in executing the command.

ceph-deploy osd prepare Storage-1:/dev/sda
ceph-deploy osd prepare Storage-1:/dev/sdb
ceph-deploy osd prepare Storage-2:/dev/sda
ceph-deploy osd prepare Storage-2:/dev/sdb
ceph-deploy osd prepare Storage-3:/dev/sda
ceph-deploy osd prepare Storage-3:/dev/sdb

If the preceding commands run without any errors, then the OSDs are ready, and we can now activate them as a running resource in the cluster:

ceph-deploy osd activate Storage-1:/dev/sda
ceph-deploy osd activate Storage-1:/dev/sdb
ceph-deploy osd activate Storage-2:/dev/sda
ceph-deploy osd activate Storage-2:/dev/sdb
ceph-deploy osd activate Storage-3:/dev/sda
ceph-deploy osd activate Storage-3:/dev/sdb


Copy over the admin keyrings to each Ceph node so that Ceph administration on each node is possible:

ceph-deploy admin admin-node Storage-1 Storage-2 Storage-3

Check the status of all OSDs on all Storage Nodes:

ceph osd tree

Check the overall status of your Ceph cluster:

ceph health

If you get a


it means that the cluster is working properly.

If you wish to see more cluster statistics, the following command should do:

ceph status


We now have a three-node Ceph cluster up and running. With this setup, the cluster may lose one node, incur no data loss, and continue to serve requests. Highly-available monitor services are available on each storage node as well. This is a very basic production-level setup. If you wish to learn more, head over to the official Ceph documentation for additional information.

1 thought on “How To Create Ceph Storage Cluster on Ubuntu 16.04”... add one

  1. After running "ceph-deploy admin admin-node Storage-1 Storage-2 Storage-3" you need to set the permissions on the /etc/ceph/ceph.client.admin.keyring file:

    sudo chmod +r /etc/ceph/ceph.client.admin.keyring

    Or else, the commands "ceph health", "ceph
    osd tree" and "ceph status" won't work.


Leave a Comment