Saturday, December 27, 2014

Ceph Pi - Mount Up - Mounting Your Ceph File System for the first time. On Raspberry Pi. 'Cause we can.

So we have an active and healthy Ceph cluster.  Great.  But we can't do anything useful with it yet.  As we discussed earlier, the point of this exercise is to create a block device we can use as the basis of a home storage solution.  So now we need to expose our cluster as a block device...

In Ceph land, this requires the installation and setup of ceph-client on a machine connected to the network.  In a production environment, this would be the system mounting the storage.  It would certainly not be one of the machines in the cluster.  Well, we are not exactly in a production environment.... yet.

In our case we will first use pi1 as the system that is going to host ceph-client.  Since ceph is already installed, we will skip the first couple of steps of the guide.

The following steps all must be run on the box that will be ceph-client.  In our case pi1

We will need lsb_release, so lets install it
 sudo apt-get install lsb-release  

Then we go to create a block device image
 rbd create ceph-block --size 4096  

We map the image to the block device

 sudo rbd map ceph-block --pool rbd --name client.admin  

...and we fail

 modinfo: ERROR: Module rbd not found.  
 modprobe: FATAL: Module rbd not found.  
 rbd: modprobe rbd failed! (256)  

Looks like daddy needs some more modules...  This is going to take some doing.

Building a custom kernel

(Later Edit: A day after this post I posted  .  It explains exactly how to do cross compiling for the kernel and saves a lot of time.  While the directions below are valid, I would encourage you to follow the guide above to save a ton of time)

(NOTE: This is a good time to examine how to do Cross-Compilation to help the fairly feeble Raspberry Pi compile the kernel in a reasonable amount of time with the aid of a beefier machine on the network.  On its lonesome it takes 18+ hours.  I am not going to be exploring Cross-Compilation in this post since I already did it the hard way.  A follow up post may come though.  And then there is distcc)

We are going to be loosely following this guide.  But since that guide does not talk about how to add new modules, will deviate a bit.  Before we start, we need to install a couple of packages that will be needed down the road

 sudo apt-get install libncurses5-dev  

First thing we got to do is pull down the sources.  If you are familiar with Debian you may think that something like sudo apt-get install linux-headers would work.  Not on Raspbian.  Here it is a place holder package or something because (I gather) of the proprietary bunch of  code that Rasberrys require.  In any case.  Here is what we got to do.

In your home directory run

 git clone --depth=1  

After you are done pulling sources there will be a directory called 'linux' in your home directory.  Next we have to make sure we have all dependencies

 sudo apt-get install bc  

Time to configure your kernel.  Luckily Raspberrys come with this really cool script that sets up all the defaults for us, so we don't have to spend hours hunting and pecking for drivers.  All the basics are taken care of and we just have to build the new things.

 cd linux  
 make bcmrpi_defconfig  

Here we diverge from the guide

 make menuconfig  

In the menu that appears, scroll and select

 Device Drivers -> Block Devices -> Rados Block Device (RBD)   

Make sure you are not selecting DRBD instead.  Once on top of RBD, press space until M appears in the 'check box' in front of it.  Then feel free to exit the config and make sure you save your changes.

Its probably a good idea to run this next command in a screen session.  It takes a long time.  Not 'lets make a tea' long time.  An 'overnight' long time.  Unless you have a cross-compile or distcc set up.  So what I heartily advise is to install screen and run make in a screen session.  That way if your connection drops for some reason you don't have to start from scratch.  If you are running Debian wheezy / testing (aka jessie) by following my previous guide (and having already installed ncruses), you should be able to do:

 sudo apt-get install screen  



...and about 12hrs later we are on the other side. One more 6 hrs biggie!

 make modules
...and breathe!  The other commands are somewhat shorter

 sudo make modules_install  
 sudo cp /boot/kernel.img /boot/kernel.img.bkup   
 sudo cp arch/arm/boot/Image /boot/kernel.img  

Reboot, yo!

Try to ssh back in!
are you alive?  I hope so!  Time to step it up!

The rest of this guide is too easy (famous last words).

Now that we have a kernel built...  Lets mount up!

We can now...  a day or so later...  re-run the command that failed earlier.  Lets get that block device mapped
 sudo rbd map ceph-block --pool rbd --name client.admin    

And now lets put a file system on it
 sudo mkfs.xfs /dev/rbd/rbd/ceph-block  

And now lets mount it!
 sudo mkdir /mnt/ceph-block-device  
 sudo mount /dev/rbd/rbd/foo /mnt/ceph-block-device  
 cd /mnt/ceph-block-device  

...and we are done!  We have a distributed storage array running on Raspberry Pis!  Could it be true???  Lets check!

 ceph@pi1 /mnt/ceph-block-device $ df -h  
 Filesystem   Size Used Avail Use% Mounted on  
 /dev/root    14G 3.1G 9.6G 25% /  
 devtmpfs    231M   0 231M  0% /dev  
 tmpfs      235M   0 235M  0% /dev/shm  
 tmpfs      235M 4.5M 231M  2% /run  
 tmpfs      5.0M 4.0K 5.0M  1% /run/lock  
 tmpfs      235M   0 235M  0% /sys/fs/cgroup  
 /dev/mmcblk0p5  60M  16M  44M 27% /boot  
 tmpfs      47M   0  47M  0% /run/user/1001  
 /dev/sda1    932G  33M 932G  1% /mnt/sda1  
 /dev/rbd1    4.0G  33M 4.0G  1% /mnt/ceph-block-device  
 ceph@pi1 /mnt/ceph-block-device $   

So how does this setup perform?  We will find shortly!