Friday, January 2, 2015

Ceph Pi - Initial Performance Measurements

Tests to be run

We are going to run 3 tests.

  • Local copy HDD to SD (control of maximum throughput)
  • Local copy SD to Ceph mount
  • SCP
  • SAMBA (network share)
We are going to look at 
  • Network Throughput
  • Throughput by protocol
  • CPU usage overall
  • CPU per key processes

Setup

This is a review of what the current setup looks like.  We are going to be using a 3.5 GB gzip file of a SD card backup for our testing purposes.

It would be also interesting to try it with a directory of small files and see what that will do.

Local Copy HDD to SD

The first test we are doing is to see what the maximum throughput of the Raspberry Pi is when copying from one on-board device to another.  We will also try to see if we can pin point the limiting factor - Disk throughput, IOPS or CPU.

The first hump in the graph is copy from the SD card (mmcblk0p6) to USB attached HDD (sda1).  The second is the reverse - HDD to SD card.

As we can see CPU is certainly the limiting factor, though SD card throughput is close second.  

Here is another graph showing the actual throughput.


The good news is that the SD card is plenty fast to read at 100Mbs.  

Local Copy SD to Ceph

We have the file loaded on the SD of Pi1, which is also our ceph-client.  Lets go ahead and copy it over from the SD over to the the Ceph cluster mounted partition.


We are getting decent throughput, certainly good enough to stream video at about 30 megabits / sec.  Lets go ahead and check out how the OSDs did.  Unfortunately I seem to have a bit of a problem with pi3, so I only have data from pi2.  In this setup I expect the two OSDs to be fairly close, though.



The OSD has some more breathing room.  It is interesting that there is a lot of traffic in the reverse direction comping out of the OSD!  The traffic is headed to the other OSD, though.  Here is a composite graph... a bit complex for my taste...

Local copy from Ceph to SD

Now lets take a look in the opposite direction - downloading the file from the ceph cluster to the local storage.  Here is the utilization on the Monitor / Client node.

Here is the utilization on the OSD

Notice that there is almost no chatter.  CPU is low as well.  So this makes it pretty clear that the client node is CPU limiting us.  

Network SCP to Ceph

Given that we are already being CPU limited, adding the extra overhead of encryption is going to make things poor indeed.  I am not even going to bother testing this, unless someone in the comments specifically requests it.

Network SAMBA ( network share to ceph ).

Uploading data to SAMBA we see

And then downloading it we get
Interestingly upload is a little faster than downloading.

At 25 Mbs we are doing well, especially considering the 35 Mbs optimum base case we got from the local copy, and the fact that we are limited to 100Mbs. It is workable for most home applications (streaming movies), but we will have to optimize a bit before we can consider this anything like production ready ( uploading or downloading will take ages ) even for a home storage solution.

Conclusion

While this is a somewhat workable setup with enough bandwidth for most everyday uses, it is pretty obvious that we can get a huge improvement by moving the client to a more powerful box.  OSDs seem to be keeping up well.  I am looking forward to expanding the test cases.

Right now we are doing pretty much at the same level as if we used a low-end SDHC card in our PC as storage.