And yet... I failed. A lot.
It really seems to have to do with the kernel. The oDroid comes with a 3.0 kernel. I had to compile the RBD module. And it was hell. I failed to get the on-board compile to work. When I failed - there was no way to get any feedback. I would have it plugged into a screen and get no feedback whatsoever. I finally got it working by doing a cross compile. Technically there is a a 3.8 branch as well, but I did not manage to get it working (well, I really did not get to it in the end, so just keep reading).
So then I tried set it up as a mon / ceph-client. And I failed. And failed. And failed. In the end, I found this article which describes the features included in various versions of the kernel. It turns out my 3.0 was just too old. And even if I get 3.8 working, its likely going to be trouble as well, since it was missing a ton of features. I tried every work around I could, but most posts really talked about 3.9 as being the earliest viable version. And the oDroid does not seem to support it... at least with Ubuntu.
So I gave, up and set up my Intel NUC mini PC which is running x64 with Debian Jessie.
I have not given up on the oDroid yet, but it will be a couple of weeks before I forget my current level of frustration and give it another whirl.
This may be a blessing in disguise because I now have a gig port.