Sunday, December 28, 2014

Ceph Pi - Kernel Cross-Compile and also SSHFS

 Intro and Data

Well, I was saying I won't be writing this for a while, but I guess I lied.  This is going to be a tutorial specifically about using Cross-Compiling to build the kernel on Raspberry Pi and not a general purpose Cross-Compiling tutorial.

Again - we are basing this writing on this (official) guide.

Log onto your fast machine.  In my case I use an Debian box which is a tiny Intel NUC i5.

So if you have any doubts as to why compiling on the Pi will take forever, and if you have taken any offense when I called the Pi feeble, here is a graph showing the CPU utilization on the Pi for the 12 hours it took to build a kernel.

Yup.  Pegged.  For 12 hours.  So while the NUC is about $300 barebones or about 10x more expensive, it is 50x faster for this application.  I guess I am digressing but I do urge you to think well about your application.  The Raspberry Pi is amazing for some things (price / power), but heavy compute it will not do.  Here is a graph of the NUC doing the same work.

 Build the build environment

First thing first - lets make sure we have all the packages needed to actually compile and build packages.  Lets run
 apt-get update && apt-get upgrade  
 apt-get install build-essential libncurses5-dev 

Now go ahead and pull down the toolchain.
 git clone  

I am logged in as root on this box, which does break best practices.  Keep this in mind and adjust accordingly in the following steps.  I want to keep things organized so I create a directory where to put all files that have to do with this project.
 mkdir raspberry-kernel  
 mv tools/raspberry-kernel/  

The next step is to adjust our path to make it easier to work with.  Go ahead and edit your .bashrc
 vi .bashrc  

Add the following line to the bottom of the file

Next thing we need to pull the kernel sources
 cd raspberry-kernel/  
 git clone --depth=1
 cd linux 

We need to update the kernel config with the settings for the Raspberry Pi.  The space in the middle of the command is intentional.
 make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- bcmrpi_defconfig  

I had troubles with the next step.

 make -j 6 ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf-  

I kept getting an error message that looked like this:

 root@charlie:~/raspberry-kernel/linux# make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- bcmrpi_defconfig  
 # configuration written to .config  
 root@charlie:~/raspberry-kernel/linux# make -j 6 ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf-  
 make: arm-linux-gnueabihf-gcc: Command not found  
 scripts/kconfig/conf --silentoldconfig Kconfig  
 make: arm-linux-gnueabihf-gcc: Command not found  
  CHK   include/config/kernel.release  
  CHK   include/generated/uapi/linux/version.h  
  CHK   include/generated/utsrelease.h  
  HOSTCC scripts/pnmtologo  
  CC   scripts/mod/empty.o  
 /bin/sh: 1: arm-linux-gnueabihf-gcc: not found  
 scripts/ recipe for target 'scripts/mod/empty.o' failed  
 make[2]: *** [scripts/mod/empty.o] Error 127  
 scripts/ recipe for target 'scripts/mod' failed  
 make[1]: *** [scripts/mod] Error 2  
 make[1]: *** Waiting for unfinished jobs....  
 make[1]: 'include/generated/mach-types.h' is up to date.  
 Makefile:518: recipe for target 'scripts' failed  
 make: *** [scripts] Error 2  
 make: *** Waiting for unfinished jobs....  
  CC   kernel/bounds.s  
 /bin/sh: 1: arm-linux-gnueabihf-gcc: not found  
 /root/raspberry-kernel/linux/./Kbuild:35: recipe for target 'kernel/bounds.s' failed  
 make[1]: *** [kernel/bounds.s] Error 127  
 Makefile:840: recipe for target 'prepare0' failed  
 make: *** [prepare0] Error 2  

The file did exist, but every time I tried to run it even by hand I got the same error of File Not Found.  I ultimately found the answer here.  I think the reason it did not work on my machine but seemed to work for the writer of the original tutorial is that I am running Debian 64bit and they are running Ubuntu (I would guess 32 bit).  In any case.  All you have to do to get past the problem is to run

 sudo update  
 sudo upgrade  
 sudo apt-get install lsb-core  

This got me further, but I still conked-out.  This time the error was

 root@charlie:~/raspberry-kernel/linux# make -j 6 ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf-  
 arm-linux-gnueabihf-gcc: error while loading shared libraries: cannot open shared object file: No such file or directory  
  CHK   include/config/kernel.release  
  CHK   include/generated/uapi/linux/version.h  
  CC   scripts/mod/empty.o  
 arm-linux-gnueabihf-gcc: error while loading shared libraries: cannot open shared object file: No such file or directory  
 scripts/ recipe for target 'scripts/mod/empty.o' failed  
 make[2]: *** [scripts/mod/empty.o] Error 127  
 make[2]: *** Waiting for unfinished jobs....  
  CC   scripts/mod/devicetable-offsets.s  
 arm-linux-gnueabihf-gcc: error while loading shared libraries: cannot open shared object file: No such file or directory  
 scripts/ recipe for target 'scripts/mod/devicetable-offsets.s' failed  
 make[2]: *** [scripts/mod/devicetable-offsets.s] Error 127  
 scripts/ recipe for target 'scripts/mod' failed  
 make[1]: *** [scripts/mod] Error 2  
 Makefile:518: recipe for target 'scripts' failed  
 make: *** [scripts] Error 2  

The solution this time ( I found it on an Android forum that I closed before I got the link ) seemed to be

 sudo apt-get install lib32stdc++6 lib32z1 lib32z1-dev  

Build that funky kernel

Now we can actually build the kernel.  So lets run the command again.  The '-j 6' option tells the compiler to use more CPUs.  I have a 4 core that is hyper-threaded.  6 is a good number since it leaves some resources for OS and such.

 make -j 6 ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf-  

Damn!  That was fast!

One thing that both the guide missed and we skipped is that you likely want to do some configuration besides taking the defaults.  In our case we wanted the driver for the RBD block device driver module built.  To do so we must run

 make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- bcmrpi_defconfig menuconfg

So that was easy.  For completeness sake go to Device Drivers->Block Devices->Rados and make sure it checks with an <M>.  Then again run

 make -j 6 ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf-  

Install and SSHFS...  in reverse order

This next part is the thing I really don't like.  The way the guide would have you go is to remove the SD card from your Pi, mount it on your linux box, and then do the install on it.  And then move it back.  I am not impressed.  If that is your thing, go follow the guide.  I can't help but think there are way better ways to do this.  Here is a brief enumeration of the things I came up with, and they are probably stupid and some who has done more Cross-Compile work will laugh.

Edit - I thought of using TAR originally, but when I did the compile on the Pi (disclaimer: using the Pi config, not the cross compile) it seems the 'make modules' section requires a ton of compile time as well ( I don't know why...  maybe the default config enables every damn module? )  It took 6 hrs this time.  The guide suggests that there is path info to be used even while only doing make modules before the make modules_install step.
  1. NFS mount
  2. SSHFS mount (my current choice)
  3. TAR the directory and SCP it (my original choice)
  4. Remove the SD card and plug it into a PC...
We have to write files to system directories.  So we need root permissions.  We can't get away with sudo on this one, because there is no way to execute 'sudo' over sftp or sshfs.  This is something we will enable only temporarily, so we should be fine.

Make sure you are on the Pi.  First we need to give root a pssword.  Root does not have a password by default to disallow anyone using it.

 sudo passwd root  

Now we have to make sshd allow root logins, which in current versions it does not do by default.

  sudo vi /etc/sshd_config  

Find the line that reads

PermitRootLogin without-password  

Comment it out and add another line, which sets the permission to yes.  This allows us to actually log in with This is what the file will look like:

 #PermitRootLogin without-password 
 PermitRootLogin yes  

Now restart sshd

 sudo /etc/init.d/ssh restart  

Ohkay.  Now log on the Cross-compile host as root.  You may have to do similar gymnastics as to what we just did on the Pi.  Run the following commands.  When prompted for pass phrase, just press enter.  Of course substitute pi1 with the name or IP of your Raspberry.

 ssh-copy-id root@pi1  

Test everything by running

 ssh root@pi1  

You should be successfully logged into your Raspberry.  Use this opportunity to edit your sshd config

  vi /etc/sshd_config  

and switch the comments on the to PrermitRootLogin lines, so they look like this

 PermitRootLogin without-password 
 #PermitRootLogin yes  

So anyone with root on your cross compile box can log into your Pi.  But no one can SSH in directly.  We should also remove the password from root.

  passwd -d root  

We are pretty munch to best security practices on our pi.  Now its time to mount up!  Lets make a directory we can use as mount point and mount up.

 cd /root/raspberry-kernel
 mkdir remote-host
 sshfs -o idmap=user  root@pi1:/ /root/raspberry-kernel/remote-host -o Cipher=blowfish

(Edit: since this article was originally published, I also wrote this analysis of the best crypto cipher to use with the Raspberry Pi)
The idmap=user makes it so the user account we are logging in as maps to the remote host's equivalent by name instead of by numeric id.  The  Cipher=blowfish  ensures that we use a pretty fast cipher.  The Pi is often constrained by CPU when it comes to doing encryption communications (someone should look into using that GPU to accelerate them).  SCP is usually pretty slow with the default ciphers, so its a good idea to choose your encryrption wisely.  Here is a good article that includes the image I post here.  You can't have a non-encrypted tunnel apparently.  I tried a couple of other ciphers which should have worked but did not.  I will look more into this later... its an interesting follow up topic.

So while researching this I stumbled on an amazing article describing how to avoid encryption in sshfs entirely.  I have searched for ways to use the ease of sshfs without having to deal with the encryption overhead.  And here it was in its glory.  I have not tested this, but I will soon and we will get some sweet test results.

... and back to the guide

So now that we have a mount on the Pi from our cross compile host with root permissions its kind of like having the SD card plugged into the cross-compile host.   So we can take the next steps from the guide with a slight adjustment for the directory names.

 make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- INSTALL_MOD_PATH=/root/raspberry-kernel/remote-host modules  
 make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- INSTALL_MOD_PATH=/root/raspberry-kernel/remote-host modules_install  

Now backup the old kernel image and copy the new one

 cp /root/raspberry-kernel/remote-host/boot/kernel.img /root/raspberry-kernel/remote-host/boot/kernel.img.orig  
 cp arch/arm/boot/Image /root/raspberry-kernel/remote-host/boot/kernel.img  


 umount /root/raspberry-kernel/remote-host   

OK...  Well...  reboot? 

... and we are back.

We can check if everything is copacetic by running

 uname -a   

and making sure that the right kernel version shows up.

In our case we can also verify that the new driver we wanted is properly installed.

 modprobe rbd  

If there is no error, then we are doing a-OK!

That's all folks.