Tuesday, December 23, 2014

Ceph Pi - Intermission and another problem solved.

Intermission...  my second son was born last week, and as these things go, my life was flipped upside down for a few days.  Its been a very happy time with everyone doing extremely well.  The holiday atmosphere (it is late December 2014) has made it so we've been able to relax into this huge change.  I've been spending time at home, but my mind has not been focused on cool projects.  Instead I've been brushing the cobwebs off my diaper-changing skills and building legos with my soon-to-be-six-year-old.

I left things in a bit of disarray last week.  I was messing around with pi1 and had left it with a half destroyed init system (failed upstart install).  Everything was rebuilt today.  I did not do much to pi2 and pi3 besides blasting away their OSDs.

As I was activating the OSDs after the rebuild I run into some trouble that I wanted to write about.  Again, I blew away the Monitor - pi1.  I re-installed everything.  I did not do much to pi2 and pi3 except destroying the OSDs.

When I typed

 ceph-deploy osd activate pi2:/mnt/sda1  

I would get the following output

 [pi2][WARNIN] DEBUG:ceph-disk:Cluster uuid is 15f7e5bd-b4aa-4b8e-ab8e-95d86eaf2f15  
 [pi2][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=fsid  
 [pi2][WARNIN] DEBUG:ceph-disk:Cluster name is ceph  
 [pi2][WARNIN] DEBUG:ceph-disk:OSD uuid is f3d2ad37-7529-4310-a41d-77780b15e0c4  
 [pi2][WARNIN] DEBUG:ceph-disk:Allocating OSD id...  
 [pi2][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd create --concise f3d2ad37-7529-4310-a41d-77780b15e0c4  
 [pi2][WARNIN] 2014-12-23 03:23:35.798919 b48ff460 0 librados: client.bootstrap-osd authentication error (1) Operation not permitted  
 [pi2][WARNIN] Error connecting to cluster: PermissionError  
 [pi2][WARNIN] ceph-disk: Error: ceph osd create failed: Command '/usr/bin/ceph' returned non-zero exit status 1:  
 [pi2][ERROR ] RuntimeError: command returned non-zero exit status: 1  
 [ceph_deploy][ERROR ] RuntimeError: Failed to execute command: ceph-disk -v activate --mark-init sysvinit --mount /mnt/sda1  

Research online brought me here.

On pi1 I run

 sudo scp /etc/ceph/ceph.client.admin.keyring ceph@pi2:  
 sudo scp /var/lib/ceph/bootstrap-osd/ceph.keyring ceph@pi2:  

Then on pi2 I run

 sudo mv ceph.client.admin.keyring /etc/ceph/  
 sudo mv ceph.keyring /var/lib/ceph/bootstrap-osd/ceph.keyring   

Then there was great success and much rejoicing when I once again run

 ceph-deploy osd activate pi2:/mnt/sda1  

Remember to do this for pi3 as well.

This is not a mandatory step.  But apparently something that can happen, and when it does, it...  took me 30 min to figure out.