Table Of Contents

Previous topic

EOS Docker Installation

Next topic

Scientific Services Installation: EOS, CERNBox, SWAN and CVMFS

This Page

EOS Kubernetes Installation

../_images/kubernetes-logo.png

A Kubernetes installation provides a fully functional distributed EOS instances on top of Kubernetes clusters, keeping the advantages of a straightforward set up and a short deployment time. The EOS team provides docker images with all the necessary components installed and ready to use.

Preparation

Note

Make sure you have kubectl installed and access rights to a Kubernetes cluster on your system. To start playing around by using a virtual cluster in your local machine, you can use Minikube.

Run EOS in Kubernetes

Checkout the eos-on-k8s project:

git clone https://gitlab.cern.ch/faluchet/eos-on-k8s.git
cd eos-on-k8s

To start a small instance with 8 storage servers (FST), 1 namespace server (MGM), 1 messaging borker (MQ), 1 client and 1 Kerberos KDC Kubernetes Pod ready to use, all you have to do is to use the create-all.sh script. It is recommended to provide a <namespace> argument, via the -n flag, representing the virtual cluster name the resources will live in. Each EOS entities (“roles”) will reside on the same network knowing about each other and are configured to be working out-of-the-box. We refer the courious reader to the official documentation to deepen his knowledge about Kubernetes concepts.

./create-all -n <your_namespace>

Wait for the resources creation and EOS setup, and you are ready to go. You can check the cluster state in any moment, i.e. with:

kubectl get nodes # get the cluster node list
kubectl get all # get all the Kubernetes resources residing on the "default" namespace.
kubectl get all -n <your_namespace>

Note

From now on we will use the namespace “tutorial”.

To connect to EOS using the eos shell CLI running in the MGM container you can do:

kubectl exec -n tutorial -it <mgm_pod_name> -- eos

It is a bit verbose though very easy getting the name of a Pod of your interest, through the use of easy-to-remeber labels:

kubectl get pods -n tutorial --no-headers -o custom-columns=":metadata.name" -l app=eos-mgm

So, all together:

kubectl exec -n tutorial -it $(kubectl get pods -n tutorial --no-headers -o custom-columns=":metadata.name" -l app=eos-mgm) -- eos
EOS Console [root://localhost] |/> whoami
whoami
Virtual Identity: uid=0 (2,99,3,0) gid=0 (99,4,0) [authz:sss] sudo* host=localhost
EOS Console [root://localhost] |/> version
version
EOS_INSTANCE=eosdockertest
EOS_SERVER_VERSION=4.4.38 EOS_SERVER_RELEASE=1
EOS_CLIENT_VERSION=4.4.38 EOS_CLIENT_RELEASE=1
EOS Console [root://localhost] |/> node ls
┌──────────┬─────────────────────────────────────────────────┬────────────────┬──────────┬────────────┬──────┬──────────┬────────┬────────┬────────────────┬─────┐
│type      │                                         hostport│          geotag│    status│      status│  txgw│ gw-queued│  gw-ntx│ gw-rate│  heartbeatdelta│ nofs│
└──────────┴─────────────────────────────────────────────────┴────────────────┴──────────┴────────────┴──────┴──────────┴────────┴────────┴────────────────┴─────┘
 nodesview  eos-fst1.eos-fst1.tutorial.svc.cluster.local:1095     docker::test     online           on    off          0       10      120                2     1
 nodesview  eos-fst2.eos-fst2.tutorial.svc.cluster.local:1095     docker::test     online           on    off          0       10      120                1     1
 nodesview  eos-fst3.eos-fst3.tutorial.svc.cluster.local:1095     docker::test     online           on    off          0       10      120                1     1
 nodesview  eos-fst4.eos-fst4.tutorial.svc.cluster.local:1095     docker::test     online           on    off          0       10      120                1     1
 nodesview  eos-fst5.eos-fst5.tutorial.svc.cluster.local:1095     docker::test     online           on    off          0       10      120                1     1
 nodesview  eos-fst6.eos-fst6.tutorial.svc.cluster.local:1095     docker::test     online           on    off          0       10      120                1     1
 nodesview  eos-fst7.eos-fst7.tutorial.svc.cluster.local:1095     docker::test     online           on    off          0       10      120                1     1

You can mount EOS to the client Pods using FUSE and KRB5 authentication:

kubectl exec -n tutorial -it $(kubectl get pods -n tutorial --no-headers -o custom-columns=":metadata.name" -l app=eos-cli1) -- eos fuse mount /eos

.... trying to create ... /eos
===> Mountpoint   : /eos
===> Fuse-Options : max_readahead=131072,max_write=4194304,fsname=eos-mgm1.eoscluster.cern.ch,url=root://eos-mgm1.eoscluster.cern.ch//eos/
===> fuse readahead        : 1
===> fuse readahead-window : 1048576
===> fuse debug            : 0
===> fuse low-level debug  : 0
===> fuse log-level        : 5
===> fuse write-cache      : 1
===> fuse write-cache-size : 67108864
===> fuse rm level protect : 1
===> fuse lazy-open-ro     : 0
===> fuse lazy-open-rw     : 1
==== fuse multi-threading  : true
info: successfully mounted EOS [root://eos-mgm1.eoscluster.cern.ch] under /eos
kubectl exec -n tutorial -it $(kubectl get pods -n tutorial --no-headers -o custom-columns=":metadata.name" -l app=eos-cli1) -- bash

ls -la /eos/
total 4
drwxrwxr-x.  1 root root    0 Jan  1  1970 .
drwxr-xr-x. 18 root root 4096 Mar 14 10:16 ..
drwxrwxr-x.  1 root root    0 Jan  1  1970 dockertest

Or by running the EOS instance testsuite:

kubectl exec -n tutorial -i $(kubectl get pods -n tutorial --no-headers -o custom-columns=":metadata.name" -l app=eos-mgm) -- eos-instance-test

Delete and clean

Use the delete-all.sh script to remove the EOS instance from your system.

./delete-all.sh tutorial

Image Repository

You can get the images for each automatic build and for each release. The release images are tagged with the release version. Regular images are tagged with the build id of their originating pipeline.

Docker images are accessible from the project’s registry.

docker pull gitlab-registry.cern.ch/dss/eos:<tag>

Example for a build

docker pull gitlab-registry.cern.ch/dss/eos:777552

Example for the latest release

docker pull gitlab-registry.cern.ch/dss/eos:4.6

Kubernetes-ready images are available since release version 4.4.37

Selfmade images

In case you would like to create a different setup, you are welcome to browse and reuse the provided scripts under the image_scripts folder of the eos-docker project to get an idea on how to do it.