4.4. Configuration¶

EOS has two parts of configuration:

Configuration File Types

Description

External config file

shared library location implementing a component and static configuration

these files are typically once edited and during the lifetime of an instance very infrequently changed - a change of these files always requires a service restart!

Internal

configuration of filesystems, policies, mappings aso

these are stored using the EOS configuration engine inside EOS itself - any change is immediately active and never requires a server restart!

There are two methods to store external configuration: the classical EOS4 and the newer EOS5 approach. The EOS5 configuration aims to make the configuration more transparent and simpler using variable substitution for hostnames aso.

EOS4/Classical configuration
EOS5/New configuration

NOTE EOS5 configuration is not (yet) used in production instances at CERN.

Classical Configuration¶

The classical configuration includes the following configuration files:

Service

Configuration File

Type

MGM

/etc/xrd.cf.mgm

XRootD

FST

/etc/xrd.cf.fst

XRootD

MQ

/etc/xrd.cf.mq

XRootD

QDB

/etxrootd/xrootd-quarkdb.cfg

XRootD

ALL

/etc/sysconfig/eos_env

Sysconfig

You can find an example sysconfig file with explanation of configuration variables under /etc/sysconfig/eos_env.example The configuration files coming from an RPM installation have useful default settings. The main changes required in the static xrootd configuration files are concerning plug-ins (e.g. add HTTP(s) plug-ins) or authentication mechanisms and their logical ordering. Individual configuration settings are picked up in topical chapters.

Classical Systemd Services¶

The systemctl services for the four types of daemon are:

systemctl start|stop|status xrootd@quarkdb
systemctl start|stop|status eos@mq
systemctl start|stop|status eos@mgm
systemctl start|stop|status eos@fst

The logical startup order justified by their dependencies should be:

QDB
MQ
MGM
FSTs

The MGM service requires QDB and MQ to be up. The FST Service requires MGM, QDB and MQ to be up. QDB and MQ service have no dependencies.

Start systemd Services at Boot¶

To start EOS services when a machine boots one needs only to enable the corresponding systemd service:

systemctl enable xrootd@quarkdb
systemctl enable eos@mq
systemctl enable eos@mgm
systemctl enable eos@fst

Classical Service Logfiles¶

Service

Logfile Location

Type

MGM

/var/log/eos/mgm

EOS

FST

/var/log/eos/fst

EOS

MQ

/var/log/eos/mq

EOS

QDB

/var/log/xrootd/quarkdb

QDB

The EOS logfiles are usually called xrdlog.(service). In the MGM directory there are sublogfiles, which filter out log lines from the main logfile:

Logfile

Contents

Balancer.log

: log information for the balancer service

Clients.log

: log information for client calls to the MGM

Converter.log

: log information of the converter service

DrainJob.log

: log information for draining

eosxd-logtraces.log

: traces requested using ‘eos fusex evict … sendlog’ from clients

eosxd-stacktraces.log

: traces requested using ‘eos fusex evict … stracktrace’ from clients

error.log

: all message with ERROR level from FSTs

FileInspector.log

: log information for the file inspector service

GeoBalancer.log

: log information for the GEO balancer service

GeoTreeEngine.log

: log information for the GEO tree engine

GroupBalancer.log

: log information for the group balancer

GroupDrainer.log

: log information for the group drainer

Grpc.log

: log information for the GRPC server

Http.log

: log information for the HTTP(S) server

logbook.log

: annotated commands stored in the logbook on user request

LRU.log

: log information for the LRU service

Master.log

: log information for HA master transitions

MetadataFlusher.log

: log information for the metadata flusher

Mounts.log

: log information for FUSE mount/umount

OAuth.log

: log information for OAUTH authentication

Recycle.log

: log information for the Recycle (purging) service

ReplicationTracker.log

: log information for the Replication tracker service

WFE::Job.log

: log information for Workflow Engine jobs

WFE.log

: log information for the Workflow Engine

xrdlog.mgm

: main log file with all log messages

ZMQ.log

: log information for the ZMQ server

EOS5 Configuration¶

Configuration Files¶

The configuration files for the EOS5 service management are located under /etc/eos/

[root@mgm root]# find /etc/eos/config/
/etc/eos/config/
/etc/eos/config/mgm
/etc/eos/config/mgm/mgm
/etc/eos/config/mq
/etc/eos/config/mq/mq
/etc/eos/config/fst
/etc/eos/config/fst/fst
/etc/eos/config/qdb
/etc/eos/config/qdb/qdb
/etc/eos/config/generic
/etc/eos/config/generic/all

Configuration Sections¶

They are internally structured into generic sections init sysonfig:

# ------------------------------------------------------------ #
[init]
# ------------------------------------------------------------ #

# ------------------------------------------------------------ #
[sysconfig]
# ------------------------------------------------------------ #

# ------------------------------------------------------------ #
[unshare]
# ------------------------------------------------------------ #

and daemon specific sections:

# ------------------------------------------------------------ #
[mgm:xrootd:mgm]
# ------------------------------------------------------------ #

# ------------------------------------------------------------ #
[fst:xrootd:fst]
# ------------------------------------------------------------ #

# ------------------------------------------------------------ #
[mq:xrootd:mq]
# ------------------------------------------------------------ #

# ------------------------------------------------------------ #
[qdb:xrootd:qdb]
# ------------------------------------------------------------ #

The first tag inside [daemon:xrootd:name] qdb mq fst mgm references the daemon where this applies. The second tag xrootd reflects that this is actually part of the XRootD configuration file generated for the respective daemon type. The last tag is the name of the daemon instance. It is possible to run one or several of each daemon type per machine. The default name is just the daemon type itself e.g. qdb daemon default name is qdb. On cane have several instances of one type of daemon e.g. mgm1 mgm2 mgm3 fst1 fst2 fst3 aso.

### Daemon Startup If you want to see the config for a specific daemon you can type:

[root@mgm root] eos daemon config fst fst  # show the configuration for the fst daemon and the fst instance name fst

This spits out the three active sections for init, sysconfig and xrootd configuration:

[root@mgm root] eos daemon config mq mq
# ---------------------------------------
# ------------- i n i t -----------------
# ---------------------------------------
mkdir -p /var/run/eos/
chown daemon:root /var/run/eos/
if [ -e /etc/eos.keytab ]; then chown daemon /etc/eos.keytab ; chmod 400 /etc/eos.keytab ; fi
mkdir -p /var/eos/md /var/eos/report
chmod 755 /var/eos /var/eos/report
mkdir -p /var/spool/eos/core/mgm /var/spool/eos/core/mq /var/spool/eos/core/fst /var/spool/eos/core/qdb /var/spool/eos/admin
mkdir -p /var/log/eos
chown -R daemon /var/spool/eos
find /var/log/eos -maxdepth 1 -type d -exec chown daemon {} \;
find /var/eos/ -maxdepth 1 -mindepth 1 -not -path "/var/eos/fs" -not -path "/var/eos/fusex" -type d -exec chown -R daemon {} \;
chmod -R 775 /var/spool/eos
mkdir -p /var/eos/auth /var/eos/stage
chown daemon /var/eos/auth /var/eos/stage
setfacl -m default:u:daemon:r /var/eos/auth/

# ---------------------------------------
# ------------- s y s c o n f i g -------
# ---------------------------------------
SERVER_HOST=...
INSTANCE_NAME=eosdev
GEO_TAG=local
EOS_XROOTD=/opt/eos/xrootd/
LD_LIBRARY_PATH=/opt/eos/xrootd//lib64
LD_PRELOAD=/usr/lib64/libjemalloc.so

# ---------------------------------------
# ------------- x r o o t d  ------------
# ---------------------------------------
# running config file: /var/run/eos/xrd.cf.mq
xrootd.fslib libXrdMqOfs.so
all.export /eos/ nolock
all.role server
xrootd.async off nosf
xrootd.seclib libXrdSec.so
sec.protocol sss -c /etc/eos.keytab -s /etc/eos.keytab
sec.protbind * only sss
xrd.sched mint 16 maxt 1024 idle 128
xrd.port 1097
xrd.network keepalive
xrd.timeout idle 120
mq.maxmessagebacklog 100000
mq.maxqueuebacklog 50000
mq.rejectqueuebacklog 100000
mq.trace low
mq.queue /eos/
#########################################

Init Sections¶

The init section are shell commands which are executed on startup. The default init sections create some of the required directories and change ownership accordingly. The init section of QDB also initializes a new QDB database automatically. Commands which should be executed for all daemons you put into /etc/eos/config/generic/all. Commands to be executed for a specific daemon you put into the daemon config file e.g. /etc/eos/config/qdb/qdb.

Sysconfig Sections¶

The sysconfig section contains variable definitions e.g. /etc/eos/config/generic/all contains:

# ------------------------------------------------------------ #
[sysconfig]
# ------------------------------------------------------------ #

# EOSHOST is replaced by the eos CLI with the current hostname
SERVER_HOST=${EOSHOST}
INSTANCE_NAME=eosdev
GEO_TAG=local

The configuration file syntax allows, that they can work on several hosts without changing host names etc. In this example you see that when you want to reference the machine where you run this command, you just use the variable ${EOSHOST}, so that you don’t have to write myhost1.foo myhost2.foo depending on the machine name. This is also the place where you define the name of your instance.

Unshare Section¶

The unshare section can be used to create a private mount namespace inside the environment of any XRootD process. This is useful if you want to mount a remote filesystem for FSTs, which are only visible to the FST process mount namespace, but to nobody else on the machine itself. A df as root will not show this external mount. You just write the needed mount commmand into the init section and it will be executed on daemon startup. It is possible also to encrypt commands in the init section, in case you have to use a mount key. To get an encrypted command for init sections you use:

eos daemon seal "mount -t nfs ... /nfs/"
enc:fmAWznYjTWqRGfeiDSpfQy3MzQpJOhVI

and you would place enc:fmAWznYjTWqRGfeiDSpfQy3MzQpJOhVI into your init section.

Service Management¶

The systemd command set to start each single daemon manually on a node is: .. code-block:

systemctl start eos5-qdb@qdb
systemctl start eos5-mq@mq
systemctl start eos5-mgm@mgm
systemctl start eos5-fst@fst

The syntax is eos5-daemon@name e.g. start fst daemon with name fst1: systemctl start eos5-fst@fst1

To enable all daemon on startup, you do:

systemctl enable eos5-qdb@qdb
systemctl enable eos5-mq@mq
systemctl enable eos5-mgm@mgm
systemctl enable eos5-fst@fst

EOS5 Service Logfiles¶

Service

Logfile Location

Type

MGM

/var/log/eos/mgm

EOS

FST

/var/log/eos/fst

EOS

MQ

/var/log/eos/mq

EOS

QDB

/var/log/eos/qdb

QDB