The following steps assume we are configuring an MGM node together with a QuarkDB cluster on the same machine. The QuarkDB cluster will consist of three instances running on different ports. The operating system is CentOS7.
Add the YUM repositories for QuarkDB, EOS Citrine and their dependencies.
yum-config-manager --add-repo=http://storage-ci.web.cern.ch/storage-ci/quarkdb/tag/el7/x86_64/
Loaded plugins: fastestmirror, kernel-module, ovl, protectbase, versionlock
adding repo from: http://storage-ci.web.cern.ch/storage-ci/quarkdb/tag/el7/x86_64/
[storage-ci.web.cern.ch_storage-ci_quarkdb_tag_el7_x86_64_]
name=added from: http://storage-ci.web.cern.ch/storage-ci/quarkdb/tag/el7/x86_64/
baseurl=http://storage-ci.web.cern.ch/storage-ci/quarkdb/tag/el7/x86_64/
enabled=1
yum-config-manager --add-repo=http://storage-ci.web.cern.ch/storage-ci/eos/citrine/tag/el-7/x86_64/
Loaded plugins: fastestmirror, kernel-module, ovl, protectbase, versionlock
adding repo from: http://storage-ci.web.cern.ch/storage-ci/eos/citrine/tag/el-7/x86_64/
[storage-ci.web.cern.ch_storage-ci_eos_citrine_tag_el-7_x86_64_]
name=added from: http://storage-ci.web.cern.ch/storage-ci/eos/citrine/tag/el-7/x86_64/
baseurl=http://storage-ci.web.cern.ch/storage-ci/eos/citrine/tag/el-7/x86_64/
enabled=1
yum-config-manager --add-repo=http://storage-ci.web.cern.ch/storage-ci/eos/citrine-depend/el-7/x86_64/
Loaded plugins: fastestmirror, kernel-module, ovl, protectbase, versionlock
adding repo from: http://storage-ci.web.cern.ch/storage-ci/eos/citrine-depend/el-7/x86_64/
[storage-ci.web.cern.ch_storage-ci_eos_citrine-depend_el-7_x86_64_]
name=added from: http://storage-ci.web.cern.ch/storage-ci/eos/citrine-depend/el-7/x86_64/
baseurl=http://storage-ci.web.cern.ch/storage-ci/eos/citrine-depend/el-7/x86_64/
enabled=1
Install the XRootD packages (>= 4.8.1) from the EPEL repository.
yum --disablerepo="*" --enablerepo="epel" install xrootd* --exclude=xrootd-fuse
Install the QuarkDB and EOS packages:
yum install -y quarkdb quarkdb-debuginfo redis
yum install -y --nogpgcheck --disablerepo="*" --enablerepo="storage*" libmicrohttpd*
yum install -y --nogpgcheck --disablerepo="cern*" eos-server eos-client eos-rocksdb eos-testkeytab
We will run three instances of QuarkDB on ports 7001, 7002 and 7003. The default contents of /etc/xrootd/xrootd-quarkdb.cfg should be the following:
if exec xrootd
xrd.port 7777
xrd.protocol redis:7777 /usr/lib64/libXrdQuarkDB.so
redis.mode raft
redis.database /var/quarkdb
redis.password_file /etc/eos.keytab
fi
In this example, we’re using the EOS keytab file as a password for QuarkDB as well. The entire file contents will be read, and used as the password. No special parsing of the keytab file occurs, the entire thing is considered as the secret string. We’re using the EOS keytab just for convenience, in principle any 32+ character string can be used as a password. Check out the QDB documentation for more information regarding password authentication.
Using this as a reference, we start customizing the configuration files for our three QuarkDB instances:
for i in {1..3}; do
# Create one configuration file per instance
cp /etc/xrootd/xrootd-quarkdb.cfg /etc/xrootd/xrootd-quarkdb${i}.cfg
# Customize the port
sed -i 's/7777/700'"${i}"'/g' /etc/xrootd/xrootd-quarkdb${i}.cfg
# Customize the storage location
sed -i 's/\/var\/quarkdb/\/var\/lib\/quarkdb\/qdb'"${i}"'/g' /etc/xrootd/xrootd-quarkdb${i}.cfg
# Add myself entry to the config
sed -i 's/^fi/ redis.myself localhost:700'"${i}"'\n&/' /etc/xrootd/xrootd-quarkdb${i}.cfg
# Prepare the log and working directories for the instances
mkdir -p /var/log/quarkdb/
mkdir -p /var/spool/quarkdb/
chown -R daemon:daemon /var/log/quarkdb/
chown -R daemon:daemon /var/spool/quarkdb/
done
All instances will run as user daemon. The ownership of the storage locations needs to be changed accordingly. For changing the ownership of the processes and the location of the log files, we can customize the systemd start-up script as follows:
for i in {1..3}; do
mkdir -p /etc/systemd/system/xrootd@quarkdb${i}.service.d
echo -e "[Service] \nExecStart= \nExecStart=/usr/bin/xrootd -l /var/log/quarkdb/xrootd.log -c /etc/xrootd/xrootd-%i.cfg -k fifo -s /var/run/quarkdb/xrootd-%i.pid -n %i \nUser=daemon \nGroup=daemon \n" > /etc/systemd/system/xrootd@quarkdb${i}.service.d/custom.conf
done
The next step is to initialize the QuarkDB database directory using the quarkdb-create command. For more details please consult the QuarkDB documentation.
for i in {1..3}; do
quarkdb-create --path /var/lib/quarkdb/qdb${i}/ --clusterID 0123456789 --nodes localhost:7001,localhost:7002,localhost:7003
# Change ownership to daemon:daemon
chown -R daemon:daemon /var/lib/quarkdb/qdb${i}/
done
We can now start the three QuarkDB instances and they should soon reach a stable configuration with one master and two followers.
# Start all the QuarkDB instances
for i {1..3}; do
systemctl start xrootd@quarkdb${i};
done
sleep 2
# Check their status
for i in {1..3}; do
systemctl status xrootd@quarkdb${i}
done
At this point the QuarkDB cluster should be up and running. The logs from the individual instances can be found in /var/log/quarkdb/quarkdb[1-3]/xrootd.log. Using the redis comand line interface, we can inspect the status of our cluster.
redis-cli -p 7001 raft_info
1) TERM 324
2) LOG-START 0
3) LOG-SIZE 2
4) LEADER localhost:7001
5) CLUSTER-ID 0123456789
6) COMMIT-INDEX 1
7) LAST-APPLIED 1
8) BLOCKED-WRITES 0
9) LAST-STATE-CHANGE 48 (48 seconds)
10) ----------
11) MYSELF localhost:7001
12) STATUS LEADER
13) ----------
14) MEMBERSHIP-EPOCH 0
15) NODES localhost:7001,localhost:7002,localhost:7003
16) OBSERVERS
17) ----------
18) REPLICA localhost:7002 ONLINE | UP-TO-DATE | NEXT-INDEX 2
19) REPLICA localhost:7003 ONLINE | UP-TO-DATE | NEXT-INDEX 2
To integrate the MGM service with the QuarkDB cluster we need to make several modifications to the default configuration file /etc/xrd.cf.mgm.
Update the mgmofs.nslib directive to load the namespace in QuarkDB implementation:
mgmofs.nslib /usr/lib64/libEosNsQuarkdb.soList the endpoints of the QuarkDB cluster which are used by the MGM daemon to connect to the back-end service:
mgmofs.qdbcluster localhost:7001 localhost:7002 localhost:7003 mgmofs.qdbpassword_file /etc/eos.keytabDo the same for the FST configuration:
fstofs.qdbcluster localhost:7001 localhost:7002 localhost:7003 fstofs.qdbpassword_file /etc/eos.keytabAs well as the MQ service:
mq.qdbcluster localhost:7001 localhost:7002 localhost:7003 mq.qdbpassword_file /etc/eos.keytab
Start the MGM daemon as a master:
systemctl start eos@master systemctl start eos@mgm
At this point you should have the following processes running on the local machine:
ps aux | grep xrootd daemon 3658 0.5 0.3 1920252 28028 ? Ssl Mar15 30:25 /usr/bin/xrootd -l /var/log/quarkdb/xrootd.log -c /etc/xrootd/xrootd-quarkdb1.cfg -k fifo -s /var/run/quarkdb/xrootd-quarkdb1.pid -n quarkdb1 daemon 4369 0.1 0.2 1067196 21688 ? Ssl Mar15 10:10 /usr/bin/xrootd -l /var/log/quarkdb/xrootd.log -c /etc/xrootd/xrootd-quarkdb2.cfg -k fifo -s /var/run/quarkdb/xrootd-quarkdb2.pid -n quarkdb2 daemon 4409 0.1 0.2 1133760 17600 ? Ssl Mar15 10:01 /usr/bin/xrootd -l /var/log/quarkdb/xrootd.log -c /etc/xrootd/xrootd-quarkdb3.cfg -k fifo -s /var/run/quarkdb/xrootd-quarkdb3.pid -n quarkdb3 daemon 4896 0.0 0.2 110324 18240 ? Ssl Mar15 0:00 /usr/bin/xrootd -n sync -c /etc/xrd.cf.sync -l /var/log/eos/xrdlog.sync -s /tmp/xrootd.sync.pid -Rdaemon daemon 5105 0.5 3.0 1457476 225548 ? Ssl Mar15 31:22 /usr/bin/xrootd -n mgm -c /etc/xrd.cf.mgm -l /var/log/eos/xrdlog.mgm -s /tmp/xrootd.mgm.pid -Rdaemon daemon 5146 0.0 0.3 340884 22972 ? S Mar15 0:00 /usr/bin/xrootd -n mgm -c /etc/xrd.cf.mgm -l /var/log/eos/xrdlog.mgm -s /tmp/xrootd.mgm.pid -Rdaemon
In a production environment the MGM daemon and each of the QuarkDB instances of the cluster should run on different machines. Furthermore, for optimal performance of the QuarkDB backend, at least the QuarkDB master should have the /var/lib/quarkdb/ directory stored on an SSD partition.
The first step in converting an in-memory namespace to QuarkDB is to compact the file and directory changelog files using the eos-log-compact tool:
eos-log-compact /var/eos/md/file.mdlog /var/eos/md/compacted_file.mdlog eos-log-compact /var/eos/md/directory.mdlog /var/eos/md/compacted_directory.mdlog
The conversion process requires that the entire namespace is loaded into memory, just like the normal booting of the namespace. Therefore, one must ensure that the machine used for the conversion has enough RAM to hold the namespace data structures. To achive optimum performance, it is recommended that both the changelog files and the /var/lib/quarkdb/ directory are stored on an SSD backed partition.
To speed up the initial import, QuarkDB has a special bulkload configuration mode in which we’re expected to do only write operations towards the backend. In this case the compaction of the data stored in QuarkDB happends only at the end, therefore minimising the number of I/O operations and thus speeding up the entire process. Create the usual QuarkDB directory structure by using the quarkdb-create tool. Below is an example of a QurkDB configuration file that uses the bulkload mode:
xrd.port 7777 xrd.protocol redis:7777 /usr/lib64/libXrdQuarkDB.so redis.mode bulkload redis.database /var/lib/quarkdb/convert/
After starting the QuarkDB service, we can use the eos-ns-convert tool to perform the actual conversion of the namespace.
eos-ns-convert /var/eos/md/compacted_file.mdlog /var/eos/md/compacted_directory localhost 7777Note
The eos-ns-convert tool must use as input the compacted changelog files.
Once the bulkload is done, shut down the instance and create a brand new QuarkDB folder using quarkdb-create in a different location, listing the nodes that will make up the new cluster. Further details on how to configure a new QuarkDB cluster can be found here QuarkDB.
The newly created QuarkDB raft-journal directory for each of the instances can be deleted. The raft journal stored in /var/lib/quarkdb/convert/ needs to be copied to the QuarkDB directory of the new instances in the cluster. For this operation, one can use simple cp/scp. Make sure that the configuration for all of the new QuarkDB instances is in raft mode and NOT bulkmode. At this point all the instances in the cluster can be started and the system should rapidly reach a stable configuration with one master and several slaves.
For further information see QuarkDB.