FUSE Client

The FUSE client allows to access EOS as a mounted file system.

There a two FUSE client modes available:

daemon	user	description
eosd	!root	An end-user private mount which is not shared between users
eosd	root	A system-wide mount shared between users

eosd End-User mount

The end user mount supports the strong authentication methods in EOS:

• KRB5
• X509

The shared mount supports these authentication methods only in the CITRINE branch.

Mouting

You can mount an EOS instance using the EOS client:

# mount my EOS instance to /home/user/eos

eos fuse mount /home/user/eos

Un-Mounting

You can unmount an EOS instance using:

# unmount my EOS instance from /home/user/eos

eos fuse umount /home/user/eos

Note

The mount point can be given as an relative or absolute path!

Warning

The mount point has to be a non-existing or an empty directory!

Authentication

The authentication method is proposed by the EOS server and the client evaluates the server list until it finds a matching one. You can test the used authentication method using (see the authz: field):

[eosdevsrv1]# eos -b whoami
Virtual Identity: uid=755 (99,3) gid=1338 (99,4) [authz:krb5] sudo* host=localhost.localdomain geo-location=513

If the filesystem is mounted you can validate the same information using:

[eosdevsrv1]# cat /home/user/eos/<instance>/proc/whoami

Log File

In case of troubles you can find a log file for private mounts under /tmp/eos-fuse-<uid>.log. If you run the single user mount as root, you find the log file in /var/log/eos/fuse/fuse.log

eosd Shared mount

If you have machines shared by many users like batch nodes it makes sense to use the shared FUSE mount. The shared FUSE mount includes several high-performance add-ons.

Configuration

You configure the FUSE mount via /etc/syconfig/eos (the first two variables have to be defined):

# Directory where to mount FUSE
export EOS_FUSE_MOUNTDIR=/eos/

# MGM URL from where to mount FUSE
export EOS_FUSE_MGM_ALIAS=eosnode.foo.bar

# If the remote directory path does not match the local, you can define the remote path to be different -
# if not defined EOS_FUSE_REMOTEDIR=EOS_FUSE_MOUNTDIR is assumed e.g. local and remote tree have the same prefix
# export EOS_FUSE_REMOTEDIR=/eos/testinstance/subtree/

# Enable FUSE debugging mode (default off)
# export EOS_FUSE_DEBUG=1

# Disable PIO mode (used for high-preformance RAIN file access)
# export EOS_FUSE_NOPIO=1

# Disable multithreading in FUSE (default on)
# export EOS_FUSE_NO_MT=1

# Disable using access for access permission check (default on)
# export EOS_FUSE_NOACCESS=0

# Disable to use the kernel cache (default on)
# export EOS_FUSE_KERNELCACHE=0

# Bypass the buffercache for write - avoids 4k chopping of IO (default off)
# (this is not what people understand under O_DIRECT !!!!
# export EOS_FUSE_DIRECTIO=1

# Disable the write-back cache (default on)
# export EOS_FUSE_CACHE=0

# Set the write-back cache size (default 300M)
# export EOS_FUSE_CACHE_SIZE=0

# Set the write-back cache pagesize (default 256k)
# export EOS_FUSE_CACHE_SIZE=262144

# Use the FUSE big write feature ( FUSE >=2.8 ) (default on)
# export EOS_FUSE_BIGWRITES=1

# Mount all files with 'x' bit to be able to run as an executable (default off)
# export EOS_FUSE_EXEC=1

# Enable protection against recursive deletion (rm -r command)
#    starting from the root of the mount (if 1)
#    or from any of its sub directories at a maximum depth (if >1) (default 1)
# EOS_FUSE_RMLVL_PROTECT=1

# Enable Kerberos authentication. This avoid need to set gateways on the mgm.
#    file cache credential should be used. (default 0)
# EOS_FUSE_USER_KRB5CC=0

# Enable X509 GSI authentication. This avoid need to set gateways on the mgm.
#    file user proxy should be used. (default 0)
# EOS_FUSE_USER_GSIPROXY=0

# When strong authentication is used (EOS_FUSE_USER_KRB5CC=1 or/and EOS_FUSE_USER_GSIPROXY=1),
#    if no strong credentials is found, try to access using unix authentication as nobody
#    note: this does not require to configure the box as a gateway on the mgm but it requires that "nobody" is allowed there
# EOS_FUSE_FALLBACKTONOBODY=0

# If a connection fails using strong authentication, this is the timeout before actully retrying
#    in the meantime, all access by the concerned user will be rejected (indicating authentication failure)
#    !! WARNING: If a low value is used on a batch machine, it could have an impact on the authentication burden on the server side
#    On interactive servers, it will be the longest time taken between refreshing the credentials and this taking effect on the fuse mount
#    (default is XRD_STREAMERRORWINDOW default value)
# EOS_FUSE_STREAMERRORWINDOW=1

# If KRB5 or X509 are enabled, specify the mapping from pid to strong authentication
#    should be kept as symlinks under /var/run/eosd/credentials/pidXXXX
#    (default 0)
# EOS_FUSE_PIDMAP=0

# Enable FUSE read-ahead (default off)
# export EOS_FUSE_RDAHEAD=0

# Configure FUSE read-ahead window (default 128k)
# export EOS_FUSE_RDAHEAD_WINDOW=131072

# Show hidden files from atomic/versioning and backup entries (default off)
# export EOS_FUSE_SHOW_SPECIAL_FILES=0

# Show extended attributes related to EOS itself - this are sys.* and emulated user.eos.* attributes for files (default off)
# export EOS_FUSE_SHOW_EOS_ATTRIBUTES=0

# Add(OR) an additional mode mask to the mode shown (default off)
# export EOS_FUSE_MODE_OVERLAY=000     (use 007 to show things are rwx for w)

# Enable lazy open on read-only files (default off)
# export EOS_FUSE_LAZYOPENRO=1

# Enable lazy open on read-write files (default on
#    this option hides a lot of latency and is recommend to be used
#    it requires how-ever that it is supported by EOS MGM version
# export EOS_FUSE_LAZYOPENRW=1

# Enable asynchronous open of files
#    it is an optimization over the lazy_open hiding even more latency
#    it is used only if lazy-open is activated
# export EOS_FUSE_ASYNC_OPEN=1

# Set the kernel attribute cache time - this is the timewindow before you can see changes done on other clients
# export EOS_FUSE_ATTR_CACHE_TIME=10

# Set the kernel entry timeout - this is the time a directory listing is cached
# export EOS_FUSE_ENTRY_CACHE_TIME=10

# Set the timeout for the kernel negative stat cache
# export EOS_FUSE_NEG_ENTRY_CACHE_TIME=30

# Set the liftime for a file creation ownership - withint this time each file re-open for update will be considered as cached locally and will not see remote changes
# export EOS_FUSE_CREATOR_CAP_LIFETIME=30

# Set the individual max. cache size per write-opened file where we have a creator capability
# export EOS_FUSE_FILE_WB_CACHE_SIZE=67108864

# Set the globa maximum in-memory size for writeback files
# export EOS_FUSE_MAX_WB_INMEMORY_SIZE=536870912

# Configure a log-file prefix - useful for several FUSE instances
# export EOS_FUSE_LOG_PREFIX=dev
# => will create /var/log/eos/fuse.dev.log

# Configure multiple FUSE mounts a,b configured in /etc/sysconfig/eos.a /etc/sysconfig/eos.b
#export EOS_FUSE_MOUNTS="a b"

In most cases one should enable the read-ahead feature with a read-ahead window of 1M on LAN and larger for WAN RTTs and if available use the big writes feature! If you want to mount several EOS instances, you can specify a list of mounts using EOS_FUSE_MOUNTS and then configure these mounts in individual sysconfig files with their name as suffix e.g. mount dev will be defined in /etc/sysconfig/eos.dev. In case of a list of mounts the log file names have the name automatically inserted like fuse.dev.log.

Starting the Service

Once you configured the FUSE mountpoint(s) you can use standard service mechanism to start, stop and check your shared mounts:

# start all eosd instances
service eosd start

# start a particular eosd instance
service eosd start myinstance

# stop all eosd instances
service eosd stop

# stop a particular eosd instance
service eosd stop myinstance

# check the status of all instances
service eosd status

# check the status of a particular instance
service eosd status myinstance

# if instances are up restart them conditional
service eosd condrestart [myinstance]

# shutdown/cleanup all eosd instances running as root
service eosd killall

Example Configuration

He is an example to configure two FUSE mounts from instance use and public

Define two FUSE mounts in /etc/sysconfig/eos

# define which instance mounts we have configured
export EOS_FUSE_MOUNTS="user public"

# #################################################################
# shared EOS FUSE options
# #################################################################
# in-memory write-back shared cache
export EOS_FUSE_CACHE_SIZE=268435456
# just normal logging
export EOS_FUSE_DEBUG=0
# not to verbose - just prints timing and errors
export EOS_FUSE_LOGLEVEL=5
# don't wast time to do parallel IO - only useful for RAIN layouts
export EOS_FUSE_NOPIO=1
# configure 256k readahead (additional to 128k kernel readahead)
export EOS_FUSE_RDAHEAD=1
export EOS_FUSE_RDAHEAD_WINDOW=262144
# stop rm -r for directories with deepness <=2
export EOS_FUSE_RMLVL_PROTECT=2
# configure JEMALLOC
test -e /usr/lib64/libjemalloc.so.1 && export LD_PRELOAD=/usr/lib64/libjemalloc.so.1

# #################################################################
# shared XrdCl options
# #################################################################
# tag xroot traffic
export XRD_APPNAME=eos-fuse
export XRD_CONNECTIONRETRY=4096
export XRD_CONNECTIONWINDOW=10
# keep connections to FSTs for 5 minutes
export XRD_DATASERVERTTL=300
# keep connections to MGM for 30 minutes
export XRD_LOADBALANCERTTL=1800
# standard verbosity for logging
export XRD_LOGLEVEL=Info
# don't follow more than 5 redirects
export XRD_REDIRECTLIMIT=5
# short request timeout of 60s - might be low for high throughput storage
export XRD_REQUESTTIMEOUT=60
export XRD_STREAMERRORWINDOW=15
export XRD_STREAMTIMEOUT=15
# interval how often timeouts are checked .. to get ~60s we have to set it to a second
export XRD_TIMEOUTRESOLUTION=1
# client worker thread pool
export XRD_WORKERTHREADS=16

Then the individual part of each FUSE mount is described in two sysconfig files:

user: /etc/sysconfig/eos.user

# from where do we mount ...
export EOS_FUSE_MGM_ALIAS=eosuser.cern.ch
# where to we mount
export EOS_FUSE_MOUNTDIR=/eos/user/

public: /etc/sysconfig/eos.public

# from where do we mount ...
export EOS_FUSE_MGM_ALIAS=eospublic.cern.ch
# where to we mount
export EOS_FUSE_MOUNTDIR=/eos/public/

Authentication

The shared FUSE mount currently support two authentication modes

• gateway mode authentication
• strong authentication mode featuring both KRB5 and X509

Only one authentication mechanism can be used with a single shared mount and it is specified using the configuration entry EOS_FUSE_USER_KRB5CC mentioned above.

Authentication in gateway mode

Each machine running a shared FUSE mount has to be configured as a gateway machine in the MGM:

Add a FUSE host

vid add gateway fusehost.foo.bar unix

It is also possible now to add a set of hosts matching a hostname pattern:

vid add gateway lxplus* sss

Remove a FUSE host

vid remove gateway fusehost.foo.bar unix

To improve security you can require sss (shared secret authentication) instead of unix (authentication) in the above commands and distribute the sss keytab file to all FUSE hosts /etc/eos.keytab.

Strong authentication mode

Enabling and configuring strong authentication is done using config keys EOS_FUSE_USER_KRB5CC, EOS_FUSE_USER_USERPROXY and EOS_FUSE_USER_KRB5FIRST (see above).

Each linux session can be bound to one credential file. A same user can access the fuse mount using multiple identities using multiple instance. To bind the current linux session to a credential file, the user has to use the script eosfusebind

The following command line

eosfusebind krb5 [credfile]

tries to find a krb5 credential cache file in the following order, stopping at the first match - optional credfile argument if specified - environment variable KRB5CCNAME - default location /tmp/krb5cc_<uid>

The following command line

eosfusebind x509 [credfile]

tries to find a x509 user proxy file in the following order, stopping at the first match - optional credfile argument if specified - environment variable X509_USER_PROXY - default location /tmp/x509up_u<uid>

Warning, eosfusebind does not check that the credential file is valid. It only checks it exists and has 600 permissions. The actual authentication is carried out by the fuse mount. Every time a new binding is made, all bindings from any terminated sessions (for the current user) are cleaned-up. Binding an already bound session replaces the previous binding.

It is possible to show the bindings for the current session or the current user with the following commands

eosfusebind --show-session
eosfusebind --show-user

It is possible to unbind a given session or all the session of the current user using the following command

eosfusebind --unbind-session
eosfusebind --unbind-user

If the process tries to access the fuse mount and if its session is not bound to a valid credential file, access will be refused.

Protection against recursive top level deletion

The configuration entry EOS_FUSE_RMLVL_PROTECT defined above allow to enable this protection. This will deny any deletion to an ‘rm -r’ command starting from the top level directory of the fuse mount down to the specified depth.

For instance, if eos is mounted in /eos and if EOS_FUSE_RMLVL_PROTECT=3, then:

• rm /eos/* WILL run
• rm -i -rf /eos will NOT run
• rm -rf /eos/level2 will NOT run
• rm -r /eos/level2/level3 will NOT run
• rm -r /eos/level2/level3/level4 WILL run.

The rule currently implemented is the following one:

The fuse mount will deny any removal coming from a command named rm with one of the short option(s) being r or one of the long option(s) being recursive if one of the non optional arguments is a path located under the mountpoint at a depth lower than the value specifed by EOS_FUSE_RMLVL_PROTECT.

mount and autofs support

If you have a defined FUSE instances and can manage them with the eosd service scripts, you use a mount wrapper to define mounts in /etc/fstab or mount manually.

Note

You should make sure that you don’t have eosd as a persistent service: /sbin/chkconfig –del eosd

To mount myinstance to the local directory /eos/myinstance you can write:

# mount
mount -t eos myinstance /eos/myinstance

# umount
umount /eos/myinstance

To define a FUSE mount in /etc/fstab you add for example:

myinstance  /eos/myinstance defaults 0 0

If you want to use autofs, you have to create a file /etc/auto.eos :

myinstance -fstype=eos :myinstance

Add to the file /etc/auto.master at the bottom:

/eos /etc/auto.eos

For convenience make sure that you enable browsing in /etc/autofst.conf:

browse_mode = yes # this lets you see the mountdir myinstance in /eos/ as /eos/myinstance/. Once you acces this directory it will be automatically mounted.

Note

Enable autofs with service autofs start

Exporting FUSE filesystems

FUSE export with NFS4

To export FUSE via NFS4 you have to disable(shorten) the attribute caching in the FUSE configuration file:

export EOS_FUSE_ATTR_CACHE_TIME=0.0000000000000001

If you mount an instance as /eos you have to configure an NFS export like this in /etc/exports:

/eos *.cern.ch(fsid=131,rw,insecure,subtree_check,async,root_squash)

You have to start/reload your nfs4 server and then you should be able to access the NFS volume using

mount -t nfs4 <server> <localhost>

FUSE export with CIFS/Samba

To export FUSE via Samba you have only to enable a mode overlay to avoid messages about permission problems during browsing in the FUSE configuration file:

export EOS_FUSE_MODE_OVERLAY=077

The rest of the CIFS server configuration is idential to a local filesystem Samba export.