1.4.2.2. CVMFS

According to CernVM-FS documentation,

CernVM-FS is implemented as a POSIX read-only file system in user space (a FUSE module). Files and directories are hosted on standard web servers and mounted in the universal namespace /cvmfs.

The key here is that it is read-only, suitable for software deployment. While there is a caching mechanism, it means that it has high latency on first launch on a node that has never fetch this software before.

We will see how it works in the following example.

1.4.2.2.1. Example

create a file cvmfs.ini,

executable              = cvmfs.sh

log                     = cvmfs.log
output                  = cvmfs.out
error                   = cvmfs.err
stream_error            = True
stream_output           = True

when_to_transfer_output = ON_EXIT

request_cpus            = 16
request_memory          = 32999
request_disk            = 32G

queue

The ClassAd involve a script cvmfs.sh,

#!/bin/bash -l

# helpers ##############################################################

COLUMNS=72

print_double_line() {
	eval printf %.0s= '{1..'"${COLUMNS}"\}
	echo
}

print_line() {
	eval printf %.0s- '{1..'"${COLUMNS}"\}
	echo
}

########################################################################

CVMFS='/cvmfs/northgrid.gridpp.ac.uk/simonsobservatory/opt/miniforge3'

print_double_line
echo "Loading a conda environment from CVMFS: $CVMFS"
. "$CVMFS/bin/activate"
print_line
echo "Environment is available at:"
which python

Here, we see that CVMFS='/cvmfs/...' is defined. The example path given here is a minimal conda environment.

In this script, the environment is first activated, and then it shows you the environment is loaded successfully, as evident by seeing which Python it is loading.

As usual, you can submit the job via

condor_submit cvmfs.ini

See Monitor your jobs to see how to monitor the status of your job. For advance use, use this command instead,

condor_submit cvmfs.ini; tail -F cvmfs.log cvmfs.out cvmfs.err

and see Streaming stdout & stderr with tail for an explanation on what it does.

The output of this job will be

========================================================================
Loading a conda environment from CVMFS: /cvmfs/northgrid.gridpp.ac.uk/simonsobservatory/opt/miniforge3
------------------------------------------------------------------------
Environment is available at:
/cvmfs/northgrid.gridpp.ac.uk/simonsobservatory/opt/miniforge3/bin/python

1.4.2.2.2. General softwares available from CVMFS

For your convenience, some commonly used system softwares are included in /cvmfs/northgrid.gridpp.ac.uk/simonsobservatory/usr/bin.

Feel free to include it in your PATH such as by

export PATH="/cvmfs/northgrid.gridpp.ac.uk/simonsobservatory/usr/bin:$PATH"

This includes curl, ffmpeg, git, htop, nano, ranger, tar, tmux, tree, zsh, zstd.

For a full list, run ls /cvmfs/northgrid.gridpp.ac.uk/simonsobservatory/usr/bin.

The list of softwares may change in the future. Their existence is not guaranteed. Please let us know if you have requests on system softwares that you’d like to be supported.