1.4.2.3. Modifying our maintained software environments for software development

When you are using the provided software environments provided by us, you may still want to add more packages that you use, perhaps one that you are developing, that are not currently maintained by us. This includes the case that it might be a new change you are making that is not merged in a released version (such as a GitHub Pull Requests you’re working on).

In these cases, if this package is something you can install via pip, this section provide a method to install it inside your job.

We will see how it works in the following example.

1.4.2.3.1. Example

create a file pip-install-user.ini,

universe       = vanilla
executable     = pip-install-user.sh
output         = pip-install-user.out
error          = pip-install-user.err
log            = pip-install-user.log
stream_output  = True
stream_error   = True

request_cpus   = 1
request_memory = 1G
request_disk   = 1G
queue

The ClassAd involve a script pip-install-user.sh,

#!/bin/bash -l

# helpers ##############################################################

COLUMNS=72

print_double_line() {
	eval printf %.0s= '{1..'"${COLUMNS}"\}
	echo
}

print_line() {
	eval printf %.0s- '{1..'"${COLUMNS}"\}
	echo
}

########################################################################

print_double_line
CONDA_PREFIX=/cvmfs/northgrid.gridpp.ac.uk/simonsobservatory/conda/so-conda-py310-20240104
. "${CONDA_PREFIX}/bin/activate"
echo "Conda environment loaded with python available at:"
which python
export PATH="/cvmfs/northgrid.gridpp.ac.uk/simonsobservatory/usr/bin:$PATH"

print_double_line
echo "Note that this package doesn't exist yet:"
python -c 'import souk; print(souk.__file__)'

print_double_line
echo 'Installing souk from pip:'
pip install --user souk

print_double_line
echo 'Note that this package now exists:'
python -c 'import souk; print(souk.__file__)'
which souk_arch_info

This example uses a package souk that was not in the original environment to demonstrate how this method works.

This uses pip install --user ... to install a package locally without having write access to our provided environment.

Note that this method includes not only a package listed in PyPI, but also a GitHub branch, such as

pip install --user https://github.com/simonsobs-uk/data-centre/archive/master.zip

As usual, you can submit the job via

condor_submit pip-install-user.ini

See Monitor your jobs to see how to monitor the status of your job. For advance use, use this command instead,

condor_submit pip-install-user.ini; tail -F pip-install-user.log pip-install-user.out pip-install-user.err

and see Streaming stdout & stderr with tail for an explanation on what it does.