use_singularity.rst 5.82 KB
Newer Older
Dmitry Romanov's avatar
Dmitry Romanov committed
1
2
3
4
5
6
Use singularity
===============

Install singularity
-------------------

Dmitry Romanov's avatar
Dmitry Romanov committed
7
8
Eic image require singularity > 3.0

Dmitry Romanov's avatar
Dmitry Romanov committed
9
10
The `oficial installation instructions <https://sylabs.io/guides/3.0/user-guide/installation.html>`_ have many steps. 

Dmitry Romanov's avatar
Dmitry Romanov committed
11
If you have ubuntu, there is a `debian repo with 3.5.2 version <https://packages.debian.org/source/sid/singularity-container>`_, 
Dmitry Romanov's avatar
Dmitry Romanov committed
12
13
which works pretty nicely (You will have to install dependencies, it will print them...)

Dmitry Romanov's avatar
Dmitry Romanov committed
14
.. code:: bash
Dmitry Romanov's avatar
Dmitry Romanov committed
15

Dmitry Romanov's avatar
Dmitry Romanov committed
16
   sudo apt install containernetworking-plugins
Dmitry Romanov's avatar
Dmitry Romanov committed
17
   wget http://ftp.fi.debian.org/debian/pool/main/s/singularity-container/singularity-container_3.5.2+ds1-1_amd64.deb
Dmitry Romanov's avatar
Dmitry Romanov committed
18
   sudo dpkg -i singularity-container_3.5.2+ds1-1_amd64.deb
Dmitry Romanov's avatar
Dmitry Romanov committed
19

Dmitry Romanov's avatar
Dmitry Romanov committed
20
21
22
Please don't install NeuroDebian repo from the repo as it holds v2.6 and eic image require singularity > 3.0.


Dmitry Romanov's avatar
Dmitry Romanov committed
23
24
25
26

Install work environment
------------------------
   
Dmitry Romanov's avatar
Dmitry Romanov committed
27
The below command automatically creates the right working environment for detector development and running the reconstruction. 
Dmitry Romanov's avatar
Dmitry Romanov committed
28
29
30
31
It checks if there are CVMFS images available (which is true for JLab and BNL farms) and links them or downloads images (which is a scenario for users laptops). 
It also creates eic_shell with the right environment setup, prepares the current dir to work with detector or etc. 


Dmitry Romanov's avatar
Dmitry Romanov committed
32
33
34
35
36
37
.. code:: bash 

   # Easy to remember link:
   curl -L get.athena-eic.org | bash

   # Which is an alias to:   
Dmitry Romanov's avatar
Dmitry Romanov committed
38
39
40
   curl https://eicweb.phy.anl.gov/containers/eic_container/-/raw/master/install.sh | bash


Dmitry Romanov's avatar
Dmitry Romanov committed
41
42
**install.sh** checks if it is run on BNL or JLab farms, so existing CVMFS images are used and 
installation is almost instant. On local systems singularity images will be downloaded. 
Dmitry Romanov's avatar
Dmitry Romanov committed
43
44


Dmitry Romanov's avatar
Dmitry Romanov committed
45
46
It might be handy to copy install.sh locally and control where singularity images are being copied,
disable CVMFS behaviour, and other parameters:
Dmitry Romanov's avatar
Dmitry Romanov committed
47

Dmitry Romanov's avatar
Dmitry Romanov committed
48
.. code:: bash 
Dmitry Romanov's avatar
Dmitry Romanov committed
49

Dmitry Romanov's avatar
Dmitry Romanov committed
50
51
52
   curl -L get.athena-eic.org -o install.sh
   chmod +x install.sh
   ./install.sh --help
Dmitry Romanov's avatar
Dmitry Romanov committed
53

Dmitry Romanov's avatar
Dmitry Romanov committed
54
55
56
57
58
59
60
61
62
63
===============  =====
Flag             Description
===============  =====
-p,--prefix      Working directory to deploy the environment (D: /home/romanov/anl)
-t,--tmpdir      Change tmp directory (D: /tmp)
-n,--no-cvmfs    Disable check for local CVMFS (D: enabled)
-c,--container   Container version (D: jug_xl)
-v,--version     Version to install (D: nightly)
-h,--help        Print this message
===============  =====
Dmitry Romanov's avatar
Dmitry Romanov committed
64

Dmitry Romanov's avatar
Dmitry Romanov committed
65
66
67
68
Example of controlling the container version and image location:          
          
          
.. code:: bash
Dmitry Romanov's avatar
Dmitry Romanov committed
69

Dmitry Romanov's avatar
Dmitry Romanov committed
70
71
72
73
   # installs testing variant and stores image at /mnt/work/images
   # (!) one has to create <prefix>/local/lib for images
   mkdir -p /mnt/work/images/local/lib/
   ./install.sh -v testing -p /mnt/work/images
Dmitry Romanov's avatar
Dmitry Romanov committed
74
75
76



Dmitry Romanov's avatar
Dmitry Romanov committed
77
78
79
80
81
Detector simulation
-------------------

After the installation you should have an executable script named **eic-shell** which basically
just runs singularity setting the proper environment (more information about the script is below)
Dmitry Romanov's avatar
Dmitry Romanov committed
82
83


Dmitry Romanov's avatar
Dmitry Romanov committed
84
85
Precompiled detector
^^^^^^^^^^^^^^^^^^^^
Dmitry Romanov's avatar
Dmitry Romanov committed
86

Dmitry Romanov's avatar
Dmitry Romanov committed
87
88
The jug_xl container comes with precompiled detecor repository. It could be used out of the box 
for simulations or even changing detector parameters that doesn't require recompilation. 
Dmitry Romanov's avatar
Dmitry Romanov committed
89

Dmitry Romanov's avatar
Dmitry Romanov committed
90
The precompiled detector is installed in **/opt/detector** directory. And can be used like this:
Dmitry Romanov's avatar
Dmitry Romanov committed
91

Dmitry Romanov's avatar
Dmitry Romanov committed
92
.. code:: bash
Dmitry Romanov's avatar
Dmitry Romanov committed
93

Dmitry Romanov's avatar
Dmitry Romanov committed
94
95
   # Setup the proper detector environemnt
   source /opt/detector/setup.sh
Dmitry Romanov's avatar
Dmitry Romanov committed
96

Dmitry Romanov's avatar
Dmitry Romanov committed
97
98
   # Run particle gun simulation
   npsim -N2 --compactFile=$DETECTOR_PATH/athena.xml --random.seed 1 --enableGun --gun.energy 2*GeV --gun.thetaMin 0*deg --gun.thetaMax 90*deg --gun.distribution uniform --outputFile ~/test.root
Dmitry Romanov's avatar
Dmitry Romanov committed
99

Dmitry Romanov's avatar
Dmitry Romanov committed
100
.. ::
Dmitry Romanov's avatar
Dmitry Romanov committed
101

Dmitry Romanov's avatar
Dmitry Romanov committed
102
103
104
105
   To note:
   
   1. After sourcing setup.sh the main xml file is ``$DETECTOR_PATH/athena.xml``
   2. One must run npsim from a writable directory, as the fieldmap will be downloaded on the first run
Dmitry Romanov's avatar
Dmitry Romanov committed
106
107


Dmitry Romanov's avatar
Dmitry Romanov committed
108
109
Clone detector from repository
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Dmitry Romanov's avatar
Dmitry Romanov committed
110

Dmitry Romanov's avatar
Dmitry Romanov committed
111
.. code:: bash
Dmitry Romanov's avatar
Dmitry Romanov committed
112

Dmitry Romanov's avatar
Dmitry Romanov committed
113
114
115
   git clone https://eicweb.phy.anl.gov/EIC/detectors/athena.git
   git clone https://eicweb.phy.anl.gov/EIC/detectors/ip6.git
   ln -s ../ip6/ip6 athena/ip6
Dmitry Romanov's avatar
Dmitry Romanov committed
116

Dmitry Romanov's avatar
Dmitry Romanov committed
117
118
119
120
121
122
   # Build athena
   mkdir athena/build && cd athena/build
   cmake ../ -DCMAKE_INSTALL_PREFIX=$ATHENA_PREFIX
   cmake --build ./ --target all -- -j 8
   cmake --install ./
   cd -
Dmitry Romanov's avatar
Dmitry Romanov committed
123

Dmitry Romanov's avatar
Dmitry Romanov committed
124
125
126
127
128
129
   # Build IP6
   mkdir ip6/build && cd ip6/build
   cmake ../ -DCMAKE_INSTALL_PREFIX=$ATHENA_PREFIX
   cmake --build ./ --target all -- -j 8
   cmake --install ./
   cd -
Dmitry Romanov's avatar
Dmitry Romanov committed
130

Dmitry Romanov's avatar
Dmitry Romanov committed
131
132
133
134
135
   # Test run geometry browser:
   dd_web_display athena/athena.xml
   
   # Test run particle gun:
   npsim -N2 --compactFile=athena/athena.xml --random.seed 1 --enableGun --gun.energy 2*GeV --gun.thetaMin 0*deg --gun.thetaMax 90*deg --gun.distribution uniform --outputFile ~/test.root
Dmitry Romanov's avatar
Dmitry Romanov committed
136
137


Dmitry Romanov's avatar
Dmitry Romanov committed
138
.. ::
Dmitry Romanov's avatar
Dmitry Romanov committed
139

Dmitry Romanov's avatar
Dmitry Romanov committed
140
141
142
143
   To note:
   
   1. One might need to rebuild athena repository but only if C++ code is changed. 
   2. No rebuilds are needed when changing only xml files
Dmitry Romanov's avatar
Dmitry Romanov committed
144
145


Dmitry Romanov's avatar
Dmitry Romanov committed
146
147
Advanced information
--------------------
Dmitry Romanov's avatar
Dmitry Romanov committed
148

Dmitry Romanov's avatar
Dmitry Romanov committed
149
150
CVMFS
^^^^^
Dmitry Romanov's avatar
Dmitry Romanov committed
151

Dmitry Romanov's avatar
Dmitry Romanov committed
152
For farms like at BNL or JLab the images are automatically replicated to CVMS:
Dmitry Romanov's avatar
Dmitry Romanov committed
153

Dmitry Romanov's avatar
Dmitry Romanov committed
154
.. code:: bash
Dmitry Romanov's avatar
Dmitry Romanov committed
155

Dmitry Romanov's avatar
Dmitry Romanov committed
156
   /cvmfs/singularity.opensciencegrid.org/eicweb/jug_xl*
Dmitry Romanov's avatar
Dmitry Romanov committed
157

Dmitry Romanov's avatar
Dmitry Romanov committed
158
159
   # example to run
   singularity run /cvmfs/singularity.opensciencegrid.org/eicweb/jug_xl:4.0-acadia-stable
Dmitry Romanov's avatar
Dmitry Romanov committed
160
161


Dmitry Romanov's avatar
Dmitry Romanov committed
162
163

eic-shell explained
Dmitry Romanov's avatar
Dmitry Romanov committed
164
165
^^^^^^^^^^^^^^^^^^^

Dmitry Romanov's avatar
Dmitry Romanov committed
166
167
168
169
170
171
172
173
174
175
176
177
There are actually two eic-shell scripts. One is created by the install scripts
and the other lives in the container. 

The one outside the container just sets ATHENA_PREFIX and runs singularity like:

.. code:: bash

   # $PREFIX here is where you installed everything (by default where install.sh executed)
   export ATHENA_PREFIX=$PREFIX/local
   export SINGULARITY_BINDPATH=/mnt
   singularity exec $PREFIX/local/lib/jug_xl-nightly.sif eic-shell $@

Dmitry Romanov's avatar
Dmitry Romanov committed
178

Dmitry Romanov's avatar
Dmitry Romanov committed
179
The **eic-shell** inside the container loads the proper environemnt and SHELL look correctly
Dmitry Romanov's avatar
Dmitry Romanov committed
180

Dmitry Romanov's avatar
Dmitry Romanov committed
181
.. code:: bash
Dmitry Romanov's avatar
Dmitry Romanov committed
182

Dmitry Romanov's avatar
Dmitry Romanov committed
183
184
   ## Properly setup environment
   . /etc/eic-env.sh
Dmitry Romanov's avatar
Dmitry Romanov committed
185

Dmitry Romanov's avatar
Dmitry Romanov committed
186
187
188
   # What eic-env does in the end is
   export LD_LIBRARY_PATH=$ATHENA_PREFIX/lib:$LD_LIBRARY_PATH
   export PATH=$ATHENA_PREFIX/bin:$PATH
Dmitry Romanov's avatar
Dmitry Romanov committed
189

Dmitry Romanov's avatar
Dmitry Romanov committed
190
   # Run bash shell
Dmitry Romanov's avatar
Dmitry Romanov committed
191