Commit daefb1c8 authored by Dmitry Romanov's avatar Dmitry Romanov
Browse files

S3 updates

parent 857cb86e
S3 access
=========
To share files S3 (Simple Storage Service) is used. In this tutorial we use `MinIO client <https://docs.min.io/docs/minio-client-quickstart-guide.html>` to work with it.
You can get access, browse and download files using your browser, using this link:
`ATHENA S3 Space <https://dtn01.sdcc.bnl.gov:9000/minio/eictest/ATHENA/>`_
**The login and password for read access is not a secret and is available for all members** (but can't be published at open places like this site). Please ask us on
`#software-helpdesk Slack channel <https://eicip6.slack.com/archives/C02267ZDARF>`_ or drop an email to one of the SWG.
Installation
------------
Installing MinIO client:
.. code:: bash
wget https://dl.min.io/client/mc/release/linux-amd64/mc
mkdir -p $HOME/bin && mv mc $HOME/bin/mc && chmod +x $HOME/bin/mc
alias mc "$HOME/bin/mc"
mc --autocompletion
> You may name the command diffirently if mc aka midnight commander which also uses `mc` is installed
Getting access
--------------
Public read-only access
.. code:: bash
mc config host add S3 https://dtn01.sdcc.bnl.gov:9000 <login> <password>
Private write-enabled access
.. code:: bash
mc config host add S3rw https://dtn01.sdcc.bnl.gov:9000/ $user $key
Private write-enabled access at BNL
.. code:: bash
mc config host add S3rw http://eicoss01.sdcc.bnl.local:9000 $user $key
Operations
----------
Listing contents (the eictest is historical and may change)
.. code:: bash
mc tree S3/eictest/ATHENA
mc ls S3/eictest/ATHENA
mc find S3/eictest/ATHENA --name '*.hepmc*'
mc du S3/eictest/ATHENA
Copying contents out
.. code:: bash
mc cp --insecure S3/eictest/ATHENA/EVGEN/JETS/crossDivNrgCrab/test_crossDivNrgCrab_25mRad_18x275_v1.hepmc .
Streaming contents out
.. code:: bash
mc cat --insecure S3/eictest/ATHENA/EVGEN/JETS/crossDivNrgCrab/test_crossDivNrgCrab_25mRad_18x275_v1.hepmc | head -n 20
mc cat S3/eictest/ATHENA/EVGEN/JETS/crossDivNrgCrab/test_crossDivNrgCrab_25mRad_18x275_v1.hepmc.gz | gunzip -c | head -n 20
Copying contents in
.. code:: bash
mc cp /gpfs02/eic/bpage/home/eicBeamSimu/Pythia8/headonTestJin/test_crossDivNrgCrab_* S3rw/eictest/ATHENA/EVGEN/JETS/crossDivNrgCrab
Streaming contents in
.. code:: bash
cat /gpfs02/eic/bpage/home/eicBeamSimu/Pythia8/headonTestJin/test_crossDivNrgCrab_25mRad_18x275_v1.hepmc | mc pipe S3rw/eictest/ATHENA/EVGEN/JETS/crossDivNrgCrab/test_crossDivNrgCrab_25mRad_18x275_v1.hepmc
gzip -c /gpfs02/eic/bpage/home/eicBeamSimu/Pythia8/headonTestJin/test_crossDivNrgCrab_25mRad_18x275_v1.hepmc | mc pipe S3rw/eictest/ATHENA/EVGEN/JETS/crossDivNrgCrab/test_crossDivNrgCrab_25mRad_18x275_v1.hepmc.gz
\ No newline at end of file
......@@ -28,6 +28,7 @@ ATHENA Software
howto/run_fast_simulation
howto/full_simulation_tutorial
howto/use_singularity
howto/s3_file_storage
howto/beagle_generator
.. toctree::
......
......@@ -45,5 +45,3 @@ Currently as the new software is being prepared, the suggested packages to do fu
- Fun4All
It is explained in the detailes further.
:ref:_`Fast simulation`
\ No newline at end of file
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment