diff --git a/docs/howto/s3_file_storage.rst b/docs/howto/s3_file_storage.rst new file mode 100644 index 0000000000000000000000000000000000000000..fddd6fa50e44182bf419e9b0153efd81cc4ac1ff --- /dev/null +++ b/docs/howto/s3_file_storage.rst @@ -0,0 +1,88 @@ +S3 access +========= + +To share files S3 (Simple Storage Service) is used. In this tutorial we use `MinIO client <https://docs.min.io/docs/minio-client-quickstart-guide.html>` to work with it. + +You can get access, browse and download files using your browser, using this link: + +`ATHENA S3 Space <https://dtn01.sdcc.bnl.gov:9000/minio/eictest/ATHENA/>`_ + +**The login and password for read access is not a secret and is available for all members** (but can't be published at open places like this site). Please ask us on +`#software-helpdesk Slack channel <https://eicip6.slack.com/archives/C02267ZDARF>`_ or drop an email to one of the SWG. + + +Installation +------------ + +Installing MinIO client: + +.. code:: bash + + wget https://dl.min.io/client/mc/release/linux-amd64/mc + mkdir -p $HOME/bin && mv mc $HOME/bin/mc && chmod +x $HOME/bin/mc + alias mc "$HOME/bin/mc" + mc --autocompletion + +> You may name the command diffirently if mc aka midnight commander which also uses `mc` is installed + +Getting access +-------------- + +Public read-only access + +.. code:: bash + + mc config host add S3 https://dtn01.sdcc.bnl.gov:9000 <login> <password> + + +Private write-enabled access + +.. code:: bash + + mc config host add S3rw https://dtn01.sdcc.bnl.gov:9000/ $user $key + + +Private write-enabled access at BNL + +.. code:: bash + + mc config host add S3rw http://eicoss01.sdcc.bnl.local:9000 $user $key + + +Operations +---------- + +Listing contents (the eictest is historical and may change) + +.. code:: bash + + mc tree S3/eictest/ATHENA + mc ls S3/eictest/ATHENA + mc find S3/eictest/ATHENA --name '*.hepmc*' + mc du S3/eictest/ATHENA + +Copying contents out + +.. code:: bash + + mc cp --insecure S3/eictest/ATHENA/EVGEN/JETS/crossDivNrgCrab/test_crossDivNrgCrab_25mRad_18x275_v1.hepmc . + +Streaming contents out + +.. code:: bash + + mc cat --insecure S3/eictest/ATHENA/EVGEN/JETS/crossDivNrgCrab/test_crossDivNrgCrab_25mRad_18x275_v1.hepmc | head -n 20 + mc cat S3/eictest/ATHENA/EVGEN/JETS/crossDivNrgCrab/test_crossDivNrgCrab_25mRad_18x275_v1.hepmc.gz | gunzip -c | head -n 20 + +Copying contents in + +.. code:: bash + + mc cp /gpfs02/eic/bpage/home/eicBeamSimu/Pythia8/headonTestJin/test_crossDivNrgCrab_* S3rw/eictest/ATHENA/EVGEN/JETS/crossDivNrgCrab + +Streaming contents in + +.. code:: bash + + cat /gpfs02/eic/bpage/home/eicBeamSimu/Pythia8/headonTestJin/test_crossDivNrgCrab_25mRad_18x275_v1.hepmc | mc pipe S3rw/eictest/ATHENA/EVGEN/JETS/crossDivNrgCrab/test_crossDivNrgCrab_25mRad_18x275_v1.hepmc + gzip -c /gpfs02/eic/bpage/home/eicBeamSimu/Pythia8/headonTestJin/test_crossDivNrgCrab_25mRad_18x275_v1.hepmc | mc pipe S3rw/eictest/ATHENA/EVGEN/JETS/crossDivNrgCrab/test_crossDivNrgCrab_25mRad_18x275_v1.hepmc.gz \ No newline at end of file diff --git a/docs/index.rst b/docs/index.rst index 72b6f477ad41138fd477e62f047cd5aa48ab0924..048258cc31a6e9faeaa27036c6f53d68c5f18c25 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -28,6 +28,7 @@ ATHENA Software howto/run_fast_simulation howto/full_simulation_tutorial howto/use_singularity + howto/s3_file_storage howto/beagle_generator .. toctree:: diff --git a/docs/overview/plan.rst b/docs/overview/plan.rst index 269c46f5e1dd9c6f39d026514d181dce084b6d92..3b4592d8f3b980b9703a45e04eeee9e827c7891d 100644 --- a/docs/overview/plan.rst +++ b/docs/overview/plan.rst @@ -45,5 +45,3 @@ Currently as the new software is being prepared, the suggested packages to do fu - Fun4All It is explained in the detailes further. - -:ref:_`Fast simulation` \ No newline at end of file