Skip to content
Snippets Groups Projects
Select Git revision
  • 30e0d33d6605e56849824e16758e502dd2ad5b29
  • master default protected
  • plan_update
3 results

s3_file_storage.rst

Blame
  • user avatar
    Dmitry Romanov authored
    30e0d33d
    History
    s3_file_storage.rst 3.55 KiB

    S3 file storage

    To share files S3 (Simple Storage Service) is used. In this tutorial we use MinIO client to work with it.

    You can get access, browse and download files using your browser, using this link:

    ATHENA S3 Space

    The LOGIN and PASSWORD for read access are available for all members (but can't be published at open places like this site). Please ask on #software-helpdesk Slack channel or email one of the SWG members.

    One can access S3 directly using TFile like this

    TFile* file = TFile::Open("s3https://dtn01.sdcc.bnl.gov:9000/eictest/ATHENA/<path>.root","AUTH=<login>:<password>");

    Or set environment variables S3_ACCESS_KEY and S3_SECRET_KEY instead of using AUTH and putting login and password to code;

    export S3_ACCESS_KEY=<login>
    export S3_SECRET_KEY=<password>

    XRootD access

    One can utilize XRootD server to get access to files without credentials using:

    root://sci-xrootd.jlab.org//osgpool/eic/ATHENA/<path to file>
    std::string fileName = "RECO/JETS/crossDivNrgCrab/DIS_NC_Q2gt10_crossDivNrgCrab_25mRad_18x275_v1.0001.root";
    TFile* file = TFile::Open("root://sci-xrootd.jlab.org//osgpool/eic/ATHENA/" + fileName);
    // ...

    (!) Currently (July 2021) only RECO is on xrootd (since we only have a 2.5 TB limit there)

    Installation

    Installing MinIO client:

    wget https://dl.min.io/client/mc/release/linux-amd64/mc
    mkdir -p $HOME/bin && mv mc $HOME/bin/mc && chmod +x  $HOME/bin/mc
    alias mc="$HOME/bin/mc"
    mc --autocompletion

    > You may name the command diffirently if mc aka midnight commander which also uses mc is installed

    Getting access

    Public read-only access

    mc config host add S3 https://dtn01.sdcc.bnl.gov:9000 <login> <password>

    Private write-enabled access

    mc config host add S3rw https://dtn01.sdcc.bnl.gov:9000/ $user $key

    Private write-enabled access at BNL

    mc config host add S3rw http://eicoss01.sdcc.bnl.local:9000 $user $key

    Operations

    Listing contents (the eictest is historical and may change)

    mc tree S3/eictest/ATHENA
    mc ls S3/eictest/ATHENA
    mc find S3/eictest/ATHENA --name '*.hepmc*'
    mc du S3/eictest/ATHENA

    Copying contents out

    mc cp --insecure S3/eictest/ATHENA/EVGEN/JETS/crossDivNrgCrab/test_crossDivNrgCrab_25mRad_18x275_v1.hepmc .

    Streaming contents out

    mc cat --insecure S3/eictest/ATHENA/EVGEN/JETS/crossDivNrgCrab/test_crossDivNrgCrab_25mRad_18x275_v1.hepmc | head -n 20
    mc cat S3/eictest/ATHENA/EVGEN/JETS/crossDivNrgCrab/test_crossDivNrgCrab_25mRad_18x275_v1.hepmc.gz | gunzip -c | head -n 20

    Copying contents in

    mc cp /gpfs02/eic/bpage/home/eicBeamSimu/Pythia8/headonTestJin/test_crossDivNrgCrab_* S3rw/eictest/ATHENA/EVGEN/JETS/crossDivNrgCrab

    Streaming contents in

    cat /gpfs02/eic/bpage/home/eicBeamSimu/Pythia8/headonTestJin/test_crossDivNrgCrab_25mRad_18x275_v1.hepmc | mc pipe S3rw/eictest/ATHENA/EVGEN/JETS/crossDivNrgCrab/test_crossDivNrgCrab_25mRad_18x275_v1.hepmc
    gzip -c /gpfs02/eic/bpage/home/eicBeamSimu/Pythia8/headonTestJin/test_crossDivNrgCrab_25mRad_18x275_v1.hepmc | mc pipe S3rw/eictest/ATHENA/EVGEN/JETS/crossDivNrgCrab/test_crossDivNrgCrab_25mRad_18x275_v1.hepmc.gz