Fix access to pip-installed python packages

Sure, you can add this. This should really be done by Spack though (and I wonder if we can get the info from Spack directly).

$(spack location -i python) would do it, but we don't have spack in the final container.

...

On Fri, Jun 3, 2022 at 12:40 PM Sylvester Joosten (@sly2j) < eicweb.gitlab@gmail.com> wrote:

Sylvester Joosten https://eicweb.phy.anl.gov/sly2j commented #115 (comment 19054):

Sure, you can add this. This should really be done by Spack though (and I wonder if we can get the info from Spack directly).

— Reply to this email directly or view it on GitLab #115 (comment 19054).

You're receiving this email because of your account on eicweb.phy.anl.gov. If you'd like to receive fewer emails, you can unsubscribe https://eicweb.phy.anl.gov/-/sent_notifications/REDACTED/unsubscribe from this thread or adjust your notification settings.

Couldn't we evaluate this during the container build and add the output to our custom environment? Like here: https://eicweb.phy.anl.gov/containers/eic_container/-/blob/master/containers/jug/dev.Dockerfile#L153

@rahmans1

Potentially addressed by !299 (merged), e.g. ref https://github.com/spack/spack/issues/29237. More debugging: compare output of

python -m pip list -v
pip list -v

We have two pips:

python+ensurepip installs its pip in the python prefix (as pip3)
py-pip installs another pip in its own prefix (as pip, and it doesn't seem to push the pip3 to the environment) so the difference is even between

pip list -v
pip3 list -v

We probably should not install py-pip anymore in the spack.yaml.

Just relying on python+ensurepip is not sufficient, though:

pip is used throughout our benchmark jobs, and does not get install by python+ensurepip,
some packages simply depend on py-pip and will pull pip in anyway, reintroducing the issue through seemingly unrelated changes (spack dependents py-pip | wc -l: 1855...).

Also, not installing py-pip leaves you with:

$ python -m pip install pandas
/opt/._local/n2x3cedhh4nka5ftj45x4bme5jnshc3r/bin/python: No module named pip

because ... PYTHONPATH doesn't include the site-packages inside the python prefix where ensurepip has installed pip.

After 280ea05e we now have equal output for:

pip list -v
pip3 list -v
python -m pip list -v

Furthermore, import uproot now works out of the box (container?).

Only remaining issue causing me to keep open (beside fragility of the solution): after starting eic-shell, you can do pip install --upgrade numpy to bump from (spack-installed) 1.22.3 to (latest available) 1.22.4 (which is installed in ~/.local/lib). That is not found, however.

jug_xl> wdconinc@menelaos:~/EIC/benchmarks/physics_benchmarks$ pip install --no-warn-script-location --upgrade numpy
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: numpy in /usr/._local/fcn7tfdqh53qzuztskzjhtzzgg55tuam/lib/python3.9/site-packages (1.22.3)
Collecting numpy
  Using cached numpy-1.22.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (16.8 MB)
Installing collected packages: numpy
Successfully installed numpy-1.22.4
WARNING: You are using pip version 22.0.4; however, version 22.1.2 is available.
You should consider upgrading via the '/opt/software/linux-debian-x86_64/gcc-11.2.0/python-3.9.12-oxf75cycwdo2v5cxb6x7zjxdskdz2ejk/bin/python3.9 -m pip install --upgrade pip' command.
jug_xl> wdconinc@menelaos:~/EIC/benchmarks/physics_benchmarks$ pip list -v | grep numpy
numpy                1.22.3      /usr/._local/fcn7tfdqh53qzuztskzjhtzzgg55tuam/lib/python3.9/site-packages pip
WARNING: You are using pip version 22.0.4; however, version 22.1.2 is available.
You should consider upgrading via the '/opt/software/linux-debian-x86_64/gcc-11.2.0/python-3.9.12-oxf75cycwdo2v5cxb6x7zjxdskdz2ejk/bin/python3.9 -m pip install --upgrade pip' command.

closed

Resolved by installing most/all python packages through spack.

Fix access to pip-installed python packages

Designs

Child items ...

Activity