Skip to content
Snippets Groups Projects
Commit 2d1430da authored by Brett Viren's avatar Brett Viren
Browse files

Address all coments in @trws's latest comment in PR #869.

I addressed them by factoring the code better to follow the visitor
pattern.  This will allow actions to be easily added in the future.
These may not even be file sytsem views.  One could add actions to
generate shell init scripts, JSON DAG-dumpers, GraphViz DOT file
generators, etc (yes, some of these are alread in there - just to give
the idea).

Also added is a top-level test

 $ source share/spack/setup-env.sh
 $ ./share/spack/examples/test_view.sh

Read the top of that script first.
parent 26c5bc9d
Branches
No related tags found
No related merge requests found
'''
Produce a file-system "view" of a Spack DAG.
Concept from Nix, implemented by brett.viren@gmail.com ca 2016.
'''
############################################################################## ##############################################################################
# Copyright (c) 2013, Lawrence Livermore National Security, LLC. # Copyright (c) 2013, Lawrence Livermore National Security, LLC.
# Produced at the Lawrence Livermore National Laboratory. # Produced at the Lawrence Livermore National Laboratory.
...@@ -27,6 +22,45 @@ ...@@ -27,6 +22,45 @@
# along with this program; if not, write to the Free Software Foundation, # along with this program; if not, write to the Free Software Foundation,
# Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
############################################################################## ##############################################################################
'''Produce a "view" of a Spack DAG.
A "view" is the product of applying a function on a set of package specs.
This set consists of:
- specs resolved from the package names given by the user (the seeds)
- all depenencies of the seeds unless user specifies `--no-depenencies`
- less any specs with names matching the regular expressions given by `--exclude`
The `view` command provides a number of functions (the "actions"):
- symlink :: a file system view which is a directory hierarchy that is
the union of the hierarchies of the installed packages in the DAG
where installed files are referenced via symlinks.
- hardlink :: like the symlink view but hardlinks are used
- statlink :: a view producing a status report of a symlink or
hardlink view.
The file system view concept is imspired by Nix, implemented by
brett.viren@gmail.com ca 2016.
'''
# Implementation notes:
#
# This is implemented as a visitor pattern on the set of package specs.
#
# The command line ACTION maps to a visitor_*() function which takes
# the set of package specs and any args which may be specific to the
# ACTION.
#
# To add a new view:
# 1. add a new cmd line args sub parser ACTION
# 2. add any action-specific options/arguments, most likely a list of specs.
# 3. add a visitor_MYACTION() function
# 4. add any visitor_MYALIAS assignments to match any command line aliases
import os import os
import re import re
...@@ -38,33 +72,48 @@ ...@@ -38,33 +72,48 @@
description = "Produce a single-rooted directory view of a spec." description = "Produce a single-rooted directory view of a spec."
def setup_parser(subparser): def setup_parser(sp):
setup_parser.parser = subparser setup_parser.parser = sp
sp.add_argument('-v','--verbose', action='store_true', default=False,
help="Display verbose output.")
sp.add_argument('-e','--exclude', action='append', default=[],
help="Exclude packages with names matching the given regex pattern.")
sp.add_argument('-d', '--dependencies', choices=['true','false','yes','no'],
default='true',
help="Follow dependencies.")
sp = subparser.add_subparsers(metavar='ACTION', dest='action')
ssp = sp.add_subparsers(metavar='ACTION', dest='action')
# The action parameterizes the command but in keeping with Spack # The action parameterizes the command but in keeping with Spack
# patterns we make it a subcommand. # patterns we make it a subcommand.
sps = [ file_system_view_actions = [
sp.add_parser('add', aliases=['link'], ssp.add_parser('symlink', aliases=['add','soft'],
help='Add packages to the view, create view if needed.'), help='Add package files to a filesystem view via symbolic links.'),
sp.add_parser('remove', aliases=['rm'], ssp.add_parser('hardlink', aliases=['hard'],
help='Remove packages from the view, and view if empty.'), help='Add packages files to a filesystem via via hard links.'),
sp.add_parser('status', aliases=['check'], ssp.add_parser('remove', aliases=['rm'],
help='Check status of packages in the view.') help='Remove packages from a filesystem view.'),
ssp.add_parser('statlink', aliases=['status','check'],
help='Check status of packages in a filesystem view.')
] ]
# All these options and arguments are common to every action. # All these options and arguments are common to every action.
for p in sps: for act in file_system_view_actions:
p.add_argument('-e','--exclude', action='append', default=[], act.add_argument('path', nargs=1,
help="exclude any packages which the given re pattern") help="Path to file system view directory.")
p.add_argument('--no-dependencies', action='store_true', default=False, act.add_argument('specs', metavar='spec', nargs='+',
help="just operate on named packages and do not follow dependencies") help="Seed specs of the packages to view.")
p.add_argument('prefix', nargs=1,
help="Path to a top-level directory to receive the view.") ## Other VIEW ACTIONS might be added here.
p.add_argument('specs', nargs=argparse.REMAINDER, ## Some ideas are the following (and some are redundant with existing cmds)
help="specs of packages to expose in the view.") ## A JSON view that dumps a DAG to a JSON file
## A DOT view that dumps to a GraphViz file
## A SHELL INIT view that dumps bash/csh script setting up to use packages in the view
return
### Util functions
def assuredir(path): def assuredir(path):
'Assure path exists as a directory' 'Assure path exists as a directory'
...@@ -79,7 +128,6 @@ def relative_to(prefix, path): ...@@ -79,7 +128,6 @@ def relative_to(prefix, path):
reldir = reldir[1:] reldir = reldir[1:]
return reldir return reldir
def transform_path(spec, path, prefix=None): def transform_path(spec, path, prefix=None):
'Return the a relative path corresponding to given path spec.prefix' 'Return the a relative path corresponding to given path spec.prefix'
if os.path.isabs(path): if os.path.isabs(path):
...@@ -92,54 +140,87 @@ def transform_path(spec, path, prefix=None): ...@@ -92,54 +140,87 @@ def transform_path(spec, path, prefix=None):
path = os.path.join(prefix, path) path = os.path.join(prefix, path)
return path return path
def action_status(spec, prefix): def purge_empty_directories(path):
'Check status of view in prefix against spec' 'Ascend up from the leaves accessible from `path` and remove empty directories.'
dotspack = os.path.join(prefix, '.spack', spec.name) for dirpath, subdirs, files in os.walk(path, topdown=False):
for sd in subdirs:
sdp = os.path.join(dirpath,sd)
try:
os.rmdir(sdp)
except OSError:
pass
def filter_exclude(specs, exclude):
'Filter specs given sequence of exclude regex'
to_exclude = [re.compile(e) for e in exclude]
def exclude(spec):
for e in to_exclude:
if e.match(spec.name):
return True
return False
return [s for s in specs if not exclude(s)]
def flatten(seeds, descend=True):
'Normalize and flattend seed specs and descend hiearchy'
flat = set()
for spec in seeds:
if not descend:
flat.add(spec)
continue
flat.update(spec.normalized().traverse())
return flat
### Action-specific helpers
def check_one(spec, path, verbose=False):
'Check status of view in path against spec'
dotspack = os.path.join(path, '.spack', spec.name)
if os.path.exists(os.path.join(dotspack)): if os.path.exists(os.path.join(dotspack)):
tty.info("Package added: %s"%spec.name) tty.info('Package in view: "%s"'%spec.name)
return return
tty.info("Package missing: %s"%spec.name) tty.info('Package not in view: "%s"'%spec.name)
return return
def action_remove(spec, prefix): def remove_one(spec, path, verbose=False):
'Remove any files found in spec from prefix and purge empty directories.' 'Remove any files found in `spec` from `path` and purge empty directories.'
if not os.path.exists(prefix): if not os.path.exists(path):
return return # done, short circuit
dotspack = transform_path(spec, '.spack', prefix) dotspack = transform_path(spec, '.spack', path)
if not os.path.exists(dotspack): if not os.path.exists(dotspack):
tty.info("Skipping nonexistent package %s"%spec.name) if verbose:
tty.info('Skipping nonexistent package: "%s"'%spec.name)
return return
tty.info("remove %s"%spec.name) if verbose:
tty.info('Removing package: "%s"'%spec.name)
for dirpath,dirnames,filenames in os.walk(spec.prefix): for dirpath,dirnames,filenames in os.walk(spec.prefix):
if not filenames: if not filenames:
continue continue
targdir = transform_path(spec, dirpath, path)
targdir = transform_path(spec, dirpath, prefix)
for fname in filenames: for fname in filenames:
src = os.path.join(dirpath, fname)
dst = os.path.join(targdir, fname) dst = os.path.join(targdir, fname)
if not os.path.exists(dst): if not os.path.exists(dst):
#tty.warn("Skipping nonexistent file for view: %s" % dst)
continue continue
os.unlink(dst) os.unlink(dst)
def action_link(spec, prefix): def link_one(spec, path, link = os.symlink, verbose=False):
'Symlink all files in `spec` into directory `prefix`.' 'Link all files in `spec` into directory `path`.'
dotspack = transform_path(spec, '.spack', prefix) dotspack = transform_path(spec, '.spack', path)
if os.path.exists(dotspack): if os.path.exists(dotspack):
tty.warn("Skipping previously added package %s"%spec.name) tty.warn('Skipping existing package: "%s"'%spec.name)
return return
tty.info("link %s" % spec.name) if verbose:
tty.info('Linking package: "%s"' % spec.name)
for dirpath,dirnames,filenames in os.walk(spec.prefix): for dirpath,dirnames,filenames in os.walk(spec.prefix):
if not filenames: if not filenames:
continue # avoid explicitly making empty dirs continue # avoid explicitly making empty dirs
targdir = transform_path(spec, dirpath, prefix) targdir = transform_path(spec, dirpath, path)
assuredir(targdir) assuredir(targdir)
for fname in filenames: for fname in filenames:
...@@ -148,70 +229,63 @@ def action_link(spec, prefix): ...@@ -148,70 +229,63 @@ def action_link(spec, prefix):
if os.path.exists(dst): if os.path.exists(dst):
if '.spack' in dst.split(os.path.sep): if '.spack' in dst.split(os.path.sep):
continue # silence these continue # silence these
tty.warn("Skipping existing file for view: %s" % dst) tty.warn("Skipping existing file: %s" % dst)
continue continue
os.symlink(src,dst) link(src,dst)
def purge_empty_directories(path):
'Ascend up from the leaves accessible from `path` and remove empty directories.'
for dirpath, subdirs, files in os.walk(path, topdown=False):
for sd in subdirs:
sdp = os.path.join(dirpath,sd)
try:
os.rmdir(sdp)
except OSError:
#tty.warn("Not removing directory with contents: %s" % sdp)
pass
### The canonically named visitor_* functions and their alias maps.
### One for each action.
def visitor_symlink(specs, args):
'Symlink all files found in specs'
path = args.path[0]
assuredir(path)
for spec in specs:
link_one(spec, path, verbose=args.verbose)
visitor_add = visitor_symlink
visitor_soft = visitor_symlink
def visitor_hardlink(specs, args):
'Hardlink all files found in specs'
path = args.path[0]
assuredir(path)
for spec in specs:
link_one(spec, path, os.link, verbose=args.verbose)
visitor_hard = visitor_hardlink
def view_action(action, parser, args): def visitor_remove(specs, args):
'The view command parameterized by the action.' 'Remove all files and directories found in specs from args.path'
to_exclude = [re.compile(e) for e in args.exclude] path = args.path[0]
def exclude(spec):
for e in to_exclude:
if e.match(spec.name):
return True
return False
specs = spack.cmd.parse_specs(args.specs, normalize=True, concretize=True)
if not specs:
parser.print_help()
return 1
prefix = args.prefix[0]
assuredir(prefix)
flat = set()
for spec in specs: for spec in specs:
if args.no_dependencies: remove_one(spec, path, verbose=args.verbose)
flat.add(spec) purge_empty_directories(path)
continue visitor_rm = visitor_remove
flat.update(spec.normalized().traverse())
for spec in flat: def visitor_statlink(specs, args):
if exclude(spec): 'Give status of view in args.path relative to specs'
tty.info('Skipping excluded package: "%s"' % spec.name) path = args.path[0]
continue for spec in specs:
if not os.path.exists(spec.prefix): check_one(spec, path, verbose=args.verbose)
tty.warn('Skipping unknown package: %s in %s' % (spec.name, spec.prefix)) visitor_status = visitor_statlink
continue visitor_check = visitor_statlink
action(spec, prefix)
if args.action in ['remove','rm']:
purge_empty_directories(prefix)
# Finally, the actual "view" command. There should be no need to
# modify anything below when new actions are added.
def view(parser, args): def view(parser, args):
'The view command.' 'Produce a view of a set of packages.'
action = {
'add': action_link, # Process common args
'link': action_link, seeds = [spack.cmd.disambiguate_spec(s) for s in args.specs]
'remove': action_remove, specs = flatten(seeds, args.dependencies.lower() in ['yes','true'])
'rm': action_remove, specs = filter_exclude(specs, args.exclude)
'status': action_status,
'check': action_status # Execute the visitation.
}[args.action] try:
view_action(action, parser, args) visitor = globals()['visitor_' + args.action]
except KeyError:
tty.error('Unknown action: "%s"' % args.action)
visitor(specs, args)
#!/bin/bash
# This will install a few bogus/test packages in order to test the
# `spack view` command. It assumes you have "spack" in your path.
# It makes sub-directories in your CWD and installs and uninstalls
# Spack packages named test-*.
set -x
set -e
view="spack -m view -v"
for variant in +nom ~nom+var +nom+var
do
spack -m uninstall -f -a -y test-d
spack -m install test-d$variant
testdir=test_view
rm -rf $testdir
echo "hardlink may fail if Spack install area and CWD are not same FS"
for action in symlink hardlink
do
$view --dependencies=no $action $testdir test-d
$view -e test-a -e test-b $action $testdir test-d
$view $action $testdir test-d
$view status $testdir test-d
$view -d false remove $testdir test-a
$view remove $testdir test-d
rmdir $testdir # should not fail
done
done
echo "Warnings about skipping existing in the above are okay"
...@@ -9,6 +9,7 @@ class TestA(Package): ...@@ -9,6 +9,7 @@ class TestA(Package):
"""The test-a package""" """The test-a package"""
url = 'file://'+source url = 'file://'+source
homepage = "http://www.example.com/"
version('0.0', '4e823d0af4154fcf52b75dad41b7fd63') version('0.0', '4e823d0af4154fcf52b75dad41b7fd63')
......
...@@ -9,6 +9,7 @@ class TestB(Package): ...@@ -9,6 +9,7 @@ class TestB(Package):
"""The test-b package""" """The test-b package"""
url = 'file://'+source url = 'file://'+source
homepage = "http://www.example.com/"
version('0.0', '4e823d0af4154fcf52b75dad41b7fd63') version('0.0', '4e823d0af4154fcf52b75dad41b7fd63')
......
...@@ -9,6 +9,7 @@ class TestC(Package): ...@@ -9,6 +9,7 @@ class TestC(Package):
"""The test-c package""" """The test-c package"""
url = 'file://'+source url = 'file://'+source
homepage = "http://www.example.com/"
version('0.0', '4e823d0af4154fcf52b75dad41b7fd63') version('0.0', '4e823d0af4154fcf52b75dad41b7fd63')
......
...@@ -9,6 +9,7 @@ class TestD(Package): ...@@ -9,6 +9,7 @@ class TestD(Package):
"""The test-d package""" """The test-d package"""
url = 'file://'+source url = 'file://'+source
homepage = "http://www.example.com/"
version('0.0', '4e823d0af4154fcf52b75dad41b7fd63') version('0.0', '4e823d0af4154fcf52b75dad41b7fd63')
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment