Remove support for generating dotkit files (#11986)

Dotkit is being used only at a few sites and has been deprecated on new
machines. This commit removes all the code that provide support for the
generation of dotkit module files.

A new validator named "deprecatedProperties" has been added to the
jsonschema validators. It permits to prompt a warning message or exit
with an error if a property that has been marked as deprecated is
encountered.

* Removed references to dotkit in the docs
* Removed references to dotkit in setup-env-test.sh
* Added a unit test for the 'deprecatedProperties' schema validator

etc/spack/defaults/config.yaml

@@ -32,7 +32,6 @@ config:
   module_roots:
     tcl:    $spack/share/spack/modules
     lmod:   $spack/share/spack/lmod
-    dotkit: $spack/share/spack/dotkit
 
 
   # Temporary locations Spack can try to use for builds.

etc/spack/defaults/modules.yaml

@@ -16,7 +16,6 @@
 modules:
   enable:
     - tcl
-    - dotkit
   prefix_inspections:
     bin:
       - PATH

lib/spack/docs/basic_usage.rst

@@ -1190,24 +1190,15 @@ Using Extensions
 ^^^^^^^^^^^^^^^^
 
 There are three ways to get ``numpy`` working in Python.  The first is
-to use :ref:`shell-support`.  You can simply ``use`` or ``load`` the
+to use :ref:`shell-support`.  You can simply ``load`` the
 module for the extension, and it will be added to the ``PYTHONPATH``
-in your current shell.
-
-For tcl modules:
+in your current shell:
 
 .. code-block:: console
 
    $ spack load python
    $ spack load py-numpy
 
-or, for dotkit:
-
-.. code-block:: console
-
-   $ spack use python
-   $ spack use py-numpy
-
 Now ``import numpy`` will succeed for as long as you keep your current
 session open.
 

lib/spack/docs/config_yaml.rst

@@ -74,7 +74,6 @@ the location for each type of module.  e.g.:
    module_roots:
      tcl:    $spack/share/spack/modules
      lmod:   $spack/share/spack/lmod
-     dotkit: $spack/share/spack/dotkit
 
 See :ref:`modules` for details.
 

lib/spack/docs/configuration.rst

@@ -463,7 +463,6 @@ account all scopes. For example, to see the fully merged
      module_roots:
        tcl: $spack/share/spack/modules
        lmod: $spack/share/spack/lmod
-       dotkit: $spack/share/spack/dotkit
      build_stage:
      - $tempdir/$user/spack-stage
      - ~/.spack/stage
@@ -514,7 +513,6 @@ down the problem:
    /home/myuser/spack/etc/spack/defaults/config.yaml:32    module_roots:
    /home/myuser/spack/etc/spack/defaults/config.yaml:33      tcl: $spack/share/spack/modules
    /home/myuser/spack/etc/spack/defaults/config.yaml:34      lmod: $spack/share/spack/lmod
-   /home/myuser/spack/etc/spack/defaults/config.yaml:35      dotkit: $spack/share/spack/dotkit
    /home/myuser/spack/etc/spack/defaults/config.yaml:49    build_stage:
    /home/myuser/spack/etc/spack/defaults/config.yaml:50    - $tempdir/$user/spack-stage
    /home/myuser/spack/etc/spack/defaults/config.yaml:51    - ~/.spack/stage

lib/spack/docs/getting_started.rst

@@ -71,7 +71,7 @@ This automatically adds Spack to your ``PATH`` and allows the ``spack``
 command to be used to execute spack :ref:`commands <shell-support>` and
 :ref:`useful packaging commands <packaging-shell-support>`.
 
-If :ref:`environment-modules or dotkit <InstallEnvironmentModules>` is
+If :ref:`environment-modules <InstallEnvironmentModules>` is
 installed and available, the ``spack`` command can also load and unload
 :ref:`modules <modules>`.
 

lib/spack/docs/module_file_support.rst

@@ -13,8 +13,8 @@ The use of module systems to manage user environment in a controlled way
 is a common practice at HPC centers that is often embraced also by individual
 programmers on their development machines. To support this common practice
 Spack integrates with `Environment Modules
-<http://modules.sourceforge.net/>`_ ,  `LMod
-<http://lmod.readthedocs.io/en/latest/>`_ and `Dotkit <https://computing.llnl.gov/?set=jobs&page=dotkit>`_ by
+<http://modules.sourceforge.net/>`_  and  `LMod
+<http://lmod.readthedocs.io/en/latest/>`_ by
 providing post-install hooks that generate module files and commands to manipulate them.
 
 .. note::
@@ -67,7 +67,7 @@ to load the ``cmake`` module:
    $ module load cmake-3.7.2-gcc-6.3.0-fowuuby
 
 Neither of these is particularly pretty, easy to remember, or
-easy to type. Luckily, Spack has its own interface for using modules and dotkits.
+easy to type. Luckily, Spack has its own interface for using modules.
 
 ^^^^^^^^^^^^^
 Shell support
@@ -108,20 +108,10 @@ that the startup time may be slightly increased because of that.
 ^^^^^^^^^^^^^^^^^^^^^^^
 
 Once you have shell support enabled you can use the same spec syntax
-you're used to:
+you're used to and you can use the same shortened names you use
+everywhere else in Spack.
 
-=========================  ==========================
-Modules                    Dotkit
-=========================  ==========================
-``spack load <spec>``      ``spack use <spec>``
-``spack unload <spec>``    ``spack unuse <spec>``
-=========================  ==========================
-
-And you can use the same shortened names you use everywhere else in
-Spack.
-
-For example, if you are using dotkit, this will add the ``mpich``
-package built with ``gcc`` to your path:
+For example this will add the ``mpich`` package built with ``gcc`` to your path:
 
 .. code-block:: console
 
@@ -129,17 +119,10 @@ package built with ``gcc`` to your path:
 
    # ... wait for install ...
 
-   $ spack use mpich %gcc@4.4.7     # dotkit
-   Prepending: mpich@3.0.4%gcc@4.4.7 (ok)
+   $ spack load mpich %gcc@4.4.7    # modules
    $ which mpicc
    ~/spack/opt/linux-debian7-x86_64/gcc@4.4.7/mpich@3.0.4/bin/mpicc
 
-Or, similarly if you are using modules, you could type:
-
-.. code-block:: console
-
-   $ spack load mpich %gcc@4.4.7    # modules
-
 These commands will add appropriate directories to your ``PATH``,
 ``MANPATH``, ``CPATH``, and ``LD_LIBRARY_PATH``.  When you no longer
 want to use a package, you can type unload or unuse similarly:
@@ -147,13 +130,12 @@ want to use a package, you can type unload or unuse similarly:
 .. code-block:: console
 
    $ spack unload mpich %gcc@4.4.7  # modules
-   $ spack unuse  mpich %gcc@4.4.7  # dotkit
 
 .. note::
 
-   These ``use``, ``unuse``, ``load``, and ``unload`` subcommands are
+   The ``load`` and ``unload`` subcommands are
    only available if you have enabled Spack's shell support *and* you
-   have dotkit or modules installed on your machine.
+   have environment-modules installed on your machine.
 
 ^^^^^^^^^^^^^^^^^^^^^^
 Ambiguous module names
@@ -292,8 +274,6 @@ that can be generated by Spack:
   +-----------------------------+--------------------+-------------------------------+----------------------------------------------+----------------------+
   |                             | **Hook name**      |  **Default root directory**   | **Default template file**                    | **Compatible tools** |
   +=============================+====================+===============================+==============================================+======================+
-  |  **Dotkit**                 | ``dotkit``         |  share/spack/dotkit           | share/spack/templates/modules/modulefile.dk  | DotKit               |
-  +-----------------------------+--------------------+-------------------------------+----------------------------------------------+----------------------+
   |  **TCL - Non-Hierarchical** | ``tcl``            |  share/spack/modules          | share/spack/templates/modules/modulefile.tcl | Env. Modules/LMod    |
   +-----------------------------+--------------------+-------------------------------+----------------------------------------------+----------------------+
   |  **Lua - Hierarchical**     | ``lmod``           |  share/spack/lmod             | share/spack/templates/modules/modulefile.lua | LMod                 |
@@ -377,7 +357,7 @@ are named ``modules.yaml``. The default configuration:
 .. literalinclude:: _spack_root/etc/spack/defaults/modules.yaml
    :language: yaml
 
-activates the hooks to generate ``tcl`` and ``dotkit`` module files and inspects
+activates the hooks to generate ``tcl`` module files and inspects
 the installation folder of each package for the presence of a set of subdirectories
 (``bin``, ``man``, ``share/man``, etc.). If any is found its full path is prepended
 to the environment variables listed below the folder name.
@@ -399,12 +379,9 @@ to the generator being customized:
    modules:
      enable:
        - tcl
-       - dotkit
        - lmod
      tcl:
        # contains environment modules specific customizations
-     dotkit:
-       # contains dotkit specific customizations
      lmod:
        # contains lmod specific customizations
 
@@ -590,15 +567,14 @@ do so by using the environment blacklist:
 .. code-block:: yaml
 
    modules:
-     dotkit:
+     tcl:
        all:
          filter:
            # Exclude changes to any of these variables
            environment_blacklist: ['CPATH', 'LIBRARY_PATH']
 
-The configuration above will generate dotkit module files that will not contain
-modifications to either ``CPATH`` or ``LIBRARY_PATH`` and environment module
-files that instead will contain these modifications.
+The configuration above will generate module files that will not contain
+modifications to either ``CPATH`` or ``LIBRARY_PATH``.
 
 
 .. _autoloading-dependencies:

lib/spack/spack/cmd/module.py

@@ -7,7 +7,6 @@
 
 import llnl.util.tty as tty
 
-import spack.cmd.modules.dotkit
 import spack.cmd.modules.lmod
 import spack.cmd.modules.tcl
 
@@ -23,7 +22,6 @@
 
 def setup_parser(subparser):
     sp = subparser.add_subparsers(metavar='SUBCOMMAND', dest='module_command')
-    spack.cmd.modules.dotkit.add_command(sp, _subcommands)
     spack.cmd.modules.lmod.add_command(sp, _subcommands)
     spack.cmd.modules.tcl.add_command(sp, _subcommands)
 

lib/spack/spack/cmd/modules/__init__.py

@@ -137,7 +137,6 @@ def loads(module_type, specs, args, out=sys.stdout):
     module_commands = {
         'tcl': 'module load ',
         'lmod': 'module load ',
-        'dotkit': 'use '
     }
 
     d = {

lib/spack/spack/cmd/modules/dotkit.py

-# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
-# Spack Project Developers. See the top-level COPYRIGHT file for details.
-#
-# SPDX-License-Identifier: (Apache-2.0 OR MIT)
-
-import functools
-
-import spack.cmd.modules
-
-
-def add_command(parser, command_dict):
-    dotkit_parser = parser.add_parser(
-        'dotkit', help='manipulate dotkit module files'
-    )
-    spack.cmd.modules.setup_parser(dotkit_parser)
-
-    command_dict['dotkit'] = functools.partial(
-        spack.cmd.modules.modules_cmd, module_type='dotkit'
-    )

lib/spack/spack/cmd/unuse.py

-# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
-# Spack Project Developers. See the top-level COPYRIGHT file for details.
-#
-# SPDX-License-Identifier: (Apache-2.0 OR MIT)
-
-import argparse
-from spack.cmd.common import print_module_placeholder_help
-
-description = "remove package from environment using dotkit"
-section = "modules"
-level = "long"
-
-
-def setup_parser(subparser):
-    """Parser is only constructed so that this prints a nice help
-       message with -h. """
-    subparser.add_argument(
-        'spec', nargs=argparse.REMAINDER,
-        help='spec of package to unuse with dotkit')
-
-
-def unuse(parser, args):
-    print_module_placeholder_help()

lib/spack/spack/cmd/use.py

-# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
-# Spack Project Developers. See the top-level COPYRIGHT file for details.
-#
-# SPDX-License-Identifier: (Apache-2.0 OR MIT)
-
-import argparse
-from spack.cmd.common import print_module_placeholder_help, arguments
-
-description = "add package to environment using dotkit"
-section = "modules"
-level = "long"
-
-
-def setup_parser(subparser):
-    """Parser is only constructed so that this prints a nice help
-       message with -h. """
-    subparser.add_argument(
-        'spec', nargs=argparse.REMAINDER,
-        help='spec of package to use with dotkit')
-    arguments.add_common_arguments(subparser, ['recurse_dependencies'])
-
-
-def use(parser, args):
-    print_module_placeholder_help()

lib/spack/spack/hooks/__init__.py

@@ -19,7 +19,7 @@
       * post_uninstall(spec)
 
    This can be used to implement support for things like module
-   systems (e.g. modules, dotkit, etc.) or to add other custom
+   systems (e.g. modules, lmod, etc.) or to add other custom
    features.
 """
 import os.path

lib/spack/spack/main.py

@@ -591,7 +591,6 @@ def shell_set(var, value):
     module_roots = spack.config.get('config:module_roots')
     module_to_roots = {
         'tcl': list(),
-        'dotkit': list(),
         'lmod': list()
     }
     for name, path in module_roots.items():

lib/spack/spack/modules/__init__.py

@@ -4,24 +4,20 @@
 # SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
 """This package contains code for creating environment modules, which can
-include dotkits, TCL non-hierarchical modules, LUA hierarchical modules, and
-others.
+include TCL non-hierarchical modules, LUA hierarchical modules, and others.
 """
 
 from __future__ import absolute_import
 
-from .dotkit import DotkitModulefileWriter
 from .tcl import TclModulefileWriter
 from .lmod import LmodModulefileWriter
 
 __all__ = [
-    'DotkitModulefileWriter',
     'TclModulefileWriter',
     'LmodModulefileWriter'
 ]
 
 module_types = {
-    'dotkit': DotkitModulefileWriter,
     'tcl': TclModulefileWriter,
     'lmod': LmodModulefileWriter
 }

lib/spack/spack/modules/dotkit.py

-# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
-# Spack Project Developers. See the top-level COPYRIGHT file for details.
-#
-# SPDX-License-Identifier: (Apache-2.0 OR MIT)
-
-"""This module implements the classes necessary to generate dotkit modules."""
-import os.path
-
-import spack.config
-from .common import BaseConfiguration, BaseFileLayout
-from .common import BaseContext, BaseModuleFileWriter
-
-#: Dotkit specific part of the configuration
-configuration = spack.config.get('modules:dotkit', {})
-
-#: Caches the configuration {spec_hash: configuration}
-configuration_registry = {}
-
-
-def make_configuration(spec):
-    """Returns the dotkit configuration for spec"""
-    key = spec.dag_hash()
-    try:
-        return configuration_registry[key]
-    except KeyError:
-        return configuration_registry.setdefault(
-            key, DotkitConfiguration(spec)
-        )
-
-
-def make_layout(spec):
-    """Returns the layout information for spec """
-    conf = make_configuration(spec)
-    return DotkitFileLayout(conf)
-
-
-def make_context(spec):
-    """Returns the context information for spec"""
-    conf = make_configuration(spec)
-    return DotkitContext(conf)
-
-
-class DotkitConfiguration(BaseConfiguration):
-    """Configuration class for dotkit module files."""
-
-
-class DotkitFileLayout(BaseFileLayout):
-    """File layout for dotkit module files."""
-
-    #: file extension of dotkit module files
-    extension = 'dk'
-
-
-class DotkitContext(BaseContext):
-    """Context class for dotkit module files."""
-
-
-class DotkitModulefileWriter(BaseModuleFileWriter):
-    """Writer class for dotkit module files."""
-    default_template = os.path.join('modules', 'modulefile.dk')

lib/spack/spack/schema/__init__.py

@@ -11,6 +11,7 @@
 import six
 
 import llnl.util.lang
+import llnl.util.tty
 import spack.spec
 
 
@@ -72,11 +73,35 @@ def _validate_spec(validator, is_spec, instance, schema):
                     '"{0}" is an invalid spec [{1}]'.format(spec_str, str(e))
                 )
 
+    def _deprecated_properties(validator, deprecated, instance, schema):
+        if not (validator.is_type(instance, "object") or
+                validator.is_type(instance, "array")):
+            return
+
+        # Get a list of the deprecated properties, return if there is none
+        deprecated_properties = [
+            x for x in instance if x in deprecated['properties']
+        ]
+        if not deprecated_properties:
+            return
+
+        # Retrieve the template message
+        msg = deprecated['message']
+        is_error = deprecated['error']
+        if not is_error:
+            for entry in deprecated_properties:
+                llnl.util.tty.warn(msg.format(property=entry))
+        else:
+            import jsonschema
+            for entry in deprecated_properties:
+                yield jsonschema.ValidationError(msg.format(property=entry))
+
     return jsonschema.validators.extend(
         jsonschema.Draft4Validator, {
             "validate_spec": _validate_spec,
             "properties": _set_defaults,
-            "patternProperties": _set_pp_defaults
+            "patternProperties": _set_pp_defaults,
+            "deprecatedProperties": _deprecated_properties
         }
     )
 

lib/spack/spack/schema/config.py

@@ -41,6 +41,13 @@
                     'lmod': {'type': 'string'},
                     'dotkit': {'type': 'string'},
                 },
+                'deprecatedProperties': {
+                    'properties': ['dotkit'],
+                    'message': 'specifying a "{property}" module root has no '
+                               'effect [support for {property} module files'
+                               ' has been dropped]',
+                    'error': False
+                },
             },
             'source_cache': {'type': 'string'},
             'misc_cache': {'type': 'string'},

lib/spack/spack/schema/modules.py

@@ -135,7 +135,16 @@
                 'default': [],
                 'items': {
                     'type': 'string',
-                    'enum': ['tcl', 'dotkit', 'lmod']}},
+                    'enum': ['tcl', 'dotkit', 'lmod']
+                },
+                'deprecatedProperties': {
+                    'properties': ['dotkit'],
+                    'message': 'cannot enable "{property}" in modules.yaml '
+                               '[support for {property} module files has been'
+                               ' dropped]',
+                    'error': False
+                },
+            },
             'lmod': {
                 'allOf': [
                     # Base configuration
@@ -164,6 +173,13 @@
                 ]
             },
         },
+        'deprecatedProperties': {
+            'properties': ['dotkit'],
+            'message': 'the section "{property}" in modules.yaml has no effect'
+                       ' [support for {property} module files has been '
+                       'dropped]',
+            'error': False
+        },
     },
 }
 

lib/spack/spack/schema/upstreams.py

@@ -20,8 +20,7 @@
                         'type': 'object',
                         'properties': {
                             'tcl': {'type': 'string'},
-                            'lmod': {'type': 'string'},
-                            'dotkit': {'type': 'string'}
+                            'lmod': {'type': 'string'}
                         }
                     }
                 }