Skip to content

TraversalError in MultiplexedPath.joinpath when parent in compound path is missing #253

Closed
#254
Closed
TraversalError in MultiplexedPath.joinpath when parent in compound path is missing#253
#254
Assignees
jaraco

Description

@MattiasDC
MattiasDC
opened on Jul 1, 2022

I have a regression between version 5.7.1 and 5.8.0. I think the cause is this commit.

I have a reproduction scenario in the zip.
To reproduce:

  • unzip repro.zip
  • cd repro
  • pip install -e .
  • python -m test.test --info

If you change the setup.cfg to use version 5.7.1, no error is raised
repro.zip

Activity

  1. MattiasDC commented on Jul 1, 2022

    @MattiasDC
    MattiasDC
    on Jul 1, 2022
    Author

    The exception callstack that should occur:

    Traceback (most recent call last):
  File "/home/MDC/git/repro/.env/lib/python3.8/site-packages/importlib_resources/abc.py", line 119, in joinpath
    match = next(matches)
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/MDC/git/repro/test/test.py", line 8, in <module>
    test()
  File "/home/MDC/git/repro/.env/lib/python3.8/site-packages/hydra/main.py", line 49, in decorated_main
    _run_hydra(
  File "/home/MDC/git/repro/.env/lib/python3.8/site-packages/hydra/_internal/utils.py", line 399, in _run_hydra
    hydra.show_info(
  File "/home/MDC/git/repro/.env/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 655, in show_info
    options[info](config_name=config_name, overrides=overrides)
  File "/home/MDC/git/repro/.env/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 588, in _print_all_info
    self._print_config_info(config_name, overrides)
  File "/home/MDC/git/repro/.env/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 470, in _print_config_info
    self._print_search_path(config_name=config_name, overrides=overrides)
  File "/home/MDC/git/repro/.env/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 403, in _print_search_path
    cfg = self._get_cfg(
  File "/home/MDC/git/repro/.env/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 163, in _get_cfg
    cfg = self.compose_config(
  File "/home/MDC/git/repro/.env/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 564, in compose_config
    cfg = self.config_loader.load_configuration(
  File "/home/MDC/git/repro/.env/lib/python3.8/site-packages/hydra/_internal/config_loader_impl.py", line 146, in load_configuration
    return self._load_configuration_impl(
  File "/home/MDC/git/repro/.env/lib/python3.8/site-packages/hydra/_internal/config_loader_impl.py", line 239, in _load_configuration_impl
    defaults_list = create_defaults_list(
  File "/home/MDC/git/repro/.env/lib/python3.8/site-packages/hydra/_internal/defaults_list.py", line 719, in create_defaults_list
    defaults, tree = _create_defaults_list(
  File "/home/MDC/git/repro/.env/lib/python3.8/site-packages/hydra/_internal/defaults_list.py", line 689, in _create_defaults_list
    defaults_tree = _create_defaults_tree(
  File "/home/MDC/git/repro/.env/lib/python3.8/site-packages/hydra/_internal/defaults_list.py", line 337, in _create_defaults_tree
    ret = _create_defaults_tree_impl(
  File "/home/MDC/git/repro/.env/lib/python3.8/site-packages/hydra/_internal/defaults_list.py", line 420, in _create_defaults_tree_impl
    return _expand_virtual_root(repo, root, overrides, skip_missing)
  File "/home/MDC/git/repro/.env/lib/python3.8/site-packages/hydra/_internal/defaults_list.py", line 262, in _expand_virtual_root
    subtree = _create_defaults_tree_impl(
  File "/home/MDC/git/repro/.env/lib/python3.8/site-packages/hydra/_internal/defaults_list.py", line 429, in _create_defaults_tree_impl
    update_package_header(repo=repo, node=parent)
  File "/home/MDC/git/repro/.env/lib/python3.8/site-packages/hydra/_internal/defaults_list.py", line 244, in update_package_header
    loaded = repo.load_config(config_path=node.get_config_path())
  File "/home/MDC/git/repro/.env/lib/python3.8/site-packages/hydra/_internal/config_repository.py", line 337, in load_config
    ret = self.delegate.load_config(config_path=config_path)
  File "/home/MDC/git/repro/.env/lib/python3.8/site-packages/hydra/_internal/config_repository.py", line 86, in load_config
    source = self._find_object_source(
  File "/home/MDC/git/repro/.env/lib/python3.8/site-packages/hydra/_internal/config_repository.py", line 130, in _find_object_source
    if source.is_config(config_path):
  File "/home/MDC/git/repro/.env/lib/python3.8/site-packages/hydra/_internal/core_plugins/importlib_resources_config_source.py", line 91, in is_config
    res = files.joinpath(config_path)
  File "/home/MDC/git/repro/.env/lib/python3.8/site-packages/importlib_resources/readers.py", line 87, in joinpath
    return super().joinpath(*descendants)
  File "/home/MDC/git/repro/.env/lib/python3.8/site-packages/importlib_resources/abc.py", line 121, in joinpath
    raise TraversalError(
importlib_resources.abc.TraversalError: ('Target not found during traversal.', 'hydra', ['config.yaml'])

  2. MattiasDC commented on Jul 1, 2022

    @MattiasDC
    MattiasDC
    on Jul 1, 2022
    Author

    Notice that I would've expected that this exception would be catched by this frame:
    File "/home/MDC/git/repro/.env/lib/python3.8/site-packages/importlib_resources/readers.py", line 87, in joinpath

    As per:
    image
    Am I missing something obvious here?

  3. MattiasDC commented on Jul 1, 2022

    @MattiasDC
    MattiasDC
    on Jul 1, 2022
    Author

    Hi @jaraco, is this something you could have a look at as you recently committed in this code?

  4. jaraco
    self-assigned this
    on Jul 2, 2022

  5. jaraco commented on Jul 2, 2022

    @jaraco
    jaraco
    on Jul 2, 2022
    Member

    Yes. I'll take a look. As described, it's difficult for me to see what's going on, so I'll need to download and unzip the repro to investigate deeper.

  6. jaraco commented on Jul 3, 2022

    @jaraco
    jaraco
    on Jul 3, 2022
    Member

    I've pushed the repro to the issue-253 branch for easier testing.

  7. jaraco commented on Jul 3, 2022

    @jaraco
    jaraco
    on Jul 3, 2022
    Member

    It works fine for me:

     repro issue-253 $ py -3.10 -m venv .venv
 repro issue-253 $ py -m pip install -q -U pip
 repro issue-253 $ py -m pip install -e .
Obtaining file:///Users/jaraco/draft/repro
  Preparing metadata (setup.py) ... done
Collecting hydra-core==1.1.0
  Using cached hydra_core-1.1.0-py3-none-any.whl (144 kB)
Collecting hydra-joblib-launcher==1.1.5
  Using cached hydra_joblib_launcher-1.1.5-py3-none-any.whl (5.2 kB)
Collecting importlib_resources==5.8.0
  Using cached importlib_resources-5.8.0-py3-none-any.whl (28 kB)
Collecting omegaconf==2.1.*
  Using cached omegaconf-2.1.2-py3-none-any.whl (74 kB)
Collecting antlr4-python3-runtime==4.8
  Using cached antlr4_python3_runtime-4.8-py3-none-any.whl
Collecting joblib>=0.14.0
  Using cached joblib-1.1.0-py2.py3-none-any.whl (306 kB)
Collecting PyYAML>=5.1.0
  Using cached PyYAML-6.0-cp310-cp310-macosx_11_0_arm64.whl (173 kB)
Installing collected packages: antlr4-python3-runtime, PyYAML, joblib, importlib_resources, omegaconf, hydra-core, hydra-joblib-launcher, test
  Running setup.py develop for test
Successfully installed PyYAML-6.0 antlr4-python3-runtime-4.8 hydra-core-1.1.0 hydra-joblib-launcher-1.1.5 importlib_resources-5.8.0 joblib-1.1.0 omegaconf-2.1.2 test-0.0.0
 repro issue-253 $ py -m test.test --info
Hydra 1.1.0
===========
Installed Hydra Plugins
***********************
        ConfigSource:
        -------------
                FileConfigSource
                ImportlibResourcesConfigSource
                StructuredConfigSource
        CompletionPlugin:
        -----------------
                BashCompletion
                FishCompletion
                ZshCompletion
        Launcher:
        ---------
                BasicLauncher
                JoblibLauncher
        Sweeper:
        --------
                BasicSweeper
        SearchPathPlugin:
        -----------------
                Dummy

Config search path
******************
| Provider                 | Search path                           |
--------------------------------------------------------------------
| hydra                    | pkg://hydra.conf                      |
| main                     | file:///Users/jaraco/draft/repro/test |
| dummy-search-path-plugin | pkg://test.config                     |
| schema                   | structured://                         |
--------------------------------------------------------------------

Defaults Tree
*************
<root>:
  hydra/config:
    hydra/output: default
    hydra/launcher: joblib
    hydra/sweeper: basic
    hydra/help: default
    hydra/hydra_help: default
    hydra/hydra_logging: default
    hydra/job_logging: default
    hydra/callbacks: null
    hydra/env: default
    _self_
  run

Defaults List
*************
| Config path                 | Package             | _self_ | Parent       | 
------------------------------------------------------------------------------
| hydra/output/default        | hydra               | False  | hydra/config |
| hydra/launcher/joblib       | hydra.launcher      | False  | hydra/config |
| hydra/sweeper/basic         | hydra.sweeper       | False  | hydra/config |
| hydra/help/default          | hydra.help          | False  | hydra/config |
| hydra/hydra_help/default    | hydra.hydra_help    | False  | hydra/config |
| hydra/hydra_logging/default | hydra.hydra_logging | False  | hydra/config |
| hydra/job_logging/default   | hydra.job_logging   | False  | hydra/config |
| hydra/env/default           | hydra.env           | False  | hydra/config |
| hydra/config                | hydra               | True   | <root>       |
| run                         |                     | False  | <root>       |
------------------------------------------------------------------------------
Config
******
{}

  8. jaraco commented on Jul 3, 2022

    @jaraco
    jaraco
    on Jul 3, 2022
    Member

    Aha. So the issue seems only to arise only on older versions of Python.

     repro issue-253 $ py -3.8 -m venv .venv
 repro issue-253 $ py -m pip install -e .
Obtaining file:///Users/jaraco/draft/repro
  Preparing metadata (setup.py) ... done
Collecting hydra-core==1.1.0
  Using cached hydra_core-1.1.0-py3-none-any.whl (144 kB)
Collecting hydra-joblib-launcher==1.1.5
  Using cached hydra_joblib_launcher-1.1.5-py3-none-any.whl (5.2 kB)
Collecting importlib_resources==5.8.0
  Using cached importlib_resources-5.8.0-py3-none-any.whl (28 kB)
Collecting omegaconf==2.1.*
  Using cached omegaconf-2.1.2-py3-none-any.whl (74 kB)
Collecting antlr4-python3-runtime==4.8
  Using cached antlr4-python3-runtime-4.8.tar.gz (112 kB)
  Preparing metadata (setup.py) ... done
Collecting joblib>=0.14.0
  Using cached joblib-1.1.0-py2.py3-none-any.whl (306 kB)
Collecting zipp>=3.1.0
  Using cached zipp-3.8.0-py3-none-any.whl (5.4 kB)
Collecting PyYAML>=5.1.0
  Using cached PyYAML-6.0.tar.gz (124 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Using legacy 'setup.py install' for antlr4-python3-runtime, since package 'wheel' is not installed.
Building wheels for collected packages: PyYAML
  Building wheel for PyYAML (pyproject.toml) ... done
  Created wheel for PyYAML: filename=PyYAML-6.0-cp38-cp38-macosx_12_0_arm64.whl size=45335 sha256=e703e181e9d6429ca0b78de2c2229d95d127275b99c98a5711870bde22f1ac0c
  Stored in directory: /Users/jaraco/Library/Caches/pip/wheels/52/84/66/50912fd7bf1639a31758e40bd4312602e104a8eca1e0da9645
Successfully built PyYAML
Installing collected packages: antlr4-python3-runtime, zipp, PyYAML, joblib, omegaconf, importlib_resources, hydra-core, hydra-joblib-launcher, test
  Running setup.py install for antlr4-python3-runtime ... done
  Running setup.py develop for test
Successfully installed PyYAML-6.0 antlr4-python3-runtime-4.8 hydra-core-1.1.0 hydra-joblib-launcher-1.1.5 importlib_resources-5.8.0 joblib-1.1.0 omegaconf-2.1.2 test-0.0.0 zipp-3.8.0
WARNING: You are using pip version 22.0.4; however, version 22.1.2 is available.
You should consider upgrading via the '/Users/jaraco/draft/repro/.venv/bin/python -m pip install --upgrade pip' command.
 repro issue-253 $ py -m test.test --info
Hydra 1.1.0
===========
Installed Hydra Plugins
***********************
        ConfigSource:
        -------------
                FileConfigSource
                ImportlibResourcesConfigSource
                StructuredConfigSource
        CompletionPlugin:
        -----------------
                BashCompletion
                FishCompletion
                ZshCompletion
        Launcher:
        ---------
                BasicLauncher
                JoblibLauncher
        Sweeper:
        --------
                BasicSweeper
        SearchPathPlugin:
        -----------------
                Dummy

Config search path
******************
Traceback (most recent call last):
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/importlib_resources/abc.py", line 119, in joinpath
    match = next(matches)
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.8/3.8.13_1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/homebrew/Cellar/python@3.8/3.8.13_1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/jaraco/draft/repro/test/test.py", line 8, in <module>
    test()
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/main.py", line 49, in decorated_main
    _run_hydra(
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/utils.py", line 399, in _run_hydra
    hydra.show_info(
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 655, in show_info
    options[info](config_name=config_name, overrides=overrides)
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 588, in _print_all_info
    self._print_config_info(config_name, overrides)
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 470, in _print_config_info
    self._print_search_path(config_name=config_name, overrides=overrides)
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 403, in _print_search_path
    cfg = self._get_cfg(
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 163, in _get_cfg
    cfg = self.compose_config(
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 564, in compose_config
    cfg = self.config_loader.load_configuration(
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/config_loader_impl.py", line 146, in load_configuration
    return self._load_configuration_impl(
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/config_loader_impl.py", line 239, in _load_configuration_impl
    defaults_list = create_defaults_list(
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/defaults_list.py", line 719, in create_defaults_list
    defaults, tree = _create_defaults_list(
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/defaults_list.py", line 689, in _create_defaults_list
    defaults_tree = _create_defaults_tree(
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/defaults_list.py", line 337, in _create_defaults_tree
    ret = _create_defaults_tree_impl(
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/defaults_list.py", line 420, in _create_defaults_tree_impl
    return _expand_virtual_root(repo, root, overrides, skip_missing)
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/defaults_list.py", line 262, in _expand_virtual_root
    subtree = _create_defaults_tree_impl(
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/defaults_list.py", line 429, in _create_defaults_tree_impl
    update_package_header(repo=repo, node=parent)
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/defaults_list.py", line 244, in update_package_header
    loaded = repo.load_config(config_path=node.get_config_path())
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/config_repository.py", line 337, in load_config
    ret = self.delegate.load_config(config_path=config_path)
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/config_repository.py", line 86, in load_config
    source = self._find_object_source(
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/config_repository.py", line 130, in _find_object_source
    if source.is_config(config_path):
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/core_plugins/importlib_resources_config_source.py", line 91, in is_config
    res = files.joinpath(config_path)
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/importlib_resources/readers.py", line 87, in joinpath
    return super().joinpath(*descendants)
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/importlib_resources/abc.py", line 121, in joinpath
    raise TraversalError(
importlib_resources.abc.TraversalError: ('Target not found during traversal.', 'hydra', ['config.yaml'])

  9. jaraco commented on Jul 3, 2022

    @jaraco
    jaraco
    on Jul 3, 2022
    Member

    A bit of debugging:

     repro issue-253 $ py -m pdb -m test.test --info
> /Users/jaraco/draft/repro/test/test.py(1)<module>()
-> import hydra
(Pdb) c
Hydra 1.1.0
===========
Installed Hydra Plugins
***********************
        ConfigSource:
        -------------
                FileConfigSource
                ImportlibResourcesConfigSource
                StructuredConfigSource
        CompletionPlugin:
        -----------------
                BashCompletion
                FishCompletion
                ZshCompletion
        Launcher:
        ---------
                BasicLauncher
                JoblibLauncher
        Sweeper:
        --------
                BasicSweeper
        SearchPathPlugin:
        -----------------
                Dummy

Config search path
******************
Traceback (most recent call last):
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/importlib_resources/abc.py", line 119, in joinpath
    match = next(matches)
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.8/3.8.13_1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/pdb.py", line 1703, in main
    pdb._runmodule(mainpyfile)
  File "/opt/homebrew/Cellar/python@3.8/3.8.13_1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/pdb.py", line 1547, in _runmodule
    self.run(code)
  File "/opt/homebrew/Cellar/python@3.8/3.8.13_1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/bdb.py", line 580, in run
    exec(cmd, globals, locals)
  File "/Users/jaraco/draft/repro/test/test.py", line 1, in <module>
    import hydra
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/main.py", line 49, in decorated_main
    _run_hydra(
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/utils.py", line 399, in _run_hydra
    hydra.show_info(
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 655, in show_info
    options[info](config_name=config_name, overrides=overrides)
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 588, in _print_all_info
    self._print_config_info(config_name, overrides)
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 470, in _print_config_info
    self._print_search_path(config_name=config_name, overrides=overrides)
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 403, in _print_search_path
    cfg = self._get_cfg(
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 163, in _get_cfg
    cfg = self.compose_config(
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 564, in compose_config
    cfg = self.config_loader.load_configuration(
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/config_loader_impl.py", line 146, in load_configuration
    return self._load_configuration_impl(
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/config_loader_impl.py", line 239, in _load_configuration_impl
    defaults_list = create_defaults_list(
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/defaults_list.py", line 719, in create_defaults_list
    defaults, tree = _create_defaults_list(
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/defaults_list.py", line 689, in _create_defaults_list
    defaults_tree = _create_defaults_tree(
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/defaults_list.py", line 337, in _create_defaults_tree
    ret = _create_defaults_tree_impl(
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/defaults_list.py", line 420, in _create_defaults_tree_impl
    return _expand_virtual_root(repo, root, overrides, skip_missing)
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/defaults_list.py", line 262, in _expand_virtual_root
    subtree = _create_defaults_tree_impl(
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/defaults_list.py", line 429, in _create_defaults_tree_impl
    update_package_header(repo=repo, node=parent)
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/defaults_list.py", line 244, in update_package_header
    loaded = repo.load_config(config_path=node.get_config_path())
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/config_repository.py", line 337, in load_config
    ret = self.delegate.load_config(config_path=config_path)
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/config_repository.py", line 86, in load_config
    source = self._find_object_source(
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/config_repository.py", line 130, in _find_object_source
    if source.is_config(config_path):
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/core_plugins/importlib_resources_config_source.py", line 91, in is_config
    res = files.joinpath(config_path)
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/importlib_resources/readers.py", line 87, in joinpath
    return super().joinpath(*descendants)
  File "/Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/importlib_resources/abc.py", line 121, in joinpath
    raise TraversalError(
importlib_resources.abc.TraversalError: ('Target not found during traversal.', 'hydra', ['config.yaml'])
Uncaught exception. Entering post mortem debugging
Running 'cont' or 'step' will restart the program
> /Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/importlib_resources/abc.py(121)joinpath()
-> raise TraversalError(
(Pdb) up 2
> /Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/hydra/_internal/core_plugins/importlib_resources_config_source.py(91)is_config()
-> res = files.joinpath(config_path)
(Pdb) files
MultiplexedPath('/Users/jaraco/draft/repro/test/config')
(Pdb) config_path
'hydra/config.yaml'
(Pdb) d
> /Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/importlib_resources/readers.py(87)joinpath()
-> return super().joinpath(*descendants)
(Pdb) descendants
('hydra/config.yaml',)
(Pdb) d
> /Users/jaraco/draft/repro/.venv/lib/python3.8/site-packages/importlib_resources/abc.py(121)joinpath()
-> raise TraversalError(
(Pdb) l
116                 traversable for traversable in self.iterdir() if traversable.name == target
117             )
118             try:
119                 match = next(matches)
120             except StopIteration:
121  ->             raise TraversalError(
122                     "Target not found during traversal.", target, list(names)
123                 )
124             return match.joinpath(*names)
125  
126         def __truediv__(self, child: StrPath) -> "Traversable":
(Pdb) !list(self.iterdir())
[PosixPath('/Users/jaraco/draft/repro/test/config/run.yaml')]
(Pdb) target
'hydra'

    It appears from the code that hydra is looking for hydra/config.yaml in /Users/jaraco/draft/repro/test/config and because hydra doesn't exist in test/config, the TraversalError is raised.

  10. jaraco commented on Jul 3, 2022

    @jaraco
    jaraco
    on Jul 3, 2022
    Member

    Looking at the calling code, it does appear as if the code is designed to trap exceptions when importlib_resources.files() is called, but not when .joinpath() is called.

    Perhaps the refactoring that happened in 5.8.0 changed the behavior around when an exception is raised.

  11. MattiasDC commented on Jul 3, 2022

    @MattiasDC
    MattiasDC
    on Jul 3, 2022
    Author

    Hi @jaraco, but shouldn´t this code catch already?

  12. jaraco commented on Jul 3, 2022

    @jaraco
    jaraco
    on Jul 3, 2022
    Member

    Here's what changed from 5.7.1 to 5.8.0.

  13. MattiasDC commented on Jul 3, 2022

    @MattiasDC
    MattiasDC
    on Jul 3, 2022
    Author

    Hi @jaraco, but shouldn´t this code catch already?

    Ah, but that code can reraise..

  14. MattiasDC commented on Jul 3, 2022

    @MattiasDC
    MattiasDC
    on Jul 3, 2022
    Author

    Can you confirm that this re-raise, which is new, is as intended?
    If that's the case, this issue can be closed and I'll log an issue on hydra instead.

  15. jaraco commented on Jul 3, 2022

    @jaraco
    jaraco
    on Jul 3, 2022
    Member

    Aha. I see the issue. Previously, MultiplexedPath.joinpath(), when passed a compound child path (like "hydra/config.yaml"), would incorrectly assume that the child was not a compound path but instead was a simple filename, so would join it to the first path (i.e. test/config/hydra/config.yaml) not raising an Exception and since that doesn't .exist(), the hydra code could handle it fine.

    With 5.8 (#250 in particular), however, joinpath is more assertive about resolving parents of any path. If a parent (in this case test/config/hydra doesn't exist, it cannot continue traversal, so raises an exception.

    Can you confirm that this re-raise, which is new, is as intended? If that's the case, this issue can be closed and I'll log an issue on hydra instead.

    The change was intended, but it wasn't necessarily intended that errors would be raised on joinpath where they weren't before. I need to consider whether it's possible and worthwhile to retain the prior, more lenient behavior.

    To be sure Hydra could work around the issue by simply moving the joinpath() up into the try block, but I don't want to recommend that until I'm certain that importlib_resources shouldn't address this issue.

  16. jaraco commented on Jul 3, 2022

    @jaraco
    jaraco
    on Jul 3, 2022
    Member

    I also want to figure out if the issue affects Traversables other that MultiplexedPath.

    Edit: After review, I believe the issue only affects MultiplexedPath, because the other implementation that changed (simple.ResourceContainer) would have raised a StopIteration under the same conditions.

  17. jaraco
    added 3 commits that reference this issue on Jul 3, 2022

    Add test capturing expectation revealed by #253.

    d63403b

    Suppress all TraversalErrors in MultiplexedPath. Fixes #253.

    bad45d5

    Update changelog. Ref #253.

    41d8e32

  18. jaraco commented on Jul 3, 2022

    @jaraco
    jaraco
    on Jul 3, 2022
    Member

    I've put together a PR that (a) captures the expectation on the prior release, (b) demonstrates that test fails on the current main, and (c) fixes the issue by removing the re-raise. I'm not yet certain that's the right approach.

    One of the things that makes this behavior interesting is that MultiplexedPath is being used at all. The purpose behind MultiplexedPath is to facilitate resources that exist in namespace packages (packages whose implementation and resources can be found in multiple directories), but in this case, the multiplexed path (and thus the namespace path) appears to be for test.config, which is a directory inside a package.

    I'm uncertain whether this usage (either by hydra or the repro) is correct. What does it mean, for example, when path="pkg://test/config" here?

    Does pkg://test/config mean the test.config Python package? If so, that seems incorrect, because config is not a Python package, and in that case, adding a test/config/__init__.py may correct the underlying expectation as well as workaround the error encountered. See ac61cce, where the repro no longer fails.

    On the other hand, if pkg://test/config means "the 'config' directory found in the 'test' package", then hydra isn't using importlib.resources properly. To get resources from there, the correct usage is files('test').joinpath('config'), which produces a PosixPath and thus won't be subject to the unexpected behavior in MultiplexedPath.

    Regardless, I believe you've identified a real potential defect here and if FFY00 agrees in the review of the PR, this will be fixed in 5.8.1 (and pertinent Pythons where 5.8 was merged). I urge you to determine if hydra or the repro are doing something incorrect that can be improved.

  19. jaraco
    changed the title [-]TraversalError thrown when it should have been catched?[/-] [+]TraversalError in MultiplexedPath.joinpath when parent in compound path is missing[/+] on Jul 3, 2022

  20. MattiasDC commented on Jul 3, 2022

    @MattiasDC
    MattiasDC
    on Jul 3, 2022
    Author

    Some additional information. Documentation for hydra search path plugins can be found here. Notice especially this part:

    pkg:// points to an importable Python module, with . being the separator. __init__.py files are needed in directories for Python to treat them as packages.

    The documentation seems to imply that sometimes you don´t want to treat it as packages though. There is an example provided by Hydra, which can be found here. Notice however that the separator in the example it is not separated by '.', but by '/'. As can be seen here. Furthermore the most nested directory which contains configuration, doesn´t have an __init__.py file. Upper directories do have an __init__.py file.

    I will check with someone of the hydra developer team whether they can give some input.

  21. MattiasDC commented on Jul 3, 2022

    @MattiasDC
    MattiasDC
    on Jul 3, 2022
    Author

    I asked a question to hydra maintainers here

  22. jaraco
    closed this as completedin #254on Jul 22, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions