Skip to content

Fix relative imports inside an app's subfolders (#726) - #911

Open
HoneyBearTech wants to merge 2 commits into
custom-components:masterfrom
HoneyBearTech:fix-app-subfolder-imports-726
Open

HoneyBearTech wants to merge 2 commits into
custom-components:masterfrom
HoneyBearTech:fix-app-subfolder-imports-726

Conversation

@HoneyBearTech

@HoneyBearTech HoneyBearTech commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Closes #726

What was broken
With this app layout:

apps/myapp/__init__.py          from .subfolder.a import foo
                                from .subfolder.b import bar
apps/myapp/subfolder/a.py       def foo(): ...
apps/myapp/subfolder/b.py       from .a import foo
  • from .a import foo in subfolder/b.py failed, although that's the correct Python. It only worked as from .subfolder.a import foo.
  • Relative imports from a plain module were registered under phantom contexts such as apps.myapp.subfolder.b.a, which loaded a.py a second time and were unloaded on the next reload.

What I changed
In GlobalContext.module_import:

  • a module imported from a subfolder (subfolder.b) now gets its own folder as the base for its relative imports, not the importing package's folder;
  • a plain module's (not __init__.py) relative imports are now resolved against its package, as in Python, instead of against the module itself.

Old form still works, with a warning
Scripts that use the old workaround, from .subfolder.a import foo inside subfolder/b.py, keep working for now. Relative imports are resolved with the new rules first. Only if no file is found are they retried from the base the old rules used, which each module's context keeps. When that finds the file, a warning says the old form is deprecated and will be removed in a future release, and shows the import written the new way (for example write it as .a).

Each module's context is now named after its file's location (apps/myapp/subfolder/a.py is apps.myapp.subfolder.a), so both routes load the same context and no phantom contexts are created. Imports from packages keep the context names they had before; only the phantom names of imports made from plain modules change.

Relative imports from a plain file inside a modules/ package are still not supported, as before.

Testing

  • New tests in tests/test_apps_modules.py: test_app_subfolder_relative_imports (the new form), test_app_subfolder_legacy_relative_imports (the old form, with its warning, before and after a reload) and test_relative_import_context_names (context names, __name__, .. imports, the suggested rewrite, and an import neither rule finds):
    • the issue's layout;
    • a deeper subfolder/deep/c.py doing from ..a import foo and from .. import b;
    • checks of the service result, module __name__ and the exact set of contexts;
    • a full reload, after which the same contexts exist and the service still works.
  • The full test suite, ruff, pylint, mypy and codespell pass on Python 3.14.

A module imported from a subfolder (from .subfolder.b import x) got its
parent folder as the base for its own relative imports, so a sibling
import (from .a import foo) failed, and imports from plain modules were
registered under phantom contexts such as apps.myapp.subfolder.b.a that
were unloaded on the next reload. Use the module's own folder as the
base and resolve a plain module's relative imports against its package.
Fixes custom-components#726.

Signed-off-by: Sean Stoube <31805425+HoneyBearTech@users.noreply.github.com>

@craigbarratt craigbarratt left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @HoneyBearTech, this is the right fix and matches Python.

To avoid breaking scripts that use the old from .subfolder.a workaround, could you add a fallback?

  • Resolve with the new rules first.
  • Only if no file is found, retry with the old base, the importing package's folder. Each context would need to keep that base.
  • When the fallback is used, log a warning that the old form is deprecated and will be removed in a future release, and say how to write the import instead.

If the context name is derived from the file's path (apps/myapp/subfolder/a.py → apps.myapp.subfolder.a) rather than from the importer's name, both routes load the same context, so the phantom contexts can't come back. A copy of your test using the old import in subfolder/b.py would cover it, including the warning.

I tried this locally on top of your branch; it's about 40 lines in module_import, and the full suite passes. Feel free to use or adapt it:

Prototype diff
diff --git a/custom_components/pyscript/global_ctx.py b/custom_components/pyscript/global_ctx.py
index 0d4ac5b..ed77a72 100644
--- a/custom_components/pyscript/global_ctx.py
+++ b/custom_components/pyscript/global_ctx.py
@@ -45,6 +45,8 @@ class GlobalContext:
         self.auto_start: bool = False
         self.module: ModuleType | None = None
         self.rel_import_path: str = rel_import_path
+        # the base the old (pre-2.2) rules used for relative imports, kept as a fallback
+        self.legacy_rel_import_path: str | None = rel_import_path
         self.source: str = source
         self.file_path: str | None = None
         self.mtime: float = mtime
@@ -160,6 +162,7 @@ class GlobalContext:
         pyscript_dir = Function.hass.config.path(FOLDER)
         module_path = module_name.replace(".", "/")
         file_paths = []
+        import_name = "." * import_level + module_name
 
         def find_first_file(file_paths: list[list[str]]) -> list[str] | None:
             for ctx_name, path, rel_path in file_paths:
@@ -171,28 +174,37 @@ class GlobalContext:
         #
         # first build a list of potential import files
         #
+        legacy_file_paths = []
         if import_level > 0:
             if self.rel_import_path is None:
                 raise ImportError("attempted relative import with no known parent package")
-            path = self.rel_import_path
-            if path.endswith("/__init__"):
-                path = os.path.dirname(path)
-            ctx_name = self.name
-            if os.path.basename(self.file_path or "") != "__init__.py" and "." in ctx_name:
-                # a plain module is relative to its parent package, so drop its own name
-                ctx_name = ctx_name[0 : ctx_name.rfind(".")]
-            for _ in range(import_level - 1):
-                path = os.path.dirname(path)
-                idx = ctx_name.rfind(".")
-                if path.find("/") < 0 or idx < 0:
-                    raise ImportError("attempted relative import above parent package")
-                ctx_name = ctx_name[0:idx]
-            ctx_name += f".{module_name}"
-            module_info = [ctx_name, f"{path}/{module_path}.py", os.path.dirname(f"{path}/{module_path}.py")]
-            path += f"/{module_path}"
-            file_paths.append([ctx_name, f"{path}/__init__.py", path])
-            file_paths.append(module_info)
-            module_name = ctx_name[ctx_name.find(".") + 1 :]
+
+            def base_path(path):
+                if path.endswith("/__init__"):
+                    path = os.path.dirname(path)
+                for _ in range(import_level - 1):
+                    path = os.path.dirname(path)
+                    if path.find("/") < 0:
+                        raise ImportError("attempted relative import above parent package")
+                return path
+
+            def candidates(path, plain_rel_path):
+                # the context name follows the file's location, so each file has one context
+                ctx_name = f"{path}/{module_path}".replace("/", ".")
+                return [
+                    [ctx_name, f"{path}/{module_path}/__init__.py", f"{path}/{module_path}"],
+                    [ctx_name, f"{path}/{module_path}.py", plain_rel_path],
+                ]
+
+            path = base_path(self.rel_import_path)
+            file_paths = candidates(path, os.path.dirname(f"{path}/{module_path}.py"))
+            try:
+                legacy_path = base_path(self.legacy_rel_import_path or self.rel_import_path)
+            except ImportError:
+                legacy_path = path
+            if legacy_path != path:
+                legacy_file_paths = candidates(legacy_path, legacy_path)
+            module_name = file_paths[0][0][file_paths[0][0].find(".") + 1 :]
 
         else:
             if self.rel_import_path is not None and self.rel_import_path.startswith("apps/"):
@@ -207,16 +219,38 @@ class GlobalContext:
         #
         # now see if we have loaded it already
         #
-        for ctx_name, _, _ in file_paths:
-            mod_ctx = self.manager.get(ctx_name)
-            if mod_ctx and mod_ctx.module:
-                self.imports.add(mod_ctx.get_name())
-                return mod_ctx.module
+        def find_loaded(file_paths):
+            for ctx_name, _, _ in file_paths:
+                mod_ctx = self.manager.get(ctx_name)
+                if mod_ctx and mod_ctx.module:
+                    self.imports.add(mod_ctx.get_name())
+                    return mod_ctx.module
+            return None
+
+        if mod := find_loaded(file_paths):
+            return mod
 
         #
         # not loaded already, so try to find and import it
         #
         file_info = await Function.hass.async_add_executor_job(find_first_file, file_paths)
+        if not file_info and legacy_file_paths:
+            #
+            # fall back to the old rules, where a module's relative imports were relative to
+            # the package that imported it, not to the module's own folder
+            #
+            mod = find_loaded(legacy_file_paths)
+            file_info = mod or await Function.hass.async_add_executor_job(find_first_file, legacy_file_paths)
+            if file_info:
+                _LOGGER.warning(
+                    "%s: relative import of %s found %s using the old rules, which will be removed in a "
+                    "future release; make the import relative to the module's own folder, as in Python",
+                    self.name,
+                    import_name,
+                    legacy_file_paths[0][0],
+                )
+            if mod:
+                return mod
         if not file_info:
             return None
 
@@ -227,6 +261,13 @@ class GlobalContext:
             ctx_name, global_sym_table=mod.__dict__, manager=self.manager, rel_import_path=rel_import_path
         )
         global_ctx.set_auto_start(self.auto_start)
+        if import_level > 0:
+            # a plain module's old-rules base is the importer's, as before
+            global_ctx.legacy_rel_import_path = (
+                rel_import_path
+                if file_path.endswith("/__init__.py")
+                else (self.legacy_rel_import_path or self.rel_import_path).removesuffix("/__init__")
+            )
         try:
             await self.manager.load_file(global_ctx, file_path)
         except Exception:
diff --git a/tests/test_apps_modules.py b/tests/test_apps_modules.py
index 7114f41..faa62b5 100644
--- a/tests/test_apps_modules.py
+++ b/tests/test_apps_modules.py
@@ -329,3 +329,118 @@ def qux():
     assert hass.services.has_service("pyscript", "myapp_func")
     ret = await hass.services.async_call("pyscript", "myapp_func", {}, return_response=True, blocking=True)
     assert ret == expected
+
+
+@pytest.mark.asyncio
+async def test_app_subfolder_legacy_relative_imports(hass, caplog):
+    """Test relative imports between modules in an app's subfolders, from #726."""
+
+    conf_dir = hass.config.path(FOLDER)
+
+    file_contents = {
+        f"{conf_dir}/apps/myapp/__init__.py": """
+from .subfolder.a import foo
+from .subfolder.b import bar, baz
+from .subfolder.deep.c import qux
+
+@service(supports_response="only")
+def myapp_func():
+    return {"done": [foo(), bar(), baz(), qux()]}
+""",
+        f"{conf_dir}/apps/myapp/subfolder/a.py": """
+def foo():
+    return "foo"
+""",
+        #
+        # a plain module in a subfolder importing its sibling
+        #
+        f"{conf_dir}/apps/myapp/subfolder/b.py": """
+from .subfolder.a import foo
+
+def bar():
+    return "bar" + foo()
+
+def baz():
+    return __name__
+""",
+        #
+        # a deeper subfolder importing from the parent subfolder
+        #
+        f"{conf_dir}/apps/myapp/subfolder/deep/c.py": """
+from ..a import foo
+from .. import b
+
+def qux():
+    return foo() + b.bar()
+""",
+    }
+
+    mock_open = MockOpen()
+    for key, value in file_contents.items():
+        mock_open[key].read_data = value
+
+    def isfile_side_effect(arg):
+        return arg in file_contents
+
+    def glob_side_effect(path, recursive=None, root_dir=None, dir_fd=None, include_hidden=False):
+        result = []
+        path_re = path.replace("*", "[^/]*").replace(".", "\\.")
+        path_re = path_re.replace("[^/]*[^/]*/", ".*")
+        for this_path in file_contents:
+            if re.match(path_re, this_path):
+                result.append(this_path)
+        return result
+
+    conf = {"apps": {"myapp": {}}}
+
+    def file_patches(mtime):
+        stack = ExitStack()
+        for this_patch in (
+            patch("custom_components.pyscript.os.path.isdir", return_value=True),
+            patch("custom_components.pyscript.glob.iglob", side_effect=glob_side_effect),
+            patch("custom_components.pyscript.global_ctx.open", mock_open),
+            patch("custom_components.pyscript.open", mock_open),
+            patch("homeassistant.config.load_yaml_config_file", return_value={"pyscript": conf}),
+            patch("custom_components.pyscript.watchdog_start", return_value=None),
+            patch("custom_components.pyscript.os.path.getmtime", return_value=mtime),
+            patch("custom_components.pyscript.global_ctx.os.path.getmtime", return_value=mtime),
+            patch("custom_components.pyscript.global_ctx.os_path_isfile", side_effect=isfile_side_effect),
+        ):
+            stack.enter_context(this_patch)
+        return stack
+
+    with file_patches(1000):
+        assert await async_setup_component(hass, "pyscript", {DOMAIN: conf})
+
+    await hass.async_start()
+    await hass.async_block_till_done()
+
+    expected = {"done": ["foo", "barfoo", "myapp.subfolder.b", "foobarfoo"]}
+
+    def ctx_names():
+        return sorted(name for name, _ in GlobalContextMgr.items())
+
+    assert hass.services.has_service("pyscript", "myapp_func")
+    ret = await hass.services.async_call("pyscript", "myapp_func", {}, return_response=True, blocking=True)
+    assert ret == expected
+
+    #
+    # no phantom contexts like apps.myapp.subfolder.b.a, so a reload keeps everything
+    #
+    names = ctx_names()
+    assert names == [
+        "apps.myapp",
+        "apps.myapp.subfolder.a",
+        "apps.myapp.subfolder.b",
+        "apps.myapp.subfolder.deep.c",
+    ]
+
+    with file_patches(2000):
+        await hass.services.async_call("pyscript", "reload", {}, blocking=True)
+    await hass.async_block_till_done()
+
+    assert ctx_names() == names
+    assert "using the old rules" in caplog.text
+    assert hass.services.has_service("pyscript", "myapp_func")
+    ret = await hass.services.async_call("pyscript", "myapp_func", {}, return_response=True, blocking=True)
+    assert ret == expected

…arning

Scripts written for the old rules import a subfolder module's siblings
relative to the folder of the package that imported it (from .subfolder.a
import foo inside subfolder/b.py). Resolve relative imports with the new
rules first, and only if no file is found, retry from the base the old
rules used, which each module's context now keeps. When that finds the
file, log a warning that the old form is deprecated and will be removed
in a future release, with the import written the new way.

Name each module's context after its file's location, so both routes
load the same context and the module's __name__ matches, and no
phantom contexts are created. Contexts for imports from packages keep
the names they had before; only the phantom names of imports made from
plain modules change.

Signed-off-by: Sean Stoube <31805425+HoneyBearTech@users.noreply.github.com>
@HoneyBearTech

Copy link
Copy Markdown
Contributor Author

Thanks @craigbarratt for the prototype! I've added the fallback based on it. Relative imports use the new rules first. Only if no file is found do they retry from the old base, which each context keeps, and then a warning shows the import written the new way (e.g. ... so write it as .a). Context names come from the file's location. While testing it against master, I made three changes to the prototype:

A module found by the old rules gets the name of the context that was found. Before, it got e.g. myapp.sub.sub.a.
A plain module's old base is the folder its import started from, after any .. steps, as on master. With the importer's own base, an app that loads on master failed to load: sub/init.py doing from ..lib.m import f, where lib/m.py does from .lib.n import g.
A module's new-rules base depends only on its file's location. So a module found through the fallback can already use the new form for its own imports, which helps half-migrated apps.
I also compared context names, name and logger names against master, on a tree with 14 apps plus modules/ and scripts/ packages. Everything that works today keeps its name, and only the phantom names from imports inside plain modules change. Your legacy test is in, plus test_relative_import_context_names. The full suite, pre-commit and the docs build pass on 3.14, also merged onto current master. I've updated the description.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Relative import not working as intended

2 participants