Repository navigation
Fix relative imports inside an app's subfolders (#726) - #911
HoneyBearTech wants to merge 2 commits into
Conversation
A module imported from a subfolder (from .subfolder.b import x) got its parent folder as the base for its own relative imports, so a sibling import (from .a import foo) failed, and imports from plain modules were registered under phantom contexts such as apps.myapp.subfolder.b.a that were unloaded on the next reload. Use the module's own folder as the base and resolve a plain module's relative imports against its package. Fixes custom-components#726. Signed-off-by: Sean Stoube <31805425+HoneyBearTech@users.noreply.github.com>
craigbarratt
left a comment
There was a problem hiding this comment.
Thanks @HoneyBearTech, this is the right fix and matches Python.
To avoid breaking scripts that use the old from .subfolder.a workaround, could you add a fallback?
- Resolve with the new rules first.
- Only if no file is found, retry with the old base, the importing package's folder. Each context would need to keep that base.
- When the fallback is used, log a warning that the old form is deprecated and will be removed in a future release, and say how to write the import instead.
If the context name is derived from the file's path (apps/myapp/subfolder/a.py → apps.myapp.subfolder.a) rather than from the importer's name, both routes load the same context, so the phantom contexts can't come back. A copy of your test using the old import in subfolder/b.py would cover it, including the warning.
I tried this locally on top of your branch; it's about 40 lines in module_import, and the full suite passes. Feel free to use or adapt it:
Prototype diff
diff --git a/custom_components/pyscript/global_ctx.py b/custom_components/pyscript/global_ctx.py
index 0d4ac5b..ed77a72 100644
--- a/custom_components/pyscript/global_ctx.py
+++ b/custom_components/pyscript/global_ctx.py
@@ -45,6 +45,8 @@ class GlobalContext:
self.auto_start: bool = False
self.module: ModuleType | None = None
self.rel_import_path: str = rel_import_path
+ # the base the old (pre-2.2) rules used for relative imports, kept as a fallback
+ self.legacy_rel_import_path: str | None = rel_import_path
self.source: str = source
self.file_path: str | None = None
self.mtime: float = mtime
@@ -160,6 +162,7 @@ class GlobalContext:
pyscript_dir = Function.hass.config.path(FOLDER)
module_path = module_name.replace(".", "/")
file_paths = []
+ import_name = "." * import_level + module_name
def find_first_file(file_paths: list[list[str]]) -> list[str] | None:
for ctx_name, path, rel_path in file_paths:
@@ -171,28 +174,37 @@ class GlobalContext:
#
# first build a list of potential import files
#
+ legacy_file_paths = []
if import_level > 0:
if self.rel_import_path is None:
raise ImportError("attempted relative import with no known parent package")
- path = self.rel_import_path
- if path.endswith("/__init__"):
- path = os.path.dirname(path)
- ctx_name = self.name
- if os.path.basename(self.file_path or "") != "__init__.py" and "." in ctx_name:
- # a plain module is relative to its parent package, so drop its own name
- ctx_name = ctx_name[0 : ctx_name.rfind(".")]
- for _ in range(import_level - 1):
- path = os.path.dirname(path)
- idx = ctx_name.rfind(".")
- if path.find("/") < 0 or idx < 0:
- raise ImportError("attempted relative import above parent package")
- ctx_name = ctx_name[0:idx]
- ctx_name += f".{module_name}"
- module_info = [ctx_name, f"{path}/{module_path}.py", os.path.dirname(f"{path}/{module_path}.py")]
- path += f"/{module_path}"
- file_paths.append([ctx_name, f"{path}/__init__.py", path])
- file_paths.append(module_info)
- module_name = ctx_name[ctx_name.find(".") + 1 :]
+
+ def base_path(path):
+ if path.endswith("/__init__"):
+ path = os.path.dirname(path)
+ for _ in range(import_level - 1):
+ path = os.path.dirname(path)
+ if path.find("/") < 0:
+ raise ImportError("attempted relative import above parent package")
+ return path
+
+ def candidates(path, plain_rel_path):
+ # the context name follows the file's location, so each file has one context
+ ctx_name = f"{path}/{module_path}".replace("/", ".")
+ return [
+ [ctx_name, f"{path}/{module_path}/__init__.py", f"{path}/{module_path}"],
+ [ctx_name, f"{path}/{module_path}.py", plain_rel_path],
+ ]
+
+ path = base_path(self.rel_import_path)
+ file_paths = candidates(path, os.path.dirname(f"{path}/{module_path}.py"))
+ try:
+ legacy_path = base_path(self.legacy_rel_import_path or self.rel_import_path)
+ except ImportError:
+ legacy_path = path
+ if legacy_path != path:
+ legacy_file_paths = candidates(legacy_path, legacy_path)
+ module_name = file_paths[0][0][file_paths[0][0].find(".") + 1 :]
else:
if self.rel_import_path is not None and self.rel_import_path.startswith("apps/"):
@@ -207,16 +219,38 @@ class GlobalContext:
#
# now see if we have loaded it already
#
- for ctx_name, _, _ in file_paths:
- mod_ctx = self.manager.get(ctx_name)
- if mod_ctx and mod_ctx.module:
- self.imports.add(mod_ctx.get_name())
- return mod_ctx.module
+ def find_loaded(file_paths):
+ for ctx_name, _, _ in file_paths:
+ mod_ctx = self.manager.get(ctx_name)
+ if mod_ctx and mod_ctx.module:
+ self.imports.add(mod_ctx.get_name())
+ return mod_ctx.module
+ return None
+
+ if mod := find_loaded(file_paths):
+ return mod
#
# not loaded already, so try to find and import it
#
file_info = await Function.hass.async_add_executor_job(find_first_file, file_paths)
+ if not file_info and legacy_file_paths:
+ #
+ # fall back to the old rules, where a module's relative imports were relative to
+ # the package that imported it, not to the module's own folder
+ #
+ mod = find_loaded(legacy_file_paths)
+ file_info = mod or await Function.hass.async_add_executor_job(find_first_file, legacy_file_paths)
+ if file_info:
+ _LOGGER.warning(
+ "%s: relative import of %s found %s using the old rules, which will be removed in a "
+ "future release; make the import relative to the module's own folder, as in Python",
+ self.name,
+ import_name,
+ legacy_file_paths[0][0],
+ )
+ if mod:
+ return mod
if not file_info:
return None
@@ -227,6 +261,13 @@ class GlobalContext:
ctx_name, global_sym_table=mod.__dict__, manager=self.manager, rel_import_path=rel_import_path
)
global_ctx.set_auto_start(self.auto_start)
+ if import_level > 0:
+ # a plain module's old-rules base is the importer's, as before
+ global_ctx.legacy_rel_import_path = (
+ rel_import_path
+ if file_path.endswith("/__init__.py")
+ else (self.legacy_rel_import_path or self.rel_import_path).removesuffix("/__init__")
+ )
try:
await self.manager.load_file(global_ctx, file_path)
except Exception:
diff --git a/tests/test_apps_modules.py b/tests/test_apps_modules.py
index 7114f41..faa62b5 100644
--- a/tests/test_apps_modules.py
+++ b/tests/test_apps_modules.py
@@ -329,3 +329,118 @@ def qux():
assert hass.services.has_service("pyscript", "myapp_func")
ret = await hass.services.async_call("pyscript", "myapp_func", {}, return_response=True, blocking=True)
assert ret == expected
+
+
+@pytest.mark.asyncio
+async def test_app_subfolder_legacy_relative_imports(hass, caplog):
+ """Test relative imports between modules in an app's subfolders, from #726."""
+
+ conf_dir = hass.config.path(FOLDER)
+
+ file_contents = {
+ f"{conf_dir}/apps/myapp/__init__.py": """
+from .subfolder.a import foo
+from .subfolder.b import bar, baz
+from .subfolder.deep.c import qux
+
+@service(supports_response="only")
+def myapp_func():
+ return {"done": [foo(), bar(), baz(), qux()]}
+""",
+ f"{conf_dir}/apps/myapp/subfolder/a.py": """
+def foo():
+ return "foo"
+""",
+ #
+ # a plain module in a subfolder importing its sibling
+ #
+ f"{conf_dir}/apps/myapp/subfolder/b.py": """
+from .subfolder.a import foo
+
+def bar():
+ return "bar" + foo()
+
+def baz():
+ return __name__
+""",
+ #
+ # a deeper subfolder importing from the parent subfolder
+ #
+ f"{conf_dir}/apps/myapp/subfolder/deep/c.py": """
+from ..a import foo
+from .. import b
+
+def qux():
+ return foo() + b.bar()
+""",
+ }
+
+ mock_open = MockOpen()
+ for key, value in file_contents.items():
+ mock_open[key].read_data = value
+
+ def isfile_side_effect(arg):
+ return arg in file_contents
+
+ def glob_side_effect(path, recursive=None, root_dir=None, dir_fd=None, include_hidden=False):
+ result = []
+ path_re = path.replace("*", "[^/]*").replace(".", "\\.")
+ path_re = path_re.replace("[^/]*[^/]*/", ".*")
+ for this_path in file_contents:
+ if re.match(path_re, this_path):
+ result.append(this_path)
+ return result
+
+ conf = {"apps": {"myapp": {}}}
+
+ def file_patches(mtime):
+ stack = ExitStack()
+ for this_patch in (
+ patch("custom_components.pyscript.os.path.isdir", return_value=True),
+ patch("custom_components.pyscript.glob.iglob", side_effect=glob_side_effect),
+ patch("custom_components.pyscript.global_ctx.open", mock_open),
+ patch("custom_components.pyscript.open", mock_open),
+ patch("homeassistant.config.load_yaml_config_file", return_value={"pyscript": conf}),
+ patch("custom_components.pyscript.watchdog_start", return_value=None),
+ patch("custom_components.pyscript.os.path.getmtime", return_value=mtime),
+ patch("custom_components.pyscript.global_ctx.os.path.getmtime", return_value=mtime),
+ patch("custom_components.pyscript.global_ctx.os_path_isfile", side_effect=isfile_side_effect),
+ ):
+ stack.enter_context(this_patch)
+ return stack
+
+ with file_patches(1000):
+ assert await async_setup_component(hass, "pyscript", {DOMAIN: conf})
+
+ await hass.async_start()
+ await hass.async_block_till_done()
+
+ expected = {"done": ["foo", "barfoo", "myapp.subfolder.b", "foobarfoo"]}
+
+ def ctx_names():
+ return sorted(name for name, _ in GlobalContextMgr.items())
+
+ assert hass.services.has_service("pyscript", "myapp_func")
+ ret = await hass.services.async_call("pyscript", "myapp_func", {}, return_response=True, blocking=True)
+ assert ret == expected
+
+ #
+ # no phantom contexts like apps.myapp.subfolder.b.a, so a reload keeps everything
+ #
+ names = ctx_names()
+ assert names == [
+ "apps.myapp",
+ "apps.myapp.subfolder.a",
+ "apps.myapp.subfolder.b",
+ "apps.myapp.subfolder.deep.c",
+ ]
+
+ with file_patches(2000):
+ await hass.services.async_call("pyscript", "reload", {}, blocking=True)
+ await hass.async_block_till_done()
+
+ assert ctx_names() == names
+ assert "using the old rules" in caplog.text
+ assert hass.services.has_service("pyscript", "myapp_func")
+ ret = await hass.services.async_call("pyscript", "myapp_func", {}, return_response=True, blocking=True)
+ assert ret == expected…arning Scripts written for the old rules import a subfolder module's siblings relative to the folder of the package that imported it (from .subfolder.a import foo inside subfolder/b.py). Resolve relative imports with the new rules first, and only if no file is found, retry from the base the old rules used, which each module's context now keeps. When that finds the file, log a warning that the old form is deprecated and will be removed in a future release, with the import written the new way. Name each module's context after its file's location, so both routes load the same context and the module's __name__ matches, and no phantom contexts are created. Contexts for imports from packages keep the names they had before; only the phantom names of imports made from plain modules change. Signed-off-by: Sean Stoube <31805425+HoneyBearTech@users.noreply.github.com>
|
Thanks @craigbarratt for the prototype! I've added the fallback based on it. Relative imports use the new rules first. Only if no file is found do they retry from the old base, which each context keeps, and then a warning shows the import written the new way (e.g. ... so write it as .a). Context names come from the file's location. While testing it against master, I made three changes to the prototype: A module found by the old rules gets the name of the context that was found. Before, it got e.g. myapp.sub.sub.a. |
Closes #726
What was broken
With this app layout:
from .a import fooinsubfolder/b.pyfailed, although that's the correct Python. It only worked asfrom .subfolder.a import foo.apps.myapp.subfolder.b.a, which loadeda.pya second time and were unloaded on the next reload.What I changed
In
GlobalContext.module_import:subfolder.b) now gets its own folder as the base for its relative imports, not the importing package's folder;__init__.py) relative imports are now resolved against its package, as in Python, instead of against the module itself.Old form still works, with a warning
Scripts that use the old workaround,
from .subfolder.a import fooinsidesubfolder/b.py, keep working for now. Relative imports are resolved with the new rules first. Only if no file is found are they retried from the base the old rules used, which each module's context keeps. When that finds the file, a warning says the old form is deprecated and will be removed in a future release, and shows the import written the new way (for examplewrite it as .a).Each module's context is now named after its file's location (
apps/myapp/subfolder/a.pyisapps.myapp.subfolder.a), so both routes load the same context and no phantom contexts are created. Imports from packages keep the context names they had before; only the phantom names of imports made from plain modules change.Relative imports from a plain file inside a
modules/package are still not supported, as before.Testing
tests/test_apps_modules.py:test_app_subfolder_relative_imports(the new form),test_app_subfolder_legacy_relative_imports(the old form, with its warning, before and after a reload) andtest_relative_import_context_names(context names,__name__,..imports, the suggested rewrite, and an import neither rule finds):subfolder/deep/c.pydoingfrom ..a import fooandfrom .. import b;__name__and the exact set of contexts;