Improved mono repo support

We talked to a lot of folks at PyCon that were complaining about our support for mono repos. This issue is to track feedback about this experience. Current hypothesis is that our multiroot support isn't great, so solving that might solve the issue for mono repos.

#21256

Just as feedback, I personally have been using multi-root satisfactorily. A couple of bugs here and there in 1.5y of usage (one was about Python extension doing crazy stuff and hogging the machine for no apparent reason. Recreating the multi-root workspace fixed the problems) and now this problem with the Black extension that you are citing here above (but I believe it's not a problem with Python extension, that works correctly and provides autocompletion and everything correctly).

Happy to provide feedback, I use this at work on a daily basis :)

Ruben Di Battista (@rdbisme)

Ruben Di Battista (@rdbisme) For our use case multi-root workspace don't work since we have tooling in our monorepo (Pants) which requires top-level files to be present in the root folder, to define "metadata" about how each project in the monorepo should function (e.g. specifying python virtual envs, building and running tests selectively). So our monorepo is not just a bunch of folders opened in a single VSCode window. If it was then there would be no issue and we'd gladly we multi-root workspaces.

See my post from which this issue is linked microsoft/vscode#181845

Yegor Ius (@yegorski)

Recently, I've published an Nx (monorepo tool) plugin @nxlv/python to enable python poetry projects in Nx workspaces, and the issue is very similar as Yegor Ius (@yegorski) described in his post microsoft/vscode#181845.

Each project has a virtual environment. The ideal scenario is the microsoft/vscode-python extension switch the virtual environment based on the sub-folder virtual environment without manually selecting the interpreter or changing the settings.json file.

Different from Yegor Ius (@yegorski) use case, I can use project.code-workspace and add a .vscode/settings.json for each Python project, but that's not the ideal scenario, and might be not a good configuration if you have a Nx workspace with Python and Node.js projects in it.

I agree with Yegor Ius (@yegorski) post, if the extension supports the .vscode/settings.json on each sub-folder would be awesome.

Lucas Vieira (@lucasvieirasilva)

We are experiencing similar issues as Lucas Vieira (@lucasvieirasilva) and Yegor Ius (@yegorski). It boils down to the following in our case:

I need to programatically set up virtual-environments and assign workspace interpreters for users to simplify onboarding. As far as I know, there is no API exposed to update the Workspace Python Interpreter setting. Manually updating the SQLite-database in which these settings are stored feels does not seem like a good solution, given that everything is stored under the ms-python key as a JSON blob. If it were more cleanly separated I would feel more comfortable about doing this without fear of corrupting the rest of the user's state. However, an optimal solution would not require altering this database at all.
If we create a separate .vscode/settings.json in each workspace-folder we experience issues with Intellisense not picking up the correct interpreter without closing and re-opening files. The button used to select an interpreter will say that no interpreter is currently selected until I close and re-open the file in question. Compared to setting the Workspace interpreter(s) we also get terrible performance with this solution. On a medium-to-large sized codebase we have to wait almost a full minute before we can start following references etc. I have to repeat this waiting time for each new project I open files in. It would be more clean, and more maintainable if the python-extension supported assigning interpreters for sub-folders via a single settings.json file at the root of the project.
We, too, have additional meta-files and tools at the top-level and need these to be visible in the monorepo. As a workaround I have created the following structure:

º base-folder
º project-folder-1
º project-folder-n
º libs-folder-1
º libs-folder-n

This works but the structure of base-folder is similar to this:

base_folder/
    meta_file_1
    meta_file_n
 
    libs/
       lib-1/
       lib-n/
    projects/
       project-1/
       project-n/

Meaning that all the project and library folders have to be repeated in the Workspace below the base-folder. Opening files via Ctrl + P will open the actual folder, not the directory in the base-folder. Therefore I can't add these and just keep them closed at all times to prevent clutter. There are a few solutions that could help alleviate these issues:

Make it possible to add additional folders to workspaces, set their interpreter and then hide them from the workspace. From my own testing it appears that the interpreter changes are still picked up correctly when navigating to individual projects inside the base-folder in the workspace. However, I have not been able to hide the folders added via the "Add Folder to Workspace..." to remove clutter, even using third-party extensions.
Make it possible to assign a folder as a workspace, giving us the benefit of selecting a separate interpreter etc. while keeping the directory structure otherwise intact.
Allow users to create meta-structures inside a workspace. If the solution above is not feasible, our experience would also be improved if we could create a workspace similar to:

º base-folder
º projects/ (this is a meta-folder, for structure only)
    º project-1
    º project-n
º libs/ (this is a meta-folder, for structure only)
    º lib-1
    º lib-n

Ruben Di Battista (@rdbisme)

I just wanted to start by thanking you all so much for taking the time to provide more context and feedback around your experiences with mono repos, it's super appreciated!

I don't often work with mono repos so I'm trying to get a better understanding of the typical workflow involved. Please correct me if I'm wrong, but from what I'm understanding there are two common approaches to how folks have been managing mono repos:

Using one shared virtual environment
Using individual virtual environments per project/subfolder

If a mono repo is set up in a way that one can use a shared virtual environment (i.e. there's no dependency conflicts between the projects in the mono repo), my understanding is that the current experience is good enough as one can simply open the root/base folder in VS Code, create a virtual environment on the project root and install all the dependencies on that same venv. The extension can automatically activate that venv and all actions can be performed inside it.

When there's a mono repo with individual virtual environments per project (because e.g. their dependencies can conflict with each one), but there's nothing on the root directory that is relevant/needed for all projects in the mono repo, then multi-root workspaces works well enough (except for the occasional bugs as called out by Ruben Di Battista (@rdbisme) 😅). The extension will switch the selected interpreter to match the corresponding virtual environment inside the workspace where the file you open lives.

The problem is really when the root folder has metadata or files that are needed for the entire project, and each one of the projects have their own virtual environment. Ideally, you want to stay at the project root level and perform any/all actions from there, so opening the projects as a workspace is not ideal. One could do as Morten Hauge (@Moortiii) suggested and open the root folder as a workspace and then add the remaining project folders as their own workspaces. There are a couple of issues with that:

Files and folders end up duplicated in the file explorer
- One workaround could be to add a .vscode/settings.json file under the root folder workspace setting "files.exclude" to hide the project folders (which would be the opposite of what Morten Hauge (@Moortiii) wants, I guess)
There's a significant perf issue once you have more than a couple of workspaces open (and I think it might be because Pylance launches a server per workspace)
The Python extension is not automatically picking up the virtual environment in the each project workspace
- The workaround here would be to do as Lucas Vieira (@lucasvieirasilva) suggested here, but I agree that's not ideal

Are there any other issues that I'm missing?

On the Pylance/Pyright side, I filed something related around walking upwards to find the nearest configuration for the analyses: microsoft/pylance-release#5564

Otherwise, I think the scenarios described above are similar issues which I ran into where it actually pushed me towards using one Python project/dependency manager over another given its ability to produce a top-level workspace .venv - something I mind less than having a top-level pyproject.toml.

Still, I think that because the .venv directory is not co-located with the pyproject.toml, the Python extension might not immediately suggest using it upon creation.

Robert Dargavel Smith (@teticio)

It would be nice to be able to explore the tests on the sub-module I'm working on similar to how Robert Dargavel Smith (@teticio)'s python-envy works. By updating "python.testing.pytestArgs": ["..."] to the root of the current module

Alita Moore (@alita-moore)

Alita Moore (@alita-moore) That's an interesting idea. Would it make sense to add (as an option) to python-envy? Would it just set python.testing.pytestArgs to the same directory where it activates the .venv? So the structure of your monorepo would be having a tests directory and a .venv directory in every sub-module directory? Let me know and I can make the changes.

Robert Dargavel Smith (@teticio)

Robert Dargavel Smith (@teticio) yeah i think it makes sense to add to python-envy (awesome package, btw, thank you very much). Yeah it would set the python.testing.pytestArgs to the root directory of where .venv is so for example if you had a project structure of

-- folder
--- .venv
--- src
---- something.py

Then when you focus something.py it would update the python.testing.pytestArgs to reflect the path to the folder (in this case just folder. Where perhaps it initially started out as a . or the root directory of vscode.

Note that I actually use individual test directories in my code / at the same level of the code it's testing, so I think having it not rely on a top-level test folder would be great. Thank you!

hey all, just wanted to give an update that we are working on a prototype that would allow a better experience for mono repos, such as allowing interpreters to be associated to folders and individual files, without relying on multi-root workspaces. No ETAs but hopefully this will be available soon :) But I also wanted to link a somewhat related issue from upstream VS Code here: microsoft/vscode#32693

Just noticed that the notebook env selection now shows the paths so they can be distinguished per submodule. Huge thanks!

When using pyenvy if you switch between multiple interpreters quickly it sometimes causes the cpu to throttle.

We've recently started trying a monorepo approach in vscode with mixed results. We're using a rye-style approach with a single venv that the packages share, each installed into it as editable.

If we use a non-multiroot workspace, i.e. just opening the monorepo at the root directory and letting the tooling work from there, most things work very well. Refactoring, import suggestions, and all of that work as well as we could hope. The primary problem we run into in this configuration is the test explorer. Some of our test modules have the same name, e.g. test_foo.py shows up in the test suites for multiple packages, and this causes pytest to complain. There's a tension, then, between a) treating the entire body of code as one for some operations like refactoring, and b) treating the packages independently for others, like test discovery and running. Ideally, the test explorer would be able to handle several independent test suites, but I haven't found a way to do that yet. (And while we could rename the conflicting test modules, this seems like putting the onus on the wrong party to fix the problem.)

The other approach we've tried is using multi-root workspaces, with each of our packages added as a folder to the workspace. In this configuration, the testing problem is fixed; the test explorer treats each package's tests separately, so naming conflicts go away. Unfortunately, we lose all of the benefits of a single language server handling all of the code. Things like refactoring and import suggestions simply don't work across packages. My understanding is that much of this is due to the fact that there's a separate language server for each "folder", and they don't cooperate in any way.

Anyhow, hopefully that's another useful data point as you work on the tooling.

If we use a non-multiroot workspace, i.e. just opening the monorepo at the root directory and letting the tooling work from there, most things work very well. Refactoring, import suggestions, and all of that work as well as we could hope. The primary problem we run into in this configuration is the test explorer.

Could you ellaborate a bit on how you make this setup work? I tried the same i.e. having a root folder that I opened in VSCode and inside that folder I have a few folders containing different Python packages, using a single venv and editable dependencies. In my case no test cases show up whatsoever.

Could you ellaborate a bit on how you make this setup work?

The main trick is to tell pytest where to find the tests. Following the rye approach, have a top-level pyproject.toml with something like this:

[tool.pytest.ini_options]
addopts = ["--import-mode=importlib"]
testpaths = [
    "packages/app/tests",
    "packages/domain/tests",
    "packages/infrastructure/tests",
    "packages/web/tests",
]

Each of 'app', 'domain', 'infrastructure', and 'web' are Python packages with their own pyproject.toml and their tests in the associated 'tests' directories. This seems to tell vscode where everything is.

Incidentally, I mentioned that my main problem with this approach was test name collision. The --import-mode=importlib bit makes this problem go away, and is the recommended way of importing tests in any event. So this setup is working very well for us.

Austin Bingham (@abingham)

Austin Bingham (@abingham) I've found that a multi-root monorepo using poetry's relative packages + pylance retains refactoring and type checking across projects, have you tried this? see microsoft/pylance-release#5995 for a link to a codesandbox that demonstrates this workflow.

Have you tried this? The biggest drawback I've found using it is that pylance starts to bug out once you get more than like 3-5k files in the monorepo and pylance switching between project interpreters can be slow and sometimes bug prone like in the case of a jupyter notebook as discussed in the linked issue.

Austin Bingham (@abingham)

Austin Bingham (@abingham) I've found that a multi-root monorepo using poetry's relative packages + pylance retains refactoring and type checking across projects, have you tried this? see microsoft/pylance-release#5995 for a link to a codesandbox that demonstrates this workflow.

Could you elaborate on this? We also have a multi-root mono repo and e.g. refactoring doesn't work across folders. Did you mean this codesandbox link?

Our setup looks like follows (with each top-level directory as a VSCode multi-root folder):

pkg_a/
  pyproject.toml
pkg_b/
  pyproject.toml

We use a "relative" dependency e.g. {path = "../pkg_b", develop = true} in pkg_a/pyproject.toml. I've tried to have a virtualenv at the top-level or inside pkg_a. What's your setup? Do you have some kind of top-level pyproject.toml?

Yes, that's the correct CodeSandbox link.

For each project, we use a separate pyproject.toml file, following the same relative dependency structure you mentioned. We do not have a single top-level pyproject.toml. Are you using Pylance?

In the CodeSandbox you linked, refactors will propagate across projects if you open it in VSCode, except for Jupyter notebooks.

Please note that we use Python Envy to switch the virtual environment to the one associated with the currently open file.

For each project, we use a separate pyproject.toml file, following the same relative dependency structure you mentioned. We do not have a single top-level pyproject.toml. Are you using Pylance?

I am using Pylance. My issue was using a venv in each package directory. I guess that made VSCode switch venv when I opened e.g. pkg_b and that venv didn't include pkg_a and thus cross-package stuff didn't work (e.g. I could find all references of identifiers in pkg_b).

It seems that the best approach for a multi-root monorepo is to have to have as few virtual envs as possible (e.g. only for the "top-level" stuff, which depends on other packages). We can't have just one, because of dependency conflicts (that's why we have multiple packages to begin with), but fewer definitely seems better.

My large workspace has projects in many different languages, and I only want to enable python analysis for specific projects/subfolders. Is that use case supported today?

When I open my workspace, I get this warning:

Enumeration of workspace source files is taking a long time. Consider opening a sub-folder instead.

What I want is something like this:

"python.analysis.exclude": [
    "**" // exclude everything by default
],
"python.analysis.include": [
    // override exclude for specific projects that are written in python
    "tools/python-project-abc/**",
    "service/python-service-xyz/**"
],

That gets rid of the warning mesasge, but the analysis is broken. I get certain syntax errors (e.g., creating a function with de instead of def) but the type checking and other analysis no longer works. Presumably the exclude takes precedence over the include, so I need a different strategy.

I read through the Mono Repo Set-up Guide and Opening Large Workspaces in VS Code. Either it's missing the info that I'm looking for, or (quite plausibly) I failed to understand it.

Navigation Menu

Repository navigation

Improved mono repo support #21204

22 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Improved mono repo support #21204

Description

Activity

luabud commented on May 10, 2023

rdbisme commented on May 16, 2023

yegorski commented on May 18, 2023

lucasvieirasilva commented on May 30, 2023

Moortiii commented on Jun 20, 2023

luabud commented on Jun 22, 2023

22 remaining items

DanielRosenwasser commented on Mar 4, 2024

alita-moore commented on Mar 11, 2024

teticio commented on Mar 15, 2024

alita-moore commented on Mar 16, 2024

luabud commented on May 6, 2024

leddy231 commented on May 13, 2024

alita-moore commented on May 16, 2024

abingham commented on Aug 1, 2024

tibbe commented on Aug 13, 2024

abingham commented on Aug 13, 2024

alita-moore commented on Aug 13, 2024

tibbe commented on Aug 13, 2024

alita-moore commented on Aug 13, 2024

tibbe commented on Aug 14, 2024

davidthing commented on Apr 2, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions