Skip to content

[proposal] Consider consolidating kiota python libraries into a single repo #89

Closed
Closed
[proposal] Consider consolidating kiota python libraries into a single repo#89

Description

@andreaTP
Andrea Peruffo (andreaTP)
opened on Jun 28, 2023

Currently, the various repos are using(or not) a variety of tools and different setup (at least I can count 2 separate styles) i.e.:

  • pipenv in kiota-http-python, kiota-serialization-json-python
  • plain global python installation in this repo and e.g. , kiota-serialization-text-python

Those different setups are causing some friction during development, where I always have to look up the CI setup to understand which command to use/where.
Also, the different setups, easily run out of sync.

There are various ways to tackle those issues i.e.:

  • use reusable GH Actions across the different repos -> this will force to have a consistent setup
  • have stronger checks and guidelines to contribute
  • [last but not least] consolidate in a single repo for the language

I'm (no surprise 😛) in favor of the latter solution and I propose switching to a language-specific monorepo (as we did for kiota-java).
Looking up on the internet I cannot find a "good story" for Python tools natively supporting multi-projects builds, but, still, with a little duplication we can have a viable setup as described here.
I do believe that this would be already an improvement with respect to the current setup, where we will share CI pipelines, documentation, contributing guidelines, and more.

Let me know your thoughts!

Activity

  1. samwelkanda commented on Jun 30, 2023

    @samwelkanda
    Samwel K. (samwelkanda)
    on Jun 30, 2023
    Contributor

    Hi Andrea Peruffo (@andreaTP) , thank you for the suggestion.

    We decided to remove the dependency on pipenv in favor of a base requirements file to allow developers to use a dependency manager of their choice. As you know all the popular package managers support installation from a requirements file. I have already harmonized this across the repos, so you'll no longer face the inconsistencies.

    I also do believe there's a lot to gain in having a mono-repo structure, especially in terms of accelerating the development workflow. I'm happy to keep this as a potential work item, but for Post-GA of the SDK.

  2. andreaTP commented on Jun 30, 2023

    @andreaTP
    Andrea Peruffo (andreaTP)
    on Jun 30, 2023
    ContributorAuthor

    Thanks a lot for your answer Samwel K. (@samwelkanda) !
    Let me first say that I'm not an expert in Python development, so, please, feel free to apply some grain of salt to my suggestions 😇

    We decided to remove the dependency on pipenv in favor of a base requirements file to allow developers to use a dependency manager of their choice.

    This sounds unexpected to me, I can share my experience as a first-time contributor to those projects:

    • I submitted a fix in the kiota-serialization-json-python repo and it was easy for me to get started with pipenv and every command copy-pasted from the CI workflow just worked on my machine
    • I was debugging something in this repo and I have had multiple issues with versions/incompatibilities, I rescued to use pipenv to get things working

    I have already harmonized this across the repos, so you'll no longer face the inconsistencies.

    I'm not sure if this is accurate, the examples that I mentioned in the first comment are still valid.

    I also do believe there's a lot to gain in having a mono-repo structure, especially in terms of accelerating the development workflow.

    I strongly agree with this sentiment 🙂

    I'm happy to keep this as a potential work item, but for Post-GA of the SDK.

    I profoundly respect your standpoint as a maintainer of those libraries, but, as an external contributor, I would say that the sooner the better.
    My experience with kiota-java dramatically improved after some refactorings in the repository and this enabled me to get fixes much quicker and reduce the friction every time I have to debug something.

    From a pragmatic POV I can notice that Python is still lacking kiota-serialization-form-python, doing it means copy-pasting a lot of stuff(CI, etc. etc.) that would eventually(and naturally) diverge from the rest of the repos.
    With the same effort driven toward a monorepo we can probably end up re-using a lot of things and cutting the development time over a mid-term instead of increasing the debt.

    If we manage to agree on the direction I can spare a few cycles to spear-head the approach and contribute the proposed changes 🙂 .

    -side note - lately, I have been playing with Poetry, and I'm quite happy with it as the experience gets closer to using build tools in other languages.

    Would also be happy to hear from Vincent Biret (@baywet) 🙂

  3. baywet commented on Jul 1, 2023

    @baywet
    Vincent Biret (baywet)
    on Jul 1, 2023
    Member

    Thanks as always for starting a great conversation.

    I don't have a strong opinion either way. The recommendation I was circulating with language teams was: choose what makes the most sense for the ecosystem of the language.

    Things to consider are subtle details like:

    • can we control different versions with the packages? If not, is bumping everything every single time an issue?
    • is the GitHub dependency graph able to track multiple packages on the same repository for that language?
    • what's the impact on maintaining a changelog?
    • can the tasks for GHA publish multiple packages at once without having to require complex scripts or sequencing?
    • does the package manager read anything from the repository? (E.g. issues, pull requests...)
    • are we able to use tasks that build packages at once locally?

    Etc...
    Ideally this is something that we need to figure out before any language becomes stable to avoid introducing changes later on.

  4. andreaTP commented on Jul 3, 2023

    @andreaTP
    Andrea Peruffo (andreaTP)
    on Jul 3, 2023
    ContributorAuthor

    Thanks for chipping in Vincent Biret (@baywet) !

    From the great list of tradeoffs you provided the only one that might be slightly compromised by a monorepo approach is:

    can the tasks for GHA publish multiple packages at once without having to require complex scripts or sequencing?

    For Python, I think that each module has to be published independently, this can be automated with GH Actions and a matrix for the working_dir as already done elsewhere.

    In general, there is a lot of grey area ... in Java support for monorepos is built in, while in Go you have to really fight against the ecosystem to avoid separating everything in multiple repos.

    The final decision is on you 🙂 , please let me know if you think you would benefit from a little help by me!

  5. Vincent Biret (baywet)
    added this to Kiotaon Feb 21, 2024
  6. github-project-automation
    moved this to Todo in Kiotaon Feb 21, 2024

  7. andrueastman commented on Aug 6, 2024

    @andrueastman
    Andrew Omondi (andrueastman)
    on Aug 6, 2024
    Contributor

    Closing this for now to be tracked via #302 which unlocks the Bundle work for python.

  8. Andrew Omondi (andrueastman)
    closed this as completedon Aug 6, 2024
  9. github-project-automation
    moved this from Todo 📃 to Done ✔️ in Kiotaon Aug 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    • Kiota
      Status
      Done ✔️

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions