Repository navigation
Splitting out collectors #183
Description
I'm going to start by laying out the way collectors work now:
- collectors are bundled inside this https://github.com/python-diamond/Diamond repo, but are not installed onto a
PYTHONPATHanywhere. Nor are they modules in thediamondnamespace. - collectors get installed by
setup.pyinto theircollectors_pathwhich is something like/usr/share/diamond/collectors/on Debian/Ubuntu. See: https://github.com/python-diamond/Diamond/blob/master/setup.py and https://github.com/python-diamond/Diamond/blob/master/src/diamond/utils/classes.py#L100 - This is relatively similar to how Salt deals with extensions as well which also poses similar downsides
- Works well as a system package and a deb package because of the
setup.pymagic
Pros:
- Diamond collectors are just a self contained directory that gets shoved onto the
collectors_pathwithout the need to be "installed". - This nicely facilitates allowing other deb packages to install collectors into Diamond. (We do this at Disqus. See https://github.com/disqus/diamond-collectors)
- Makes system packaging pretty easy
Cons:
- They are all packaged into this one monolithic repo.
- We can't version individual collectors and maintain the outside of the core.
- Not very pythonic (they're not real modules or installable by
pip) - They require their own dependencies typically, which
Diamonddoes not depend on, and there's no way to message this without reading the source of the module. See previous point.
Requirements:
- Split out non-core collectors (TBD what "core" vs "non-core" will actually be)
- Provide a mostly batteries included experience for the 90% use case, meaning most stuff someone wants, they just get by default.
- Facilitate different versioning and iteration cycles per collector independent of the core, and the core's responsible for providing a stable/reliable API through proper versioning.
- Must support installing core and modules through normal python packaging (
pip install diamond diamond-collector-foobar) - Must support installing via deb packages (
apt-get install diamond diamond-collector-foobar) - A way to automatically build deb packages and host them in bintray or somewhere else as module components.
Proposal (WIP):
- Handle the current process today (just crawl a directory of files)
- Handle new style as plugins.
- Plugins get installed through pip. ala
pip install diamond-collector-something-awesomewhich lists it's own dependencies - Utilize
setuptoolsandentry_pointsand declare adiamond.collectorsnamespace that plugins get registered into. Initial support for collectors being installed into an entry_point #263 - Provide a template project that includes a setup.py/Makefile so the same repo can easily be packaged up properly for pip, and gets dropped into
collectors_pathfor system packaging.
Thoughts?
/cc @josegonzalez
Activity
[-]RFC: Splitting out collectors[/-][+]Splitting out collectors[/+]josegonzalez commentedon May 14, 2015
Sounds reasonable to me. As an alternative, here is how python-beaver handled it. Your way looks neater though. Want to come up with a PR for this?
redref commentedon Jun 1, 2016
👍
May be a good way to re-launch "active" development on this project.
Seems quite in pause. Anything to help ?
mattrobenolt commentedon Jun 1, 2016
afaik all of the necessary internal work has been done and collectors can be split out whenever.
I created an example a while back: https://github.com/python-diamond/diamond-redis which also includes tests, etc. Also includes a pytest plugin: https://github.com/python-diamond/pytest-diamond
shortdudey123 commentedon Jun 1, 2016
I should be able to help with this as well. Splitting things out will help when it comes to running different components for testing in a way that doesn't become bloated. (i.e. my work in #467)
redref commentedon Jun 9, 2016
Problem may not be developers as work to do is not so big, but consistency and release process. It would be tricky to get everything done in a realistic timing so far. I think we need to get somebody (I know it's not simple) to actively review / comment / merge on this subject.
Beside, I thought about collectors groups (suggest)
jantman commentedon Jun 9, 2017
I've been an on-and-off user of Diamond for quite a few years now. When I first started using it (and the two times I've come back to it since then), I was horribly confused by how the plugins are packaged. While it does facilitate easier distribution of plugins, managing dependencies is unintuitive and it's very un-pythonic.
I'd like to express my 1000% support for this proposal. Keeping compatibility with the current method is great for people who already have deployment/management tooling built around it, but supporting separate plugin packages discovered via entrypoints is definitely what I'd expect from Python software.
For what it's worth, my opinion on "core" collectors vs separate packages is:
Just some thoughts.