You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
It is currently not possible to publish the Ballista crates to crates.io from a DataFusion source release because Ballista has dependencies based on relative path only. For example, the ballista-client crate has this dependency:
datafusion = { path = "../../../datafusion" }
We need to update it to include the version number as well:
datafusion = { path = "../../../datafusion", version = "4.0.0-SNAPSHOT" }
We will also need to update the release scripts to update these version numbers as part of the release process.
Describe the solution you'd like
Add version to each relative dependency
Implement a release script to update these versions. In the monorepo we used a Python script with regex search and replace. We may want to consider a different solution now? Perhaps cargo has plugins to help with changing dependency versions?
At some point, we should probably figure out a more regular release cadence for DataFusion (right now, my project is just using the head directly from git, but that isn't appropriate for ballista).
Do you have plans about what you would like Ballista to be? I.e. if we release Ballista as a binary (e.g. in artifactory, or in a docker image), we do not need DataFusion released, as we can point to a github hash. The limitation emerges when placing it in crates.io.
I think that the same applies for other consumers of DataFusion. For example, the Python bindings do not require DataFusion in crate.io: we can build the wheels and ship them to pypi.
Regardless, I agree with @alamb with the cadence, as some consumers may want to ship libraries built on top of datafusion.
@alamb , I am curious: do you need datafusion released for iox? I imagine that if you ship it as a binary or docker, you can point to the hash, right?
And I don't think we expect anyone to use IOx as a Rust library so I don't think it is likely we would publish it to crates.io -- anything that we think is useful / reusable to others we are trying to contribute back to arrow or datafusion
Activity
[-]Update Ballista crate dependencies to include version numbers[/-][+]Prepare Ballista crates for publishing[/+]andygrove commentedon Jun 5, 2021
@alamb @jorgecarleitao fyi would be good to get your input on this
alamb commentedon Jun 5, 2021
Sounds reasonable to me.
At some point, we should probably figure out a more regular release cadence for DataFusion (right now, my project is just using the head directly from git, but that isn't appropriate for ballista).
jorgecarleitao commentedon Jun 7, 2021
Do you have plans about what you would like Ballista to be? I.e. if we release Ballista as a binary (e.g. in artifactory, or in a docker image), we do not need DataFusion released, as we can point to a github hash. The limitation emerges when placing it in crates.io.
I think that the same applies for other consumers of DataFusion. For example, the Python bindings do not require DataFusion in crate.io: we can build the wheels and ship them to pypi.
Regardless, I agree with @alamb with the cadence, as some consumers may want to ship libraries built on top of datafusion.
@alamb , I am curious: do you need datafusion released for iox? I imagine that if you ship it as a binary or docker, you can point to the hash, right?
alamb commentedon Jun 8, 2021
@jorgecarleitao -- The short answer is no -- we use a git hash in IOx for datafusion
And I don't think we expect anyone to use IOx as a Rust library so I don't think it is likely we would publish it to crates.io -- anything that we think is useful / reusable to others we are trying to contribute back to arrow or datafusion
andygrove commentedon Aug 11, 2021
@houqp I'm not sure if this issue is still valid or not?
houqp commentedon Aug 12, 2021
@andygrove i think you already fixed this one :)
feat: expose PyWindowFrame (apache#509)