You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Release a new version of DataFusion to crates.io #771
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
The current version of datafusion on crates.io is 4.0.0 (which among other things doesn't work with the newly released arrow 5.0.0)
Describe the solution you'd like
It would be great if we released a DataFusion 5.0.0 (or some other number if we want to diverge from arrow) -- doing so would likely involve porting the arrow-rs release scripts, from https://github.com/apache/arrow-rs/tree/master/dev/release and then sending it to the dev mailing list for a formal vote
@andygrove@jorgecarleitao I couldn't find the exact commit for the 4.0.0 version we have in crates.io, Do you know where can we find it? It would be good to retroactively create the 4.0.0 tag for change log automation. I am guessing the commit is around ddaea81?
@jorgecarleitao do we want to bundle the python release with the main datafusion release as well? Or do we want to release python binding in its own cadence? I think if the python binding wants to have a more frequent release cadence than the core, then it's better to have a separate release process for it. Otherwise it would be less work to bundle it with the core Datafusion release.
We also need to consider what to do about Ballista release cadence and version numbers. My preference would be to release DataFusion and Ballista together. Ballista version is currently 0.5.0-SNAPSHOT and I think that releasing as 0.5.0 would be appropriate.
That would be awesome, I would +1 on aligning to 0.5.0, also for Python Datafusion (i.e. bump from 0.2.1 to 0.5.0); I think we do have a problem with DataFusion, that is already 5.0.0 on crates.io?
Given that DataFusion doesn't have the associated arrow ecosystem I think using an entirely different versioning scheme is fine for DataFusion and Ballista - I don't really have a strong opinion on versions for DataFusion (e.g I would be happy to have it be 0.5.0 or something, though I don't know how we would handle the existing versions)
BTW @houqp / @jimexist I am happy to help here / run whatever scripts / share my experience with doing the release process with arrow-rs -- just let me know what I can do to help
I also prefer we go with our own versioning scheme to better adhere to semantic versioning at the very least for ballista and datafusion python binding.
For datafusion the ship has already sailed unfortunately. If we are to release datafusion and ballista together, how do we manage the version differences? Do we just name the same tarball with different names? Or do we actually produce two different release tarballs with project specific sources?
Regardless what versioning scheme we use, the datafusion python documentation would still be needed. @jimexist perhaps you could help start building the automation for that?
Here is what I think could work for us, release all project sources in a single signed repo source tarball, which gets uploaded to apache svn. Then from within that same source release, we publish datafusion crate, ballista crate and datafusion python wheels with different versions.
For example, in the upcoming release, we could have datafusion-5.0.0, ballista-0.5.0 and datafusion python 0.3.0 (or 0.5.0 if we want to align it with ballista) published from the same source tarball apache-arrow-datafusion-5.0.0.tar.gz. Name of the source tarball won't matter much here since I don't expect downstream consumers of these projects to use it directly. Instead they would use crates.io or pypi with the proper project name and version.
Consequence of this is every time we need to release a new version of either datafusion, python binding or ballista, we would need to vote and release a new version the datafusion repo as a whole.The repo source release should be pretty light weight given we don't need to do maintenance releases in current state of the project. All we have to do is to run create-tarball.sh , send to dev list for vote, then finally run release-tarball.sh.
Changelog for each project will need to be generated separately before we propose a source release in apache svn.
Should I push this tree to our repo and tag it with 4.0.0? Alternatively, the state of Rust code in that release maps to 31dd3cd in our master branch, so we could also just tag that commit as 4.0.0. But I think this might not align with Apache's release policy.
Regardless what versioning scheme we use, the datafusion python documentation would still be needed. @jimexist perhaps you could help start building the automation for that?
Then from within that same source release, we publish datafusion crate, ballista crate and datafusion python wheels with different versions.
I think that would work nicely and I think it makes a lot of sense.
Should I push this tree to our repo and tag it with 4.0.0?
I would recommend not doing this (because it will effectively bring along the (large) history of the arrow repo with it, effectively making the arrow-datafusion repo several times larger).
Alternatively, the state of Rust code in that release maps to 31dd3cd in our master branch, so we could also just tag that commit as 4.0.0. But I think this might not align with Apache's release policy.
I think tagging 31dd3cd as 4.0.0 is fine (and I did something similar in arrow-rs) -- the official apache release policy, at least as I understand and discussed on the arrow-dev mailing lists, was centered around the tarballs as the artifacts. Since the 4.0.0 release / announcement was built from https://github.com/apache/arrow/commits/f959141ece4d660bce5f7fa545befc0116a7db79 if anyone wants to know exactly what was in the release they can use that reference. tagging 31dd3cd in this repo to compute the changelog seems like it would be fine
Read the docs is certainly the route with the least amount of work, but I am not 100% sure if it's something allowed for apache projects. The dev mailing list recommended hosting our docs as a sub path under the main arrow website. This incurs more work since this is not a turnkey solution. But it has the added benefit of us leveraging the arrow brand for marketing Datafusion and potentially improve SEO. We might be able to reuse the same automation that's used to generate https://arrow.apache.org/docs/python/api.html for datafusion too.
(@houqp the credentials I use for svn is my apache username/password). I am not sure what, if any, additional permissions are needed to upload to the release svn directory
I do not have permissions on the datafusion or ballista projects on crates.io. I believe @andygrove will have to grant one or both of us those permissions
I have invited @alamb to become an owner on all of the crates (I think this is currently a PMC role) and I have published the crates. @houqp Could you call the vote on the mailing list?
@alamb I am double checking with asf infra on the permission, but it looks like only PMC member has write access to the release folder. There is one thing that I noticed is not right, the release tarball is signed with my signing key, I have added it to the end of https://dist.apache.org/repos/dist/dev/arrow/KEYS, but I also don't have permission to add it to release/arrow/KEYS. I believe my key should be added to the release KEYS file too so people can use it to verify the signature.
@houqp Could you call the vote on the mailing list?
@andygrove do we still need to call a vote for the crates.io release? I thought the vote on the source tarball should have alredy covered the creates.io source release?
On Sat, Aug 14, 2021, 11:20 AM QP Hou ***@***.***> wrote:
@alamb <https://github.com/alamb> I am double checking with asf infra on
the permission, but it looks like only PMC member has write access to the
release folder. There is one thing that I noticed is not right, the release
tarball is signed with my signing key, I have added it to the end of
https://dist.apache.org/repos/dist/dev/arrow/KEYS, but I also don't have
permission to add it to release/arrow/KEYS. I believe my key should be
added to the release KEYS file too so people can use it to verify the
signature.
@houqp <https://github.com/houqp> Could you call the vote on the mailing
list?
@andygrove <https://github.com/andygrove> do we still need to call a vote
for the crates.io release? I thought the vote on the source tarball
should have alredy covered the creates.io source release?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#771 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAHEBRDEQSKIOY3SYY5WNOTT42QVLANCNFSM5A4TC45A>
.
Activity
houqp commentedon Jul 27, 2021
@andygrove @jorgecarleitao I couldn't find the exact commit for the 4.0.0 version we have in crates.io, Do you know where can we find it? It would be good to retroactively create the 4.0.0 tag for change log automation. I am guessing the commit is around ddaea81?
@jorgecarleitao do we want to bundle the python release with the main datafusion release as well? Or do we want to release python binding in its own cadence? I think if the python binding wants to have a more frequent release cadence than the core, then it's better to have a separate release process for it. Otherwise it would be less work to bundle it with the core Datafusion release.
alamb commentedon Jul 27, 2021
I think 4.0.0 was released when datafusion was in the https://github.com/apache/arrow repo, specifically https://github.com/apache/arrow/tree/f959141ece4d660bce5f7fa545befc0116a7db79
I dug that version out of the email thread for the 4.0.0 RC3 release:
https://lists.apache.org/thread.html/rfb4e0065347235571cd0e0e71a7fa38c9652f07e27f98f89b4398feb%40%3Cdev.arrow.apache.org%3E
andygrove commentedon Jul 27, 2021
We also need to consider what to do about Ballista release cadence and version numbers. My preference would be to release DataFusion and Ballista together. Ballista version is currently 0.5.0-SNAPSHOT and I think that releasing as 0.5.0 would be appropriate.
jorgecarleitao commentedon Jul 27, 2021
That would be awesome, I would +1 on aligning to 0.5.0, also for Python Datafusion (i.e. bump from 0.2.1 to 0.5.0); I think we do have a problem with DataFusion, that is already 5.0.0 on crates.io?
alamb commentedon Jul 27, 2021
FWIW, the most recent release to crates.io for datafusion is 4.0.0: https://crates.io/crates/datafusion
Given that DataFusion doesn't have the associated arrow ecosystem I think using an entirely different versioning scheme is fine for DataFusion and Ballista - I don't really have a strong opinion on versions for DataFusion (e.g I would be happy to have it be 0.5.0 or something, though I don't know how we would handle the existing versions)
alamb commentedon Jul 27, 2021
BTW @houqp / @jimexist I am happy to help here / run whatever scripts / share my experience with doing the release process with arrow-rs -- just let me know what I can do to help
houqp commentedon Jul 28, 2021
I also prefer we go with our own versioning scheme to better adhere to semantic versioning at the very least for ballista and datafusion python binding.
For datafusion the ship has already sailed unfortunately. If we are to release datafusion and ballista together, how do we manage the version differences? Do we just name the same tarball with different names? Or do we actually produce two different release tarballs with project specific sources?
Regardless what versioning scheme we use, the datafusion python documentation would still be needed. @jimexist perhaps you could help start building the automation for that?
houqp commentedon Jul 28, 2021
Here is what I think could work for us, release all project sources in a single signed repo source tarball, which gets uploaded to apache svn. Then from within that same source release, we publish datafusion crate, ballista crate and datafusion python wheels with different versions.
For example, in the upcoming release, we could have datafusion-5.0.0, ballista-0.5.0 and datafusion python 0.3.0 (or 0.5.0 if we want to align it with ballista) published from the same source tarball
apache-arrow-datafusion-5.0.0.tar.gz. Name of the source tarball won't matter much here since I don't expect downstream consumers of these projects to use it directly. Instead they would use crates.io or pypi with the proper project name and version.Consequence of this is every time we need to release a new version of either datafusion, python binding or ballista, we would need to vote and release a new version the datafusion repo as a whole.The repo source release should be pretty light weight given we don't need to do maintenance releases in current state of the project. All we have to do is to run
create-tarball.sh, send to dev list for vote, then finally runrelease-tarball.sh.Changelog for each project will need to be generated separately before we propose a source release in apache svn.
WDYT?
houqp commentedon Jul 28, 2021
on a side note, turns out the tree we used for 4.0.0 release (https://github.com/apache/arrow/commits/f959141ece4d660bce5f7fa545befc0116a7db79) is not in our repo.
Should I push this tree to our repo and tag it with 4.0.0? Alternatively, the state of Rust code in that release maps to 31dd3cd in our master branch, so we could also just tag that commit as 4.0.0. But I think this might not align with Apache's release policy.
jimexist commentedon Jul 28, 2021
@jorgecarleitao any previous work already done this area?
also there's how and where docs are supposed to be hosted, e.g.
alamb commentedon Jul 28, 2021
I think that would work nicely and I think it makes a lot of sense.
I would recommend not doing this (because it will effectively bring along the (large) history of the arrow repo with it, effectively making the arrow-datafusion repo several times larger).
I think tagging 31dd3cd as 4.0.0 is fine (and I did something similar in arrow-rs) -- the official apache release policy, at least as I understand and discussed on the arrow-dev mailing lists, was centered around the tarballs as the artifacts. Since the 4.0.0 release / announcement was built from https://github.com/apache/arrow/commits/f959141ece4d660bce5f7fa545befc0116a7db79 if anyone wants to know exactly what was in the release they can use that reference. tagging 31dd3cd in this repo to compute the changelog seems like it would be fine
houqp commentedon Jul 28, 2021
@jimexist there were some previous discussion on how to host a website for datafusion on the dev list: https://lists.apache.org/thread.html/r0ed76cc60cdf651e8cf5c82a21cc64114c1f6d174dc5487434bd32ef%40%3Cdev.arrow.apache.org%3E.
Read the docs is certainly the route with the least amount of work, but I am not 100% sure if it's something allowed for apache projects. The dev mailing list recommended hosting our docs as a sub path under the main arrow website. This incurs more work since this is not a turnkey solution. But it has the added benefit of us leveraging the arrow brand for marketing Datafusion and potentially improve SEO. We might be able to reuse the same automation that's used to generate https://arrow.apache.org/docs/python/api.html for datafusion too.
11 remaining items
houqp commentedon Aug 7, 2021
@alamb @andygrove @Dandandan @jorgecarleitao @nevi-me @jimexist #801 is now ready for review. Once it's merged, we will be able to push tags, release tarball and send a rc voting email to the dev list.
houqp commentedon Aug 7, 2021
Filed #837 to track the python doc release as a quick follow up so we don't have to block the current release.
houqp commentedon Aug 14, 2021
OK, release has been approved on the dev list. I have pushed the final release tags on Github. However, I don't have write access to arrow's release directory in SVN, crates.io nor PyPI. So I will need anyone who has access to these resources help me finish up the release. The steps are documented at https://github.com/houqp/arrow-datafusion/blob/qp_release_doc/dev/release/README.md#finalize-the-release.
The remaining steps are:
./dev/release/release-tarball.sh 5.0.0 3to upload to release SVN dirpublish python binding to PyPIneed more follow up on document release process #875 (comment).alamb commentedon Aug 14, 2021
I will run the script to upload to SVN...
alamb commentedon Aug 14, 2021
(@houqp the credentials I use for svn is my apache username/password). I am not sure what, if any, additional permissions are needed to upload to the release svn directory
alamb commentedon Aug 14, 2021
The release files can be found here: https://dist.apache.org/repos/dist/release/arrow/arrow-datafusion-5.0.0/
I do not have permissions on the datafusion or ballista projects on crates.io. I believe @andygrove will have to grant one or both of us those permissions
Thanks again for all this work @houqp 🚀
andygrove commentedon Aug 14, 2021
I have invited @alamb to become an owner on all of the crates (I think this is currently a PMC role) and I have published the crates. @houqp Could you call the vote on the mailing list?
https://crates.io/crates/datafusion/
https://crates.io/crates/ballista/
https://crates.io/crates/ballista-core/
https://crates.io/crates/ballista-executor/
https://crates.io/crates/ballista-scheduler/
This whole process went very smoothly. Thank you @houqp and @alamb.
houqp commentedon Aug 14, 2021
@alamb I am double checking with asf infra on the permission, but it looks like only PMC member has write access to the release folder. There is one thing that I noticed is not right, the release tarball is signed with my signing key, I have added it to the end of https://dist.apache.org/repos/dist/dev/arrow/KEYS, but I also don't have permission to add it to
release/arrow/KEYS. I believe my key should be added to the release KEYS file too so people can use it to verify the signature.@andygrove do we still need to call a vote for the crates.io release? I thought the vote on the source tarball should have alredy covered the creates.io source release?
andygrove commentedon Aug 14, 2021
houqp commentedon Aug 14, 2021
OK, I have called the vote on the dev list, will add this step to the release document as well.
houqp commentedon Aug 15, 2021
Filed #887 to track Python PyPI release as a follow up.
I believe the only remaining item is to copy my code signing key from
dev/arrow/KEYStorelease/arrow/KEYS.alamb commentedon Aug 15, 2021
@houqp I have added your key here: https://dist.apache.org/repos/dist/release/arrow/KEYS
houqp commentedon Aug 15, 2021
Thank you @alamb we are all good to close this issue then :)
Upgrade Datafusion 40 (apache#771)