Repository navigation
[R] package development builds broken for Windows and MacOS #35472
Description
Describe the bug, including details regarding any error messages, version, and platform.
If we try to build the R package on Windows or MacOS from a checkout, it fails at a point where it is supposed to download a suitable nightly build of libarrow:
remotes::install_github('apache/arrow/r')
# -- R CMD build -------------------------------------------
# - preparing 'arrow': (392ms)
# - checking DESCRIPTION meta-information
# - cleaning src
# - checking for LF line-endings in source and make files and shell scripts
# - checking for empty or unneeded directories
# - building 'arrow_12.0.0.9000.tar.gz'
# Installing package into ‘C:/Users/Jeroen/AppData/Local/R/win-library/4.3’
# (as ‘lib’ is unspecified)
# * installing *source* package 'arrow' ...
# ** using staged installation
# Warning message:
# In unzip("lib.zip", exdir = "windows") :
# error 1 in extracting from zip file
...
# using C++17
# g++ -std=gnu++17 -I"C:/PROGRA~1/R/R-43~1.0/include" -DNDEBUG -I../windows//include -DARROW_STATIC -DPARQUET_STATIC -DARROW_DS_STATIC -DARROW_ACERO_STATIC -DARROW_R_WITH_PARQUET -DARROW_R_WITH_ACERO -DARROW_R_WITH_DATASET -DARROW_R_WITH_JSON -DARROW_R_WITH_S3 -DARROW_R_WITH_GCS -I'C:/Users/Jeroen/AppData/Local/R/win-library/4.3/cpp11/include' -I"C:/rtools43/x86_64-w64-mingw32.static.posix/include" -O2 -Wall -mfpmath=sse -msse2 -mstackrealign -c RTasks.cpp -o RTasks.o
# In file included from RTasks.cpp:18:
# ./r_task_group.h:20:10: fatal error: arrow/util/parallel.h: No such file or directory
# 20 | #include <arrow/util/parallel.h>
# | ^~~~~~~~~~~~~~~~~~~~~~~
# compilation terminated.Same on MacOS.
I'm not sure if it is related, but the builds on https://nightlies.apache.org/arrow/r/libarrow/bin/ have not been updated for 6 weeks. But even before that it was not working, I think the R package script does not not find the correct URLs.
Component(s)
R
Activity
paleolimbot commentedon May 8, 2023
I'm not sure that
remotes::install_github()has ever correctly downloaded nightlies...it's on my TODO list to make this work on Windows (#35049), though, since it's incredibly time-consuming to test development versions on Windows without it. Neal just did #35147 which should fixremotes::install_github('apache/arrow/r', build = FALSE).Either way we'll need the hosted nightly builds! 😬
[-]R package development builds broken for Windows and MacOS[/-][+][R] package development builds broken for Windows and MacOS[/+]nealrichardson commentedon May 8, 2023
(Setting aside Windows),
R CMD build && R CMD INSTALLwon't work without some additional build prep to put a copy the C++ source inside the R package directory.R CMD INSTALL(akaremotes::install_github('apache/arrow/r', build = FALSE)) does work out of the box because the build script can look for the C++ source in the monorepo parent directory at../cpp/src. This is all using our bundled build script, not autobrew on mac.Our nightly packaging job has this pre-build step to copy in the C++ source, along with putting a copy of the homebrew/autobrew formula inside the package and pinning the C++ library version in the formula to the same git SHA as the R package, so that they are in sync: https://github.com/ursacomputing/crossbow/blob/nightly-packaging-2023-05-08-0-github-r-binary-packages/.github/workflows/crossbow.yml#L47-L86 After doing this,
R CMD build && R CMD INSTALLwill work.Windows is different and we don't currently have a path that will call the mingw makepkg command to build the C++ libraries at R package installation time. Presumably it could be done, but I don't think there's much interest in it.
As @paleolimbot says, we could (just) download the latest nightly binary packages instead of building from source, and that will almost always work. That's probably the way to go on Windows; on mac/linux, there's probably an env var we'll want to expose to control that behavior--perhaps it's the fallback you get instead of
*** Proceeding without libarrow (no local source), and you can opt-in to it even if you have the source available?As for our nightly binary packaging workflow, I'm not sure what's going on. The job that builds them continues to report success (latest: https://github.com/ursacomputing/crossbow/actions/runs/4906238724/jobs/8760994170). I don't see where they are uploaded to the nightlies repository though, and I have a faint recollection that that's a separate process. @assignUser do you know what's up, and why we aren't seeing that the packages aren't being successfully uploaded?
assignUser commentedon May 8, 2023
tools/winlibs.Rtries to download a libarrow zip matching the version (12.0.0.900 here) but the nightlies are numbered consecutively (which is a recent'ish change from being date based) so we would need to beef up the script in that regard.I will have a look at the upload @nealrichardson
assignUser commentedon May 8, 2023
#35478 fixes the nightly issue, not sure why I failed to update that workflow when we updated the rest after the switch...
paleolimbot commentedon May 8, 2023
The version of pkgbuild that includes r-lib/pkgbuild#157 isn't released yet, but we can migrate the c++ vendoring + pruning to bootstrap.R + add a line to DESCRIPTION, after which we don't need
build = FALSE(when the required version of pkgbuild is installed).nealrichardson commentedon May 8, 2023
@paleolimbot very cool. That should help with the
remotes::install_github()path. I think we still have a gap if someone is doing theR CMDstuff at the CLI (which we do in our CI scripts), but still that will allow us to consolidate that bootstrapping logic in one place, which is an improvement for sure.jeroen commentedon May 10, 2023
@paleolimbot while on-the-fly vendoring can be useful for some, it deviates the CI even further from the build process used for (CRAN) releases, which relies on 3rd parties (me) providing libarrow builds for win/mac that the R package can download.
I think the priority should be to make the arrow CI self sustaining and focus on getting such libarrow nightlies and releases built and published for the platforms that you want to support.
nealrichardson commentedon May 10, 2023
We vendor the C++ source before submitting to CRAN, so the .tar.gz source package one installs there has it included. What @paleolimbot was mentioning would make that pre-build step we do become a part of the remotes/devtools workflow, so it would be more like the CRAN scenario.
For windows, we already build and host the static libraries for releases (e.g. https://apache.jfrog.io/ui/native/arrow/r/12.0.0/libarrow/bin/windows/), so we may be able to just drop the rwinlib path if that's helpful. Mac is more involved with the autobrew/bottle setup, but I'm sure we could find ways to make that more self-sustaining.
jeroen commentedon May 11, 2023
OK that makes sense. How is the vendored libarrow code currently copied into the CRAN source package? Is this scripted somewhere I can reproduce it?
nealrichardson commentedon May 11, 2023
It's currently
make sync-cpp, though putting that inbootstrap.Rlike Dewey proposed would help so that you don't have to maintain ad-hocif arrowlogic.For the nightly/on-demand autobrew build, we don't use that vendored source (though presumably there would be a way?). Instead we use a copy of the formula(e) where we pin to a git SHA, so that when we
brew install --HEADit builds the right C++ code. That looks like https://github.com/ursacomputing/crossbow/blob/nightly-packaging-2023-05-08-0-github-r-binary-packages/.github/workflows/crossbow.yml#L47-L86 (the workflow is templated).jeroen commentedon May 11, 2023
OK thanks, this fixes the Linux build. If I try to build a copy of arrow with vendored source on Windows, it fails with:
I see the nightlies on https://nightlies.apache.org/arrow/r/libarrow/bin/windows/ have a different version scheme for example todays build is
arrow-12.0.0.100000043.zip.nealrichardson commentedon May 11, 2023
Right, we don't do a bundled build on windows, the makepkg-mingw happens out of band. Pulling a nightly binary (doing the appropriate version matching of .9000 to latest) on Windows was what @paleolimbot was discussing doing in #35049.