Skip to content

fiddle gem requires fileutils gem at install time #153

Closed
#155
Closed
fiddle gem requires fileutils gem at install time#153
#155

Description

@headius
headius
opened on Oct 16, 2024

JRuby does not version any of the sources of default gems in our repository. Instead, we do a build step that fetches, unpacks, and installs those gems from their released versions.

At the time we do this, none of those default gems are present in the codebase.

fiddle's install process requires mkmf, even to generate a dummy makefile, which in turn depends on fileutils. As a result, it's not possible to install fiddle during our build because the fileutils default gem is not yet present.

This illustrates a problem with depending on mkmf to generate a dummy makefile; mkmf itself is nontrivial and has other dependencies.

We may be able to restructure our build to install the fileutils default gem files into stdlib before attempting to build fiddle, but I wanted to put this out there in case we can think of other options.

The error that results during our build if we install fiddle as a default gem at build time:

[INFO] --- polyglot:0.5.0:execute (install_gems) @ jruby-stdlib ---
using jruby 9.4.5.0
install gems unless already installed
/Users/headius/.m2/repository/rubygems/rubygems-update/3.5.21/rubygems-update-3.5.21.gem
/Users/headius/.m2/repository/rubygems/benchmark/0.3.0/benchmark-0.3.0.gem
/Users/headius/.m2/repository/rubygems/bundler/2.5.21/bundler-2.5.21.gem
/Users/headius/.m2/repository/rubygems/cgi/0.4.1/cgi-0.4.1.gem
/Users/headius/.m2/repository/rubygems/date/3.3.4/date-3.3.4.gem
/Users/headius/.m2/repository/rubygems/delegate/0.3.1/delegate-0.3.1.gem
/Users/headius/.m2/repository/rubygems/did_you_mean/1.6.3/did_you_mean-1.6.3.gem
/Users/headius/.m2/repository/rubygems/digest/3.1.1/digest-3.1.1.gem
/Users/headius/.m2/repository/rubygems/english/0.8.0/english-0.8.0.gem
/Users/headius/.m2/repository/rubygems/erb/4.0.4/erb-4.0.4.gem
/Users/headius/.m2/repository/rubygems/error_highlight/0.6.0/error_highlight-0.6.0.gem
/Users/headius/.m2/repository/rubygems/ffi/1.16.3/ffi-1.16.3.gem
/Users/headius/.m2/repository/rubygems/fileutils/1.7.2/fileutils-1.7.2.gem
/Users/headius/.m2/repository/rubygems/fiddle/1.1.3/fiddle-1.1.3.gem
Building native extensions. This could take a while...
LoadError: no such file to load -- fileutils
  require at org/jruby/RubyKernel.java:1186
   <main> at /Users/headius/work/jruby/lib/ruby/stdlib/mkmf.rb:7
  require at org/jruby/RubyKernel.java:1186
   <main> at extconf.rb:2

Activity

  1. kou commented on Oct 16, 2024

    @kou
    kou
    on Oct 16, 2024
    Member

    Does this work?

    diff --git a/ext/fiddle/extconf.rb b/ext/fiddle/extconf.rb
index 2899980..e4bd46f 100644
--- a/ext/fiddle/extconf.rb
+++ b/ext/fiddle/extconf.rb
@@ -1,11 +1,17 @@
 # frozen_string_literal: true
-require 'mkmf'
 
 unless RUBY_ENGINE == "ruby"
-  File.write('Makefile', dummy_makefile("").join)
+  File.write('Makefile', <<-MAKEFILE)
+all install clean:
+	#{CONFIG["NULLCMD"]}
+
+.PHONY: all install clean
+  MAKEFILE
   return
 end
 
+require 'mkmf'
+
 # :stopdoc:
 
 def gcc?

  2. headius commented on Oct 17, 2024

    @headius
    headius
    on Oct 17, 2024
    Author

    @kou Yes, that should work! It is less "standard" than generating dummy_makefile with mkmf but for core gems it's probably a better choice (avoiding external dependencies).

  3. headius commented on Oct 17, 2024

    @headius
    headius
    on Oct 17, 2024
    Author

    @eregon @deivid-rodriguez I had not considered that mkmf has a dependency on fileutils, which means all ext gems basically depend on both of them (as well as make but that's a harder thing to fix). If we can modify RubyGems to ignore blank Makefiles, couldn't we also add logic for generating those files? (Or something like Gem.mark_this_extension_as_built that could be called in extconf.rb?)

  4. headius commented on Oct 17, 2024

    @headius
    headius
    on Oct 17, 2024
    Author

    @kou If we can push a -pre gem I could test it in our build.

  5. eregon commented on Oct 17, 2024

    @eregon
    eregon
    on Oct 17, 2024
    Member

    Right, so this is a problem while building JRuby itself.
    fileutils is a default gem, so it should always be available.

    And mkmf seems a stdlib, it's not a gem at all as far as I can see (no gemspec or repo for it).

    We may be able to restructure our build to install the fileutils default gem files into stdlib before attempting to build fiddle, but I wanted to put this out there in case we can think of other options.

    I think this is the better solution. Plenty of gems expect that require 'mkmf' works.
    It is a stdlib, so should be always available, even without RubyGems.

    I'm not against the File.write workaround though.
    FWIW, Prism does this too, for other reasons.

  6. headius commented on Oct 17, 2024

    @headius
    headius
    on Oct 17, 2024
    Author

    fileutils is a default gem, so it should always be available

    fiddle does not depend on fileutils once it has been built and installed. Ideally, if it does not need to generate a real makefile, it shouldn't depend on fileutils there either. Dependencies between core gems should be kept as clean as possible.

    FWIW all of the other extension gems in stdlib provide a -java version that does not require a Makefile or make. We have tried to compromise here because the FFI code will be used for implementations other than MRI, but that compromise is harder to support if we have to make special affordances for generating a dummy Makefile that will end up doing literally nothing.

    And mkmf seems a stdlib

    This is a separate problem (non-gem stdlib depending on gem-based stdlib).

    Plenty of gems expect that require 'mkmf' works.

    Gems that are not default and installed into a fully-prepared JRuby would have no problem here. Gems that are considered "default" and which may be installed at build time into a partially-complete runtime should avoid interdependencies whenever possible.

    I'm not against the File.write workaround though.

    Great, let's do that then.

  7. kou
    added 2 commits that reference this issue on Oct 18, 2024

    Don't use mkmf to generate dummy Makefile

    399fa5c

    Don't use mkmf to generate dummy Makefile

    6962460
  8. kou
    closed this as completedin df4bd21on Oct 22, 2024
  9. kou
    closed this as completedin #155on Oct 22, 2024
  10. headius
    added a commit that references this issue on Oct 22, 2024

    Update fiddle to 1.1.4

    1d114d2

  11. headius commented on Oct 22, 2024

    @headius
    headius
    on Oct 22, 2024
    Author

    Confirmed working in JRuby 10 branch; I can install both fiddle and fileutils at build time and there's no dependency issue.

    Thank you @kou! 🙇‍♂️

  12. pull
    added a commit that references this issue on Oct 23, 2024

    Don't use mkmf to generate dummy Makefile (ruby#155)

    df4bd21
  13. hsbt
    added 2 commits that reference this issue on Oct 25, 2024

    [ruby/fiddle] Don't use mkmf to generate dummy Makefile

    27c8d93

    [ruby/fiddle] Don't use mkmf to generate dummy Makefile

    4fd21f1
  14. headius
    added a commit that references this issue on Oct 25, 2024

    Update fiddle to 1.1.4

    f82e2b6
  15. hsbt
    added 2 commits that reference this issue on Oct 26, 2024

    [ruby/fiddle] Don't use mkmf to generate dummy Makefile

    a7e0e0a

    [ruby/fiddle] Don't use mkmf to generate dummy Makefile

    2ec2051
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions