provide hook for resolving binary dependencies and bundling them in zip file  #6

Activity

1. bernardobelchior commented on Feb 10, 2020

   

   bernardobelchior

   on Feb 10, 2020

   Contributor

   More actions

   I have been giving some thought on how to solve this and would like your opinion.

   How would you feel of having a scripts folder that would contain hooks to run?
   This way downstream users would have a lot of flexibility. There could be default scripts like build.sh (that could be overriden), but also have other hooks, e.g., install.sh (which could install binary dependencies), package.sh (which could package the application).

   By using inversion of control, the final user would have much more flexibility and we could also set sane defaults.
   What are your thoughts on this?

   👍React with 👍1Reacted by StitchYourPhoto
2. triptec commented on Mar 13, 2020

   

   triptec

   on Mar 13, 2020

   More actions

   Any update on this?
3. bernardobelchior commented on Apr 3, 2020

   

   bernardobelchior

   on Apr 3, 2020

   Contributor

   More actions

   I would still like to try to tackle this issue, but I'd like a green light from a contributor to make sure the effort wouldn't be for nothing.

   @softprops you seem to be the main contributor, could you give your opinion here?
4. softprops commented on Apr 3, 2020

   

   softprops

   on Apr 3, 2020

   OwnerAuthor

   More actions

   Sounds good. What's the approach you'd be leaning towards? I don't want to be a blocker here
5. bernardobelchior commented on Apr 4, 2020

   

   bernardobelchior

   on Apr 4, 2020

   Contributor

   More actions

   What I'm imagining is something like this:

   1. There would be a scripts/ directory inside the lambda-rust root directory with two scripts: one for building and one for packaging, which would be the current build.sh split into these two phases.
   2. Inside the scripts/ directory, every script would start (by convention) with a 3-digit number so that scripts can be run in a predictable order. The order would be the one given by sort.

   The consumers of lambda-rust could have their own scripts/ directory where they would also add their own scripts. If a user script has the exact same name of a lambda-rust script, then only the user script would be run.

   Using the 3-digit notation before the name, I propose the initial scripts directory of lambda-rust to be something like:

   • 300package.sh
   • 700build.sh

   The reason behind these numbers is the ability to add a significant amount of scripts before or after each step of the build process.

   Example:

   lambda-rust/scripts:

   .
   ├── 300package.sh
   └── 700build.sh
   

   consumer-lambda/scripts:

   .
   ├── 100install-native-dependencies.sh
   └── 300package.sh
   

   The scripts would be run in the following order:

   1. consumer-lambda/scripts/100install-native-dependencies.sh
   2. consumer-lambda/scripts/300package.sh
   3. lambda-rust/scripts/700build.sh

   If one scripts fails (i.e., returns non-zero) then the whole process would fail.
   Note that the lambda-rust/scripts/300package.sh was overridden and would not run.

   ---

   Possible problems and solutions:

   1. User could already have a scripts/ directory in their project. To solve this, the name of this directory would be configurable using a environment variable, .e.g SCRIPTS_DIR.
   2. By adding this new functionality without a version bump, we could break consumers (e.g., if they already have a scripts directory that would be run without them knowing that would happen). To address this, we can either (1) bump the version of the docker image or (2) make the SCRIPTS_DIR functionality opt-in, meaning that if the environment variable is not set, then it will just run the default scripts.

   I can't think of other problems or possible things that could break. Do you have any insights here or have I covered all the cases?

   What is your opinion on the approach?
6. softprops commented on Apr 4, 2020

   

   softprops

   on Apr 4, 2020

   OwnerAuthor

   More actions

   My preference would be to remove as much ceremony as possible so there's less to maintain for users that don't need this feature. I bit of the recent debug options have has started to feel that way. As much as possible only design for what you need and nothing more.

   As few considerations. I separately maintain a serverless framework plugin for rust which leverages this builder for building rust lambdas consistent with the lambda runtime env. There are some discussions around adding support for building lambdas in a host env outside of docker. That becomes more complicated as this docker image does more than just a cargo build.
   A consideration there is how that works for windows users.

   I see the default case for cargo is not needing anything at all. Cargo provides built-in support for a build script that will automatically get ran.

   I like the idea of lifecycle hooks but I'm not sure about the need for having multiple copies of them. There operations I see are

   Install

   Currently there's a hook to yum install extra deps if you need them as docker run args

   lambda-rust/build.sh

   Line 19 in cff2b3a

   yum install -y "$@"

   . this impl could be a fallback in absence of an install script or env var

   Build

   Caro build is pretty basic as mentioned above it comes out of the box with build script support

   lambda-rust/build.sh

   Line 25 in cff2b3a

   if [ "${PROFILE}" == "release" ]; then

   could be the fallback of a build script were absent but I'd like to hear more about how cargo build doesn't fit your rust bin needs

   Package

   This one is very has a very specific contract. The lambda runtime expects a zip file with an executable called "bootstrap" inside it. That's essentially all we're doing now

   lambda-rust/build.sh

   Line 46 in cff2b3a

   zip "$file.zip" bootstrap

   . I'm not sure what a package script hook would look like but I'm open to suggestions. The problem to solve there is a need to generate a list to additional assets to collect into the zip.

   "scripts" is pretty common name for a utility dir. I'd like to avoid surprises if you happened to have a scripts dir with a script sharing the same name has one of the hook scripts we'd be introducing and have it not be intended for use with this yool. I'd suggest something a bit more specific to the tool. In this case perhaps .lambda-rust/{install,build,package}

   If you'd like to start sketching something out given the above feedback is be open to it. I'm a fan of iterating with small changes then learning how well they work in practice before making big changes. If there's a way to implement one of these hooks and see if it solves a problem that would be cool too.
7. bernardobelchior commented on Apr 7, 2020

   

   bernardobelchior

   on Apr 7, 2020

   Contributor

   More actions

   I like your ideas. I'll your concerns point by point.

   As few considerations. I separately maintain a serverless framework plugin for rust which leverages this builder for building rust lambdas consistent with the lambda runtime env. There are some discussions around adding support for building lambdas in a host env outside of docker. That becomes more complicated as this docker image does more than just a cargo build.
   A consideration there is how that works for windows users.

   I don't understand what you mean. If there are discussions about adding support for building lambdas outside docker, what does it have to do with this project?

   ---

   I like the idea of lifecycle hooks but I'm not sure about the need for having multiple copies of them.

   The rationale was to provide users that need high flexibility the possibility to customize the process as they see fit. But I agree that it may be over-engineering at this point.

   ---

   I'm not sure what a package script hook would look like but I'm open to suggestions. The problem to solve there is a need to generate a list to additional assets to collect into the zip.

   It's possible to add files to a zip file using the zip command. This is how I'm adding files to the zip in my use case.

   ---

   "scripts" is pretty common name for a utility dir. I'd like to avoid surprises if you happened to have a scripts dir with a script sharing the same name has one of the hook scripts we'd be introducing and have it not be intended for use with this yool. I'd suggest something a bit more specific to the tool. In this case perhaps .lambda-rust/{install,build,package}

   I agree that scripts would not be a good name if hardcoded. Do you think having a SCRIPTS_DIR would be useful? If not, is there any reason to call it .lambda-rust rather than lambda-rust?

   ---

   Thanks for the thorough feedback! I plan to start working on this shortly