Repository navigation
Closed
Description
When pushing an app, staging fails on PWS when the buildpack tries to untar Ruby. We see the same behavior for 2.1.4 and 2.1.3 (haven't tried 2.1.2). Below is the output and corresponding logs:
Output from push:
|2.1.3| montgomery in ~/workspace/feedback
± ek+wr |master ✗| → cf push web
Using manifest file /Users/pivotal/workspace/feedback/manifest.yml
Updating app web in org Pivotal Feedback / space staging as labs-feedback-team@pivotal.io...
OK
Uploading web...
Uploading app files from: /Users/pivotal/workspace/feedback
Uploading 1.5M, 842 files
OK
Binding service redis to app web in org Pivotal Feedback / space staging as labs-feedback-team@pivotal.io...
OK
Stopping app web in org Pivotal Feedback / space staging as labs-feedback-team@pivotal.io...
OK
Starting app web in org Pivotal Feedback / space staging as labs-feedback-team@pivotal.io...
OK
-----> Downloaded app package (812K)
FAILED
Server error, status code: 400, error code: 170004, message: App staging failed in the buildpack compile phase
TIP: use 'cf logs web --recent' for more information
And logs:
± ek+wr |master ✗| → cf logs web --recent
Connected, dumping recent logs for app web in org Pivotal Feedback / space staging as labs-feedback-team@pivotal.io...
2014-12-04T09:48:51.48-0800 [API] OUT Tried to stop app that never received a start event
2014-12-04T09:48:51.50-0800 [API] OUT Updated app with guid 8ab75c22-5c8b-411a-94ed-a82a79999ba9 ({"state"=>"STOPPED"})
2014-12-04T09:48:51.56-0800 [App/0] OUT [37] - Gracefully shutting down workers...
2014-12-04T09:48:51.56-0800 [App/0] ERR
2014-12-04T09:48:51.57-0800 [App/1] ERR
2014-12-04T09:48:52.69-0800 [DEA] OUT Stopping app instance (index 0) with guid 8ab75c22-5c8b-411a-94ed-a82a79999ba9
2014-12-04T09:48:52.69-0800 [DEA] OUT Stopped app instance (index 0) with guid 8ab75c22-5c8b-411a-94ed-a82a79999ba9
2014-12-04T09:48:52.69-0800 [DEA] OUT Stopping app instance (index 1) with guid 8ab75c22-5c8b-411a-94ed-a82a79999ba9
2014-12-04T09:48:52.69-0800 [DEA] OUT Stopped app instance (index 1) with guid 8ab75c22-5c8b-411a-94ed-a82a79999ba9
2014-12-04T09:48:55.13-0800 [API] OUT Updated app with guid 8ab75c22-5c8b-411a-94ed-a82a79999ba9 ({"name"=>"web", "command"=>"PRIVATE DATA HIDDEN", "instances"=>2, "memory"=>2048, "environment_json"=>"PRIVATE DATA HIDDEN"})
2014-12-04T09:50:23.29-0800 [DEA] OUT Got staging request for app with id 8ab75c22-5c8b-411a-94ed-a82a79999ba9
2014-12-04T09:50:34.08-0800 [API] OUT Updated app with guid 8ab75c22-5c8b-411a-94ed-a82a79999ba9 ({"state"=>"STARTED"})
2014-12-04T09:50:39.40-0800 [STG] OUT -----> Downloaded app package (64M)
2014-12-04T09:50:41.59-0800 [STG] OUT -----> Downloaded app buildpack cache (58M)
2014-12-04T09:50:45.23-0800 [STG] ERR Cloning into '/tmp/buildpacks/ruby-buildpack'...
2014-12-04T09:50:46.28-0800 [STG] ERR Note: checking out 'b476e5a351b69e4c3acfeacdcddd6fc02de5358e'.
2014-12-04T09:50:46.28-0800 [STG] ERR You are in 'detached HEAD' state. You can look around, make experimental
2014-12-04T09:50:46.28-0800 [STG] ERR changes and commit them, and you can discard any commits you make in this
2014-12-04T09:50:46.28-0800 [STG] ERR state without impacting any branches by performing another checkout.
2014-12-04T09:50:46.28-0800 [STG] ERR If you want to create a new branch to retain commits you create, you may
2014-12-04T09:50:46.28-0800 [STG] ERR do so (now or later) by using -b with the checkout command again. Example:
2014-12-04T09:50:46.28-0800 [STG] ERR git checkout -b new_branch_name
2014-12-04T09:50:46.48-0800 [STG] OUT Submodule 'buildpack-packager' (https://github.com/cf-buildpacks/buildpack-packager.git) registered for path 'buildpack-packager'
2014-12-04T09:50:46.54-0800 [STG] ERR Cloning into 'buildpack-packager'...
2014-12-04T09:50:46.82-0800 [STG] OUT Submodule path 'buildpack-packager': checked out '87560fd0a76ccaaeb71232bf34dc616ea8876338'
2014-12-04T09:50:46.99-0800 [STG] OUT -------> Buildpack version 1.1.2
2014-12-04T09:50:47.30-0800 [STG] OUT -----> Compiling Ruby/Rails
2014-12-04T09:50:49.25-0800 [STG] OUT !
2014-12-04T09:50:49.25-0800 [STG] OUT ! Command: 'set -o pipefail; curl --fail --retry 3 --retry-delay 1 --connect-timeout 10 --max-time 90 https://s3-external-1.amazonaws.com/heroku-buildpack-ruby/ruby-2.1.4.tgz -s -o - | tar zxf - ' failed unexpectedly:
2014-12-04T09:50:49.25-0800 [STG] OUT !
2014-12-04T09:50:49.25-0800 [STG] OUT ! gzip: stdin: unexpected end of file
2014-12-04T09:50:49.25-0800 [STG] OUT ! tar: Child returned status 1
2014-12-04T09:50:49.25-0800 [STG] OUT ! tar: Exiting with failure status due to previous errors
2014-12-04T09:50:49.25-0800 [STG] OUT !
2014-12-04T09:50:49.27-0800 [STG] OUT Staging failed: Buildpack compilation step failed
2014-12-04T09:50:49.47-0800 [API] ERR Encountered error: App staging failed in the buildpack compile phase
2014-12-04T09:52:03.27-0800 [API] OUT Updated app with guid 8ab75c22-5c8b-411a-94ed-a82a79999ba9 ({"name"=>"web", "command"=>"PRIVATE DATA HIDDEN", "instances"=>2, "memory"=>2048, "environment_json"=>"PRIVATE DATA HIDDEN"})
2014-12-04T09:52:17.25-0800 [API] OUT Tried to stop app that never received a start event
2014-12-04T09:52:17.27-0800 [API] OUT Updated app with guid 8ab75c22-5c8b-411a-94ed-a82a79999ba9 ({"state"=>"STOPPED"})
2014-12-04T09:52:20.36-0800 [DEA] OUT Got staging request for app with id 8ab75c22-5c8b-411a-94ed-a82a79999ba9
2014-12-04T09:52:24.75-0800 [API] OUT Updated app with guid 8ab75c22-5c8b-411a-94ed-a82a79999ba9 ({"state"=>"STARTED"})
2014-12-04T09:52:25.04-0800 [STG] OUT -----> Downloaded app package (816K)
2014-12-04T09:52:27.29-0800 [STG] OUT -----> Downloaded app buildpack cache (58M)
2014-12-04T09:52:27.55-0800 [STG] ERR Cloning into '/tmp/buildpacks/ruby-buildpack'...
2014-12-04T09:52:28.54-0800 [STG] ERR Note: checking out 'b476e5a351b69e4c3acfeacdcddd6fc02de5358e'.
2014-12-04T09:52:28.54-0800 [STG] ERR You are in 'detached HEAD' state. You can look around, make experimental
2014-12-04T09:52:28.54-0800 [STG] ERR changes and commit them, and you can discard any commits you make in this
2014-12-04T09:52:28.54-0800 [STG] ERR state without impacting any branches by performing another checkout.
2014-12-04T09:52:28.54-0800 [STG] ERR If you want to create a new branch to retain commits you create, you may
2014-12-04T09:52:28.54-0800 [STG] ERR do so (now or later) by using -b with the checkout command again. Example:
2014-12-04T09:52:28.54-0800 [STG] ERR git checkout -b new_branch_name
2014-12-04T09:52:28.78-0800 [STG] OUT Submodule 'buildpack-packager' (https://github.com/cf-buildpacks/buildpack-packager.git) registered for path 'buildpack-packager'
2014-12-04T09:52:28.83-0800 [STG] ERR Cloning into 'buildpack-packager'...
2014-12-04T09:52:29.13-0800 [STG] OUT Submodule path 'buildpack-packager': checked out '87560fd0a76ccaaeb71232bf34dc616ea8876338'
2014-12-04T09:52:29.30-0800 [STG] OUT -------> Buildpack version 1.1.2
2014-12-04T09:52:29.57-0800 [STG] OUT -----> Compiling Ruby/Rails
2014-12-04T09:52:29.88-0800 [STG] OUT !
2014-12-04T09:52:29.88-0800 [STG] OUT ! Command: 'set -o pipefail; curl --fail --retry 3 --retry-delay 1 --connect-timeout 10 --max-time 90 https://s3-external-1.amazonaws.com/heroku-buildpack-ruby/ruby-2.1.4.tgz -s -o - | tar zxf - ' failed unexpectedly:
2014-12-04T09:52:29.88-0800 [STG] OUT !
2014-12-04T09:52:29.88-0800 [STG] OUT ! gzip: stdin: unexpected end of file
2014-12-04T09:52:29.88-0800 [STG] OUT ! tar: Child returned status 1
2014-12-04T09:52:29.88-0800 [STG] OUT ! tar: Exiting with failure status due to previous errors
2014-12-04T09:52:29.88-0800 [STG] OUT !
2014-12-04T09:52:29.90-0800 [STG] OUT Staging failed: Buildpack compilation step failed
2014-12-04T09:52:30.08-0800 [API] ERR Encountered error: App staging failed in the buildpack compile phase
2014-12-04T09:54:19.63-0800 [API] OUT Updated app with guid 8ab75c22-5c8b-411a-94ed-a82a79999ba9 ({"name"=>"web", "command"=>"PRIVATE DATA HIDDEN", "instances"=>2, "memory"=>2048, "environment_json"=>"PRIVATE DATA HIDDEN"})
2014-12-04T09:54:39.63-0800 [API] OUT Tried to stop app that never received a start event
2014-12-04T09:54:39.66-0800 [API] OUT Updated app with guid 8ab75c22-5c8b-411a-94ed-a82a79999ba9 ({"state"=>"STOPPED"})
2014-12-04T09:54:42.63-0800 [DEA] OUT Got staging request for app with id 8ab75c22-5c8b-411a-94ed-a82a79999ba9
2014-12-04T09:54:45.80-0800 [API] OUT Updated app with guid 8ab75c22-5c8b-411a-94ed-a82a79999ba9 ({"state"=>"STARTED"})
2014-12-04T09:54:46.05-0800 [STG] OUT -----> Downloaded app package (812K)
2014-12-04T09:54:48.33-0800 [STG] OUT -----> Downloaded app buildpack cache (58M)
2014-12-04T09:54:48.58-0800 [STG] ERR Cloning into '/tmp/buildpacks/ruby-buildpack'...
2014-12-04T09:54:49.56-0800 [STG] ERR Note: checking out 'b476e5a351b69e4c3acfeacdcddd6fc02de5358e'.
2014-12-04T09:54:49.57-0800 [STG] ERR You are in 'detached HEAD' state. You can look around, make experimental
2014-12-04T09:54:49.57-0800 [STG] ERR changes and commit them, and you can discard any commits you make in this
2014-12-04T09:54:49.57-0800 [STG] ERR state without impacting any branches by performing another checkout.
2014-12-04T09:54:49.57-0800 [STG] ERR If you want to create a new branch to retain commits you create, you may
2014-12-04T09:54:49.57-0800 [STG] ERR do so (now or later) by using -b with the checkout command again. Example:
2014-12-04T09:54:49.57-0800 [STG] ERR git checkout -b new_branch_name
2014-12-04T09:54:49.77-0800 [STG] OUT Submodule 'buildpack-packager' (https://github.com/cf-buildpacks/buildpack-packager.git) registered for path 'buildpack-packager'
2014-12-04T09:54:49.82-0800 [STG] ERR Cloning into 'buildpack-packager'...
2014-12-04T09:54:50.11-0800 [STG] OUT Submodule path 'buildpack-packager': checked out '87560fd0a76ccaaeb71232bf34dc616ea8876338'
2014-12-04T09:54:50.28-0800 [STG] OUT -------> Buildpack version 1.1.2
2014-12-04T09:54:50.55-0800 [STG] OUT -----> Compiling Ruby/Rails
2014-12-04T09:54:50.95-0800 [STG] OUT !
2014-12-04T09:54:50.96-0800 [STG] OUT ! Command: 'set -o pipefail; curl --fail --retry 3 --retry-delay 1 --connect-timeout 10 --max-time 90 https://s3-external-1.amazonaws.com/heroku-buildpack-ruby/ruby-2.1.3.tgz -s -o - | tar zxf - ' failed unexpectedly:
2014-12-04T09:54:50.96-0800 [STG] OUT !
2014-12-04T09:54:50.96-0800 [STG] OUT ! gzip: stdin: unexpected end of file
2014-12-04T09:54:50.96-0800 [STG] OUT ! tar: Child returned status 1
2014-12-04T09:54:50.96-0800 [STG] OUT ! tar: Exiting with failure status due to previous errors
2014-12-04T09:54:50.96-0800 [STG] OUT !
2014-12-04T09:54:50.97-0800 [STG] OUT Staging failed: Buildpack compilation step failed
2014-12-04T09:54:51.16-0800 [API] ERR Encountered error: App staging failed in the buildpack compile phase
We then tried curling the buildpack ourselves, and got a 403:
curl --fail --retry 3 --retry-delay 1 --connect-timeout 10 --max-time 90 https://s3-external-1.amazonaws.com/heroku-buildpack-ruby/ruby-2.1.3.tgz
curl: (22) The requested URL returned error: 403 Forbidden
Metadata
Metadata
Assignees
Labels
No labels
Activity
cf-gitbot commentedon Dec 4, 2014
We have created an issue in Pivotal Tracker to manage this. You can view the current status of your issue at: https://www.pivotaltracker.com/story/show/83951350.
jchesterpivotal commentedon Dec 4, 2014
Hey guys, is there any reason you're using Ruby BP 1.1.2?
BP 1.1.2 is broken for Ruby 2.1.3 and didn't include Ruby 2.1.4 at all. We added an error message, starting in 1.1.3, to say when a resource is not available in a buildpack.
Can you check whether behaviour is the same for 1.1.3 or 1.1.4?
Cheers,
JC.
jchesterpivotal commentedon Dec 4, 2014
Oh, another note. Starting with Ruby 2.1.3, Heroku changed their URL structure -- which is why Ruby BP 1.1.2 is broken for Ruby 2.1.3. If you do this:
You should see where they put the binaries now.
pivotal-feedback commentedon Dec 4, 2014
We could force use of the new buildpack, but surprised it's not just always using the latest and greatest.
Here's our current manifest.yml, note no buildpack is being specified:
jchesterpivotal commentedon Dec 4, 2014
I'm surprised too. Ruby BP 1.1.3 was in cf-release v192 and 1.1.4 was in v194.
Looks like v194 has not yet been rolled out to PWS, but I would've thought you'd get 1.1.3 and a more helpful error message in your logs.
I'm going to see if ops have an idea about it.
pivotal-feedback commentedon Dec 4, 2014
Forcing the buildpack to 1.1.3 worked. Between that AND giving ourselves more memory during deploy we now have our app running on Ruby 2.1.4!!!!!!!!!!!!!!
jchesterpivotal commentedon Dec 4, 2014
In the log the staging process is pulling Ruby BP 1.1.2 from github instead of using the PWS buildpack. My understanding is that this only happens if you specify a buildpack URL at push time.
Is there a script or rake task or something?
pivotal-feedback commentedon Dec 4, 2014
Nope, nothing fancy. It seemed like there was some sort of buildpack cache. Maybe it cached 1.1.2 when we were specifying it trying to work through issue #24
jchesterpivotal commentedon Dec 5, 2014
How am I supposed to blame the user if you're not at fault? Sheesh :)
My understanding is that re-pushing an app after a buildpack update should cause the new buildpack to be applied. I'll check with runtime if the CC decides to remember -b options even when a buildpack is updated,
jchesterpivotal commentedon Dec 5, 2014
Another question: do you have the session and log from pinning the buildpack to 1.1.3? It'd be interesting to compare to the previous output.
And did you use -b or change the manifest?
pivotal-feedback commentedon Dec 5, 2014
We changed the manifest. Here's our new manifest and logs from a successful push of one app:
Manifest:
Deploy logs of pushing the
webapp:jchesterpivotal commentedon Dec 9, 2014
OK, we spent some time today and we're able to recreate your problem.
First things first. Here's a complete transcript of two pushes to PWS, the first with a -b, the second without. We'll call out what we think is interesting in the next comment.
jchesterpivotal commentedon Dec 9, 2014
So we think this may be a Cloud Controller issue. Here's the interesting bit:
So the sequence is:
cf push some_app -b https://example.org/some/broken/buildpackcf push some_appSo at a guess it seems as though a failed staging event causes CC to not clear the associated buildpack upon a re-push.
Thanks for the report and your patience, I think we're going to refer this to Runtime for further investigation. As with #24, I will leave this issue open until Runtime advises that they have resolved the issue.
jchesterpivotal commentedon Dec 9, 2014
OK, after discussion in Pivotal Tracker with Runtime and with the CLI PM, looks like this is actually the expected behaviour for CC -- to recall buildpack settings between pushes.
We now think that this requires a UI change to help users understand that if they are pinned to a remote buildpack, they remain pinned to it upon subsequent pushes. That was the root of the behaviour you and we observed.
I'm going to open an issue with CLI, and then close this one, because it looks like a clean handoff.
jchesterpivotal commentedon Dec 9, 2014
We've logged our request here. Feel free to hop in.
Closing.
jchesterpivotal commentedon Dec 10, 2014
Looks like CLI are on top of it and 6.8.0 will report the buildpack being used.