Repository navigation
Can't push new app with ruby-buildpack v1.1.3 #24
Description
Ruby version: 2,.1.3
Org: Pivotal Feedback
Space: staging
We're trying to push a new scheduler/cron app for our existing web app. The push fails during the bundle step with the following message:
FAILED
Server error, status code: 400, error code: 170001, message: Staging error: cannot get instances since staging failed
The commit we're pushing is https://github.com/pivotal/pivotal-feedback/commit/1909f0421933735e84a7784c53d056053f699616
We deleted one of our apps that was initially deployed using ruby-buildpack 1.1.2, then redeployed and got the same failure during push.
Dropping the ruby version down to 2.1.1, and specifying ruby-buildpack 1.1.2 allows us to deploy without issues.
Activity
cf-gitbot commentedon Nov 14, 2014
We have created an issue in Pivotal Tracker to manage this. You can view the current status of your issue at: https://www.pivotaltracker.com/story/show/82704022.
jchesterpivotal commentedon Nov 14, 2014
Hi Will;
Ruby 2.1.3 is broken on buildpack 1.1.2, because of an unexpected change in the URLs of upstream binaries.
Buildpack 1.1.3 resolved this issue and added code to make the cause of errors more visible. It was bundled with cf-release v192.
Our releases page on github shows the exciting story of how it unfolded. :)
Cheers,
JC.
CardboardMechanic commentedon Nov 14, 2014
Thank you for the info about Ruby 2.1.3 on buildpack 1.1.2, but our issue is different. Yesterday, around 3pm PT we could not push a new app using Ruby 2.1.3 and buildpack 1.1.3. It failed during bundle install. Please reopen this issue.
jchesterpivotal commentedon Nov 14, 2014
Sorry, I got confused. I was looking at the text of the original report, not the title.
What do you get from cf logs?
CardboardMechanic commentedon Nov 14, 2014
Here's the push, followed by cf logs
jchesterpivotal commentedon Nov 14, 2014
Thanks.
To start, can you try vendoring gems before pushing? (They don't need to be checked into git, just physically present). In the past we've found apps with lots of dependencies failing at push time due to rubygems.org flakiness.
Second, if that doesn't work, can you point us to a repo? Like you, I don't see any other obvious errors in the log.
Cheers,
JC.
CardboardMechanic commentedon Nov 14, 2014
Running
bundle packageand then pushing a new app with Ruby 2.1.3 and ruby-buildpack 1.1.3 does indeed work. I don't think it's fair to attribute this to rubygems.org flakiness. 1.1.3 failed every time on bundling, while 1.1.2 pushed successfully during the same periods of time. I ran some more experiments for you:The repo/commit we're using were referenced in the original comment for this issue, as a Labs pivot you should have access to that repo.
jchesterpivotal commentedon Nov 14, 2014
That's very, very interesting, thanks for that fault isolation.
It makes me suspicious of the code we included to change fetching behaviour depending on the requested Ruby version (since Heroku changed binary URLs starting with 2.1.3).
jchesterpivotal commentedon Nov 14, 2014
(and I wonder, if the bad workman blames his tools, what do you call the software dev who blames someone else's tools?)
jchesterpivotal commentedon Nov 17, 2014
OK, our working hypothesis is this:
There are several workarounds that we've identified, but the best is to increase the amount of RAM allocated to your application using
cf scale.@TildeWill can you try this and verify that it's a successful workaround in your case?
CardboardMechanic commentedon Nov 18, 2014
We tried increasing the memory early on in this process from 512M to 1G - deploys still failed the same way.
jchesterpivotal commentedon Nov 18, 2014
Hmm.
Was the log you gave above for 512M or 1G?
Did you
cf restateorcf pushagain?Cheers,
JC.
CardboardMechanic commentedon Nov 19, 2014
Log above was @512m. All tests have been
cf pushto a new app,jchesterpivotal commentedon Nov 19, 2014
Thanks.
FYI, we've transferred the Tracker story for this issue to CF Runtime to look at. The 255 error suggests Warden is the component faulting and we just don't have much expertise on that component in Buildpacks.
In the meantime, given that you've already tried scaling up, I'm not sure what workaround to offer, except pinning to 2.1.1 as you are currently. Sorry :|
We'll leave this open pending advice from Runtime.
CardboardMechanic commentedon Dec 2, 2014
Sounded like Runtime looked at this, and concluded the same thing you did - that app instance memory had to be increased to 2G. What is troubling is that the app only needs ~256MB to run... why does the size of the running app impact the size of the stager? It seems like the stager should get as much memory as is needed to build a droplet. Another question is why does a smaller memory allocation work for Ruby 2.1.1, but not 2.1.3?
jchesterpivotal commentedon Dec 2, 2014
Our working hypothesis was that the compilation of Nokogiri on 2.1.3 triggered the OOM condition. The reason we threw it over the fence to Runtime was basically that we were out of our depth on Warden's OOM behaviour.
I'm not sure if we can predict staging-time memory requirements in advance. So the second reason I wanted to get Runtime involved was because I think that we need to give more explicit, visible feedback to developers about why an application has failed to stage. That's something I worry about in Buildpacks land, and we've shipped lots of small changes over the past few months to that end.
So maybe the next question is: why can't we separate buildpack staging and app production settings for RAM etc?
As before, leaving open pending further advice from Runtime.
flavorjones commentedon Dec 2, 2014
Hi there!
We use Pivotal Tracker to provide visibility into what our team is working on. A story for this issue has been automatically created.
The current status is as follows:
This comment, as well as the labels on the issue, will be automatically updated as the status in Tracker changes.
jfmyers9 commentedon Dec 3, 2014
Hi all,
Sorry for not responding to the Github issue. We left a comment on the tracker story here explaining what might be the cause of the issues seen here. We are planning on investigating this behavior pending prioritization. Thanks.
@jfmyers9 && @mhess, CF Runtime Team