Caching support (was: Conditional Gets)

Would be nice to have some form of Response caching here, with a simple Hash Store and Memcache API Store, seeing how Github itself doesn't count successfull Conditional Gets towards the API limit.

@sferik

Here's a library to look to for ideas:

https://github.com/paul/resourceful

I think this would be a cool feature to have, particularly if it were opt-in and off by default (so as not to affect what @sferik is doing ;)

Another one (using Celluloid):

https://github.com/paul/hippie

Finally took the time to look at resourceful, looks like it has everything "http" misses and has a adapter API, guess I'll just write one for the http gem :)

FWIW, Resourceful is not really maintained anymore. If it works for you, great, otherwise I'd be happy to have similar features built into the HTTP gem itself.

@tarcieri

Made a rough caching implementation at https://github.com/Asmod4n/http/compare/caching, could need some help with docs and specs ^^

@tarcieri @ixti @sferik

Am having issues unrelated to this patch, redirects write headers servers don't seam to understand.
HTTP.with_follow(true).get('http://google.com/')

it sends

GET /foo HTTP/1.1
User-Agent: RubyHTTPGem/0.6.0.pre
Host: ["localhost", "localhost"]
Date: ["Thu, 27 Feb 2014 13:19:11 GMT", "Thu, 27 Feb 2014 13:19:14 GMT"]
Content-Length: 3

1.1

upon redirects oO

I guess ill just read rfc2616-sec13 and rfc2616-sec14.9 and implement it on my own without looking elsewhere, resourceful implements it in a way at least i cannot validate with what is written in rfc2616.

For ex: http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.9.2 says "no-store" send from origin servers must delete the cache entry, resourceful only checks for it in requests.

@Asmod4n

@Asmod4n I like the idea of implementing it RFC-compliant. About redirects - it's a defect indeed, thanks for mentioning that.

Upon reading the spec sections twice i belive its mostly spec compliant, there is just one catch.

When a response doesn't include max-age or Expire headers it will be cached forever, guess that needs a extra section in the README so request.headers['Cache-Control'] include a max-age value when you don't want something cached forever.

Its especially important to note when HTTP is used as a API Client, some services properly speak http (github for instance) so you can cache their responses, while others ( etcd ) don't.

What is mostly missing now are comments and specs, i can add the former over time, but desperately need help with specs, never wrote them before and my free time is rather limited for the rest of the year.

@Asmod4n

@Asmod4n defect with Host header upon redirects was fixed in master btw.

@Asmod4n

@Asmod4n If I'll find time over next week, I'll try to help, but not promising anything. Would like to propose you following:

rebase your branch over current master Request.new signature was changed and include proxy options now.
think to combine cache_mode and cache_adapter settings into a single one, e.g.

HTTP.with_cache(:public) # Implicits :adapter => InMemory
HTTP.with_cache(:mode => :public, :adapter => MyAdapter)
HTTP.with_cache(false) # disables cache

@ixti

@ixti got the latest master changes 3 hours ago.

The InMemory Adapter Instance has to be a class variable atm, or else every new request will use a new instance of the adapter, rendering it useless.

@Asmod4n

@Asmod4n well default adapter does not needs to be class exactly. It was a simplification. So I assume it might be something like this: InMemoryAdapter.current:

class HTTP::Cache::InMemoryAdapter
  # Default instance of cache adapter
  def self.current
    @current ||= new
  end
end

Could someone else take https://github.com/Asmod4n/http/compare/caching over? I am afraid i don't use http.rb anymore and haven't found fixes for some pitfalls for caching (attackers can allocate any amount of ram they want with certain conditions).

A Faraday Adapter would be good, but there are gems which don't even work with faraday if you don't use Net::HTTP (octokit.rb doesn't work with typhoeus for instance)

@Asmod4n

@Asmod4n I don't think I will be able to jump anytime soon on this issue, but I would like to keep it open if anybody would like to jump on. I have merged your caching branch of your repo into wip/caching-support on this repo.

@ixti

@ixti thanks a lot, i will try to take a look at it after i have read the new HTTP rfcs regarding caching.

Closed by #177 which is in master and was released as 0.8.0.pre

@ixti

@ixti This issue should also be moved into the 0.8.0 milestone. Are you able to do that? I’m still getting an error.

@sferik

@sferik Same here.

Navigation Menu

Repository navigation

Caching support (was: Conditional Gets) #77

28 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Caching support (was: Conditional Gets) #77

Description

Activity

tarcieri commented on Feb 3, 2014

tarcieri commented on Feb 3, 2014

Asmod4n commented on Feb 20, 2014

tarcieri commented on Feb 20, 2014

Asmod4n commented on Feb 27, 2014

Asmod4n commented on Feb 27, 2014

Asmod4n commented on Feb 27, 2014

Asmod4n commented on Feb 28, 2014

ixti commented on Feb 28, 2014

Asmod4n commented on Mar 18, 2014

ixti commented on Mar 18, 2014

ixti commented on Mar 18, 2014

Asmod4n commented on Mar 18, 2014

ixti commented on Mar 18, 2014

28 remaining items

Asmod4n commented on Jan 2, 2015

ixti commented on Jan 2, 2015

Asmod4n commented on Jan 2, 2015

ixti commented on Mar 27, 2015

sferik commented on Mar 27, 2015

ixti commented on Mar 27, 2015

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions