Discussion about dropping sync support.

I'm opening this issue so that we can have a discussion about something a bit radical. 😇

Right now httpx support standard threaded concurrency, plus asyncio and trio.

I think that it may be in the projects best interests to drop threaded concurrency support completely, and focus exclusively on providing a kick-ass async HTTP client.

😨 What? Why would we do that?

The big design goals of HTTPX have been to meet two features lacking in requests...

HTTP/2 (and eventually HTTP/3) support.
Async support.

Which is great, but here's the thing... the primary motivation for HTTP/2 over HTTP/1.1 is it's ability to handle large numbers of concurrent requests. Which also really means that you should probably only care about HTTP/2 support if you're working in an async context.

For users working with the standard threaded concurrency, HTTP/2 is a shiny new headline feature, that plenty of folks will want to jump at, that isn't actully providing them with a substantial benefit.

Given that requests already provides a battle tested HTTP/1.1 client for the threaded concurrency masses, my inclination is that rather than trying to meet all possible use cases we should focus on httpx being a "the right tool for the right job" rather than "one size fits all".

Here are the benefits we'd gain from dropping sync support:

We no longer need awkward subclasses such as BaseRequest, BaseResponse that are cluttering up our API surface area.
We no longer need differing sets of type annotations for async vs. sync cases, such as async iterators vs. iterators for streaming content.
We no longer need awkward bridging code in our concurrency backends.
We can focus exclusively on providing great support for asyncio and trio.
Our documentation can focus on the async case, rather than "Here's this API, and by the way there's two different variants".
We can explain use-case motivations for choosing HTTPX much more clearly. (API gateways, backing onto HTTP/2 services, with the ability to easily service 1000's of concurrent requests. Web spidering tools with wonderful performance. etc.)

It's absolutely more of a niche (right now) than just aiming at being a requests alternative, but it's one that I'm personally far more invested in. It seems to me that we may as well embrace the split between the sync and async concurrency models, and build something that excels in one particular case, rather than trying to plaster over the differences.

So, tentatively (hopefully)... what do folks think?

Aside: It's also worth noting that the community could still perfectly well maintain an HTTPX based sync client, either using the bridging techniques we've used, or using a code-gen style approach, but that wouldn't neccessarily be an "encode" project.

Big 👍 from me. I agree 100% with embracing the async model and focusing on that. This was the reason I became interested in HTTPX in the first place.

I do think however that a lot of users have come to HTTPX looking for an alternative to requests given the recent news about its development. We'll need to consider carefully the migration path for them.

It's possible that the code-gen approach being raised in #508 could make some of this moot, since it would largely mean that our codebase could become async-only, and we'd only create a sync version at package build time. (Which would resolve some of the issues I've mentioned here anyways.)

Tho we've not really considered if the code gen approach could present race conditions in a threaded context, that are not present in our async case, because the context switching can occur anywhere, not just at await points.

I’d be perfectly fine personally on having HTTPX only provide an async interface.

I’d imagine that the high-level interface would go async, right? What I was wondering is whether HTTPX would still address the use case of “let me fire up an interpreter and make HTTP requests in the shell”. Thankfully, IPython and the 3.8+ asyncio shell would allow us to use HTTPX smoothly. And then people can just asyncio.run(httpx.get(...)) if they’re in a regular shell.

Overall, I think the benefits for both the development team and users (mainly thanks to better clarity at all levels) are very convincing. So, yup, +1 for me.

Btw, I think trying to keep a story around Requests and how easy it is to migrate would still be nice. Basically it would be “just import httpx instead, and put an await in front of all method calls.” Not sure if we should still commit to be Requests-compatible after that point, although the general approach seems to be “mostly compatible” already.

I think it's too soon (if ever?) to jettison non-async users. Much of python code are in scripts, not daemons, and I haven't seen asyncio pop into too many of those just yet. In fact I think sigalarm based concurrency is probably more prevalent then threading there.

My opinion is that HTTPX should support synchronous requests, and should implement synchronous requests without any async calls internally. Perhaps we can make synchronous requests be HTTP/1.1-only because a synchronous HTTP/2 request doesn't make much sense.

Sync code will always be used in Python for experimentation, tools, utilities, and one-time scripts and shutting out that section of users isn't going to help adoption of HTTPX. I guarantee the first thing every user does with HTTPX is use the sync requests, not the async ones.

We want HTTPX to bring a lot more to the table than just async HTTP and give those benefits to as many users as we can.

@sethmlarson

I think @sethmlarson and @toppk have a point here w.r.t. user adoption. My previous comments suffered from quick reasoning, so I'll try to pause and ponder things longer here.

It seems to me there are more fundamental questions at play here, all related to the vision we want this project to pursue:

How do we want to position HTTPX in the ecosystem landscape, in particular w.r.t. existing tooling (i.e. Requests)?
Who are our target users?
What do we want HTTPX to achieve?

Vision 1: build the go-to, rock-solid async HTTP client

Relationship with Requests: complementary, since Requests would still be used for sync use cases.
Target users: web developers using async (a growing, but still niche, use case), backend developers writing async programs such as web crawlers (definitely niche).
Goal: ?

I put "?" as the goal here because I actually can't think of a convincing one. I could have put "provide an async-capable HTTP client", but there's already one: aiohttp.

In fact, it seems the above is precisely the vision that aiohttp pursues. Not sure it succeeded there — partly because it tries to do too much, and people need to learn a different API from that of Requests — so maybe there's room for an alternative.

But by pursuing this vision, I suspect we'd end up in the same situation, and we won't truly shake up the status quo. 80-90% of Python users will continue using Requests because they don't need async, and thus don't want to (shouldn't have to?) deal with the associated cognitive overhead.

Vision 2: build the modern, async-capable successor for Requests

Relationship with Requests: exclusive, i.e. users that use HTTPX will stop using Requests.
Target users: anyone who wants to make HTTP requests in modern Python: web developers, backend developers, data scientists, prototypers, etc. Again, realistically only a fraction of these users will use async.
Goal: eventually replace Requests, i.e. become the de-facto HTTP client for both sync and async use cases in Python 3.6+. In other words: break the status quo for users of modern Python while sticking to a familiar API.

Some data

According to the Python Developers Survey 2018:

58% of respondants use Python for data analysis
50% for web development
43% for DevOps/sysadmin/automation
39% for ML
38% for web parsers/scrapers/crawlers.

Vision 1 would only allow us to address 1) and 5), while vision 2 addresses all use cases, because most of them just need a quick (synchronous) way to make HTTP requests.

Also, among Python users:

16% are on 2.7 — we aren't targetting them either way.
84% are on 3.x
- 84% of these 3.x users are on 3.6+.

So, here we go: either we address and solve problems for 84% of all Python 3 users, or we do so for a small fraction of them (a fraction that will be even smaller because of the friction induced by async).

My +1 from a previous comment was probably too centered on my position as a maintainer. There is indeed a lot of (nearly-duplicated) code that's solely targeted at supporting sync usage. But shouldn't the ultimate goal be to serve our users well, at the cost of the implementation being not as elegant as it could because of the hard dichotomy between the sync and async worlds?

I see the fact that people doing web development and data science (see #508) are starting to adopt HTTPX as a hint that the community is seeking the modern Requests successor we've been shaping up. It's probably most sensible to support this trend, and address a need that would otherwise stay unsatisfied until another project takes over, and goes through the same process we've been going through in the past few months.

I'd now say +1 to @sethmlarson suggestions on ensuring that the sync client is as purely sync as possible (no async calls, HTTP/1.1-only). #525 is a good first step towards that.

I could have put "provide an async-capable HTTP client", but there's already one: aiohttp.

It seems to me that Python users looking for a synchronous requests are already well served by... requests and they can continue to use it indefinitely.

I'm just popping up as a user who's looking for a the best async http client. API aside, aiohttp doesn't fit my needs as I'm after something which works with trio. The competition here is asks rather than aiohttp and the decision I have to make is which one is better for my usecase and which will be around for the long term (both are very young projects)

With IPython's async repl and native support in Python 3.8 I don't think mucking around with an async api necessarily introduces that much friction these days.

Anyway, just popping up here to provide a user perspective FWIW. I got interested in httpx as an option when I saw tomchristie pop up on the trio gitter...

It seems to me that Python users looking for a synchronous requests are already well served by... requests and they can continue to use it indefinitely.

Exactly. Use the right tool, for the right job.

As much as the project would clearly be more popular if it covers both cases, that's actually more just a "ohh... shiny new things" effect. There isn't actually any genuine pragmatic benefit in us re-implementing "requests".

The grunge of bridging across to supporting threaded concurrency makes the project significantly less comprehesible when digging into it, and I think it's holding us back from excelling as "the high performance HTTP client" option.

I'm way more invested in the potential of async Python in helping the langauge remain competative against Node and Go, even if it is currently a niche ecosystem.

I'd personally rather see us completely nail HTTP for asyncio and trio, and only then figure out if and how we want to extended that to threaded concurrency.

- And no, neither of the existing candidates fits the bill. I don't think aiohttp or asks covers HTTP/2, and aiohttp happens to come coupled to a server framework.

Right, I see my analysis of other async HTTP clients was quite incomplete. There's indeed a lot more that HTTPX brings to the table relative to e.g. aiohttp (HTTP/2, not-just-asyncio, not-also-a-server-framework, familiar Requests-like API, etc).

To my mind, both options make sense, with different objectives and trade-offs, and I agree that pushing for excellence by keeping the scope down is attractive. It's also an approach that's consistent with the general push for focus and modularity across Encode projects.

I wrote previously that going async-only would not break status quo as much as trying to address both sync and async. Actually, it probably would: as you said there's currently no other option that compares with what we have, even today.

And it would make our lives as maintainers and contributors easier, because the scope would be narrowed-down, resulting in a multitude of issues today becoming non-issues tomorrow (e.g. everything that's related to using HTTPX in a sync context).

So, hmm, I think it's a vision I'd be willing to support, yup — as long as we make it as loud and clear as possible to our users. Example re-worded README headline:

**HTTPX** - High performance HTTP client for Python

> Note: HTTPX is meant to be used in programs that use `async`/`await`. If you don't use async, you should probably use [Requests](https://github.com/psf/requests/).

Features:

- Requests-compatible API.
- ...

From the outside, as someone who's been watching and has on the list to pick up HTTPX just the very next time I would have reached for requests, I want a sync wrapper, even if just to start with, so I don't have to spin up an event loop just for a get(...). As long as it's got that...

(Because the real reason is, "Yes, I probably should write this as async" — but I don't want to change library just to do that.)

Hope that's useful/makes sense.

(Because the real reason is, "Yes, I probably should write this as async" — but I don't want to change library just to do that.)

☝️ Couldn't have put it better myself.

@sethmlarson

Why not best of both worlds? Async-only but with sync versions without the threading nonsense. Expose sync functions which make event loops temporarily and run the request in that (Don't bind that to the current thread). It works very well in my opinion. This way, the users calling the sync functions don't even notice they are calling coroutines. So I strongly disagree with @sethmlarson, because I think this approach works really well. Has some overhead (starting a new event loop every time) but probably still better than starting threads!
What's even better is that the users calling the "sync" versions will benefit the async advantages, because they are running them almost the same way as if they were running an event loop themselves.
The only gotcha I think of it has to be documented, because if users are running an event loop, but calling the sync version, there might be problems, but I would just raise an Exception in that case, not a valid use-case IMO.

My use-case with this approach was that I wrote a Slack bot in fully-async style, but I needed to call some method from Django, sometimes even running event loops in different threads. But it was very easy to understand and write. Here is a concrete example.

This is all the code I needed to "convert" the async methods to sync:
https://github.com/kissgyorgy/gerrit-slack-bot/blob/ba83dce88ad8a57818204ca566389fa8a36d9140/slack.py#L250-L272

class Api(AsyncApi):
    def __init__(self, token):
        try:
            self._loop = asyncio.get_event_loop()
        except RuntimeError:
            # when running in a thread, get_event_loop doesn't create another one
            self._loop = asyncio.new_event_loop()
            asyncio.set_event_loop(self._loop)

        self._session = aiohttp.ClientSession(loop=self._loop)
        super().__init__(token, self._session)

    @lru_cache(maxsize=None)
    def __getattribute__(self, name):
        attr = super().__getattribute__(name)
        if name.startswith("_") or not asyncio.iscoroutinefunction(attr):
            return attr

        def call_sync(*args, **kwargs):
            coro = attr(*args, **kwargs)
            return self._loop.run_until_complete(coro)

        return call_sync

Then I can simply call them from Django:
https://github.com/kissgyorgy/gerrit-slack-bot/blob/ba83dce88ad8a57818204ca566389fa8a36d9140/web/slackbot/models.py#L67-L68

slack_api = slack.Api(config.BOT_ACCESS_TOKEN)
slack_api.delete_message(self.channel_id, self.ts)

Maybe there could be a global event loop when a sync version is first called, so the overhead of creating one is not that bad, because it only have to do once.

I support dropping sync support. Not as a permanent "mission statement", but atleast till the foreseeable future.

I am looking at this from two angles - usage and effort. I'm primarily looking at this from the point of view of existing production codebases and not really start-a-new-project.

Async and sync usage dont generally intersect, so most people who are already using requests will most likely not move anytime soon, thereby reducing the usage of that part of httpx. People who will use httpx will only use it for async usecases.
This directly impacts the effort to support sync usecase by the small team of developers (to whom we are grateful for). There is just not enough ROI.

Another happy side-effect that I hope will come about is that the new async frameworks (Starlette, Fastapi, etc) will better support requests ... thereby improving the migration path from sync frameworks. So there's a larger tie in.

Done, and released as 0.8. 😬

I'd can see us coming back to covering sync at a later date, but I just don't dig our approach onto it right now - it's making the API & implementation significantly poorer.

@tomchristie

Haha. Was wondering why my calls to AsyncClient broke. Good change though.

Thank you @tomchristie

Right, the project is looking much better already, with our bridging approach to sync+async out of the way.

I think we've got a much better tack onto this on the horizon...

✨ "Supporting Sync. Done right." #572 ✨

The big, big, big difference with #572, is that it won't impact the quality of the codebase in the same way that our prior approach was doing. We'll also be using regular ol' function calls and sync socket operations in the sync case, rather than hiding async stuff underneath a sync facade.

@sethmlarson

I'd just like to make a point here that @sethmlarson assumed that a "synchronous HTTP/2 request doesn't make much sense".
My use case is that I have a endless http/2 stream that I cannot use with await. I need to read it synchronously.
It's a stream of events that generates further synchronous http requests that need to be performed before the next event is read off the stream. Having a 100% synchronous interface with an async backend was perfect for my use case.
So basically the concept is having multiple streams open, but pausing them on the fly to switch to reading from a different stream.
Forcing everything to be purely async makes the design very hard to work with as it now requires all the async to sync implementation to be rehashed, something that was previously be handled by the library.

I don't understand exactly why is this a problem for you, because I don't see your code, but would asyncio.run(handle_stream_event) not work for you? So basically, you would not run the asyncio loop all the time, just for the short period of reading one event at a time. It's actually pretty simple to control the event loop yourself. If you can show code I would take a look.

@kissgyorgy

@kissgyorgy
https://github.com/phillmac/py-orbit-db-http-client-dev/blob/b785cc3289ece70d797f672742d7cfc0a2bc140e/tests/test.py#L159
https://github.com/phillmac/py-orbit-db-http-client-dev/blob/b785cc3289ece70d797f672742d7cfc0a2bc140e/tests/test.py#L170

The problem is I need to consume events until I get the one I want, do some stuff, consume some more events, then do some more stuff. I don't want to have to go fully async just to pause the stream

See also https://github.com/phillmac/py-orbit-db-http-client-dev/blob/b785cc3289ece70d797f672742d7cfc0a2bc140e/orbitdbapi/client.py#L97

Fix for httpx 0.8.

you should probably only care about HTTP/2 support if you're working in an async context.
For users working with the standard threaded concurrency, HTTP/2 is a shiny new headline feature, that plenty of folks will want to jump at, that isn't actully providing them with a substantial benefit.

we can make synchronous requests be HTTP/1.1-only because a synchronous HTTP/2 request doesn't make much sense.

Python users looking for a synchronous requests are already well served by... requests and they can continue to use it indefinitely

(note for posterity)
Regardless of whether HTTP/2 provides synchronous benefits, my use-case is simply calling an external service that surprisingly blocks HTTP/1 (w/ 464s), so it's worth highlighting that many devs simply work with services they do not control

Navigation Menu

Repository navigation

Discussion about dropping sync support. #522

Vision 1: build the go-to, rock-solid async HTTP client

Vision 2: build the modern, async-capable successor for Requests

Some data

22 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Discussion about dropping sync support. #522

Description

Activity

lovelydinosaur commented on Nov 11, 2019

yeraydiazdiaz commented on Nov 11, 2019

lovelydinosaur commented on Nov 11, 2019

lovelydinosaur commented on Nov 11, 2019

florimondmanca commented on Nov 11, 2019

florimondmanca commented on Nov 11, 2019

toppk commented on Nov 13, 2019

sethmlarson commented on Nov 13, 2019

florimondmanca commented on Nov 13, 2019

Vision 1: build the go-to, rock-solid async HTTP client

Vision 2: build the modern, async-capable successor for Requests

Some data

dhirschfeld commented on Nov 14, 2019

lovelydinosaur commented on Nov 14, 2019

florimondmanca commented on Nov 14, 2019

carltongibson commented on Nov 14, 2019

sethmlarson commented on Nov 14, 2019

kissgyorgy commented on Nov 14, 2019

22 remaining items

sandys commented on Nov 16, 2019

lovelydinosaur commented on Nov 27, 2019

gnat commented on Nov 28, 2019

lovelydinosaur commented on Nov 30, 2019

phillmac commented on Dec 1, 2019

kissgyorgy commented on Dec 1, 2019

phillmac commented on Dec 1, 2019

phillmac commented on Dec 1, 2019

PatMyron commented on Dec 5, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions