From 9785ec2d842e0b6067ade95eb24828a4bf624600 Mon Sep 17 00:00:00 2001 From: Tom Christie Date: Tue, 9 Jan 2024 11:27:06 +0000 Subject: [PATCH 01/14] Tweak docs layout --- mkdocs.yml | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/mkdocs.yml b/mkdocs.yml index c0ccd80566..840af03846 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -16,8 +16,6 @@ theme: toggle: icon: 'material/lightbulb-outline' name: 'Switch to light mode' - features: - - navigation.sections repo_name: encode/httpx repo_url: https://github.com/encode/httpx/ @@ -25,9 +23,9 @@ edit_uri: "" nav: - Introduction: 'index.md' - - Usage: - - QuickStart: 'quickstart.md' - - Advanced Usage: 'advanced.md' + - QuickStart: 'quickstart.md' + - Clients: + - Overview: 'advanced.md' - Guides: - Async Support: 'async.md' - HTTP/2 Support: 'http2.md' From 6fb27d0c24456508ba7bc9e5def20b476ed1a60c Mon Sep 17 00:00:00 2001 From: Tom Christie Date: Tue, 9 Jan 2024 11:30:28 +0000 Subject: [PATCH 02/14] Move client docs into folder --- docs/{advanced.md => clients/overview.md} | 0 mkdocs.yml | 2 +- 2 files changed, 1 insertion(+), 1 deletion(-) rename docs/{advanced.md => clients/overview.md} (100%) diff --git a/docs/advanced.md b/docs/clients/overview.md similarity index 100% rename from docs/advanced.md rename to docs/clients/overview.md diff --git a/mkdocs.yml b/mkdocs.yml index 840af03846..73b9511cca 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -25,7 +25,7 @@ nav: - Introduction: 'index.md' - QuickStart: 'quickstart.md' - Clients: - - Overview: 'advanced.md' + - Overview: 'clients/overview.md' - Guides: - Async Support: 'async.md' - HTTP/2 Support: 'http2.md' From 7bb920d776a2f47112fd95e0c0dd421707f6ec0b Mon Sep 17 00:00:00 2001 From: Tom Christie Date: Tue, 9 Jan 2024 11:52:21 +0000 Subject: [PATCH 03/14] Add clients/authentication section --- docs/clients/authentication.md | 225 +++++++++++++++++++++++++++++++++ docs/clients/overview.md | 191 ---------------------------- mkdocs.yml | 1 + 3 files changed, 226 insertions(+), 191 deletions(-) create mode 100644 docs/clients/authentication.md diff --git a/docs/clients/authentication.md b/docs/clients/authentication.md new file mode 100644 index 0000000000..f5be1b1af4 --- /dev/null +++ b/docs/clients/authentication.md @@ -0,0 +1,225 @@ +```pycon +>>> auth = ... +>>> response = httpx.get("https://www.example.com/", auth=auth) +``` + +```pycon +>>> auth = ... +>>> client = httpx.Client(auth=auth) +>>> response = client.get("https://www.example.com/") +``` + +## Basic authentication + +HTTP basic authentication is an unencrypted authentication scheme that uses a simple encoding of the username and password in the request `Authorization` header. Since it is unencrypted it should only be used over `https`, although this is not strictly enforced. + +```pycon +>>> auth = httpx.BasicAuthentication(username="user@example.com", password="password123") +>>> response = httpx.get("https://www.example.com/", auth=auth) +>>> response + +``` + +## Digest authentication + +HTTP digest authentication is a challenge-response authentication scheme. Unlike basic authentication it provides encryption **TODO** + +```pycon +>>> auth = httpx.DigestAuth(username="user@example.com", password="password123") +>>> response = httpx.get("https://example.com/", auth=auth) +>>> response + +>>> response.history +... # TODO +``` + +## NetRC authentication + +HTTPX can be configured to use [a `.netrc` config file](https://everything.curl.dev/usingcurl/netrc) for authentication. + +The `.netrc` config file allows authentication credentials to be associated with specified hosts. When a request is made to a host that is found in the netrc file, the username and password will be included using HTTP basic auth. + +Example `.netrc` file: + +``` +machine example.org +login example-username +password example-password + +machine python-httpx.org +login other-username +password other-password +``` + +Some examples of configuring `.netrc` authentication with `httpx`. + +Use the default `.netrc` file in the users home directory: + +```pycon +>>> auth = httpx.NetRCAuth() +>>> client = httpx.Client(auth=auth) +``` + +Use an explicit path to a `.netrc` file: + +```pycon +>>> auth = httpx.NetRCAuth(file="/path/to/.netrc") +>>> client = httpx.Client(auth=auth) +``` + +Use the `NETRC` environment variable to configure a path to the `.netrc` file, +or fallback to the default. + +```pycon +>>> auth = httpx.NetRCAuth(file=os.environ.get("NETRC")) +>>> client = httpx.Client(auth=auth) +``` + +The `NetRCAuth()` class uses [the `netrc.netrc()` function from the Python standard library](https://docs.python.org/3/library/netrc.html). See the documentation there for more details on exceptions that may be raised if the netrc file is not found, or cannot be parsed. + +## Custom authentication schemes + +When issuing requests or instantiating a client, the `auth` argument can be used to pass an authentication scheme to use. The `auth` argument may be one of the following... + +* A two-tuple of `username`/`password`, to be used with basic authentication. +* An instance of `httpx.BasicAuth()`, `httpx.DigestAuth()`, or `httpx.NetRCAuth()`. +* A callable, accepting a request and returning an authenticated request instance. +* An instance of subclasses of `httpx.Auth`. + +The most involved of these is the last, which allows you to create authentication flows involving one or more requests. A subclass of `httpx.Auth` should implement `def auth_flow(request)`, and yield any requests that need to be made... + +```python +class MyCustomAuth(httpx.Auth): + def __init__(self, token): + self.token = token + + def auth_flow(self, request): + # Send the request, with a custom `X-Authentication` header. + request.headers['X-Authentication'] = self.token + yield request +``` + +If the auth flow requires more than one request, you can issue multiple yields, and obtain the response in each case... + +```python +class MyCustomAuth(httpx.Auth): + def __init__(self, token): + self.token = token + + def auth_flow(self, request): + response = yield request + if response.status_code == 401: + # If the server issues a 401 response then resend the request, + # with a custom `X-Authentication` header. + request.headers['X-Authentication'] = self.token + yield request +``` + +Custom authentication classes are designed to not perform any I/O, so that they may be used with both sync and async client instances. If you are implementing an authentication scheme that requires the request body, then you need to indicate this on the class using a `requires_request_body` property. + +You will then be able to access `request.content` inside the `.auth_flow()` method. + +```python +class MyCustomAuth(httpx.Auth): + requires_request_body = True + + def __init__(self, token): + self.token = token + + def auth_flow(self, request): + response = yield request + if response.status_code == 401: + # If the server issues a 401 response then resend the request, + # with a custom `X-Authentication` header. + request.headers['X-Authentication'] = self.sign_request(...) + yield request + + def sign_request(self, request): + # Create a request signature, based on `request.method`, `request.url`, + # `request.headers`, and `request.content`. + ... +``` + +Similarly, if you are implementing a scheme that requires access to the response body, then use the `requires_response_body` property. You will then be able to access response body properties and methods such as `response.content`, `response.text`, `response.json()`, etc. + +```python +class MyCustomAuth(httpx.Auth): + requires_response_body = True + + def __init__(self, access_token, refresh_token, refresh_url): + self.access_token = access_token + self.refresh_token = refresh_token + self.refresh_url = refresh_url + + def auth_flow(self, request): + request.headers["X-Authentication"] = self.access_token + response = yield request + + if response.status_code == 401: + # If the server issues a 401 response, then issue a request to + # refresh tokens, and resend the request. + refresh_response = yield self.build_refresh_request() + self.update_tokens(refresh_response) + + request.headers["X-Authentication"] = self.access_token + yield request + + def build_refresh_request(self): + # Return an `httpx.Request` for refreshing tokens. + ... + + def update_tokens(self, response): + # Update the `.access_token` and `.refresh_token` tokens + # based on a refresh response. + data = response.json() + ... +``` + +If you _do_ need to perform I/O other than HTTP requests, such as accessing a disk-based cache, or you need to use concurrency primitives, such as locks, then you should override `.sync_auth_flow()` and `.async_auth_flow()` (instead of `.auth_flow()`). The former will be used by `httpx.Client`, while the latter will be used by `httpx.AsyncClient`. + +```python +import asyncio +import threading +import httpx + + +class MyCustomAuth(httpx.Auth): + def __init__(self): + self._sync_lock = threading.RLock() + self._async_lock = asyncio.Lock() + + def sync_get_token(self): + with self._sync_lock: + ... + + def sync_auth_flow(self, request): + token = self.sync_get_token() + request.headers["Authorization"] = f"Token {token}" + yield request + + async def async_get_token(self): + async with self._async_lock: + ... + + async def async_auth_flow(self, request): + token = await self.async_get_token() + request.headers["Authorization"] = f"Token {token}" + yield request +``` + +If you only want to support one of the two methods, then you should still override it, but raise an explicit `RuntimeError`. + +```python +import httpx +import sync_only_library + + +class MyCustomAuth(httpx.Auth): + def sync_auth_flow(self, request): + token = sync_only_library.get_token(...) + request.headers["Authorization"] = f"Token {token}" + yield request + + async def async_auth_flow(self, request): + raise RuntimeError("Cannot use a sync authentication class with httpx.AsyncClient") +``` \ No newline at end of file diff --git a/docs/clients/overview.md b/docs/clients/overview.md index bb003a1a52..b3f756ddfb 100644 --- a/docs/clients/overview.md +++ b/docs/clients/overview.md @@ -458,50 +458,6 @@ httpx.post("https://httpbin.org/post", content=gen()) ![tqdm progress bar](img/tqdm-progress.gif) -## .netrc Support - -HTTPX can be configured to use [a `.netrc` config file](https://everything.curl.dev/usingcurl/netrc) for authentication. - -The `.netrc` config file allows authentication credentials to be associated with specified hosts. When a request is made to a host that is found in the netrc file, the username and password will be included using HTTP basic auth. - -Example `.netrc` file: - -``` -machine example.org -login example-username -password example-password - -machine python-httpx.org -login other-username -password other-password -``` - -Some examples of configuring `.netrc` authentication with `httpx`. - -Use the default `.netrc` file in the users home directory: - -```pycon ->>> auth = httpx.NetRCAuth() ->>> client = httpx.Client(auth=auth) -``` - -Use an explicit path to a `.netrc` file: - -```pycon ->>> auth = httpx.NetRCAuth(file="/path/to/.netrc") ->>> client = httpx.Client(auth=auth) -``` - -Use the `NETRC` environment variable to configure a path to the `.netrc` file, -or fallback to the default. - -```pycon ->>> auth = httpx.NetRCAuth(file=os.environ.get("NETRC")) ->>> client = httpx.Client(auth=auth) -``` - -The `NetRCAuth()` class uses [the `netrc.netrc()` function from the Python standard library](https://docs.python.org/3/library/netrc.html). See the documentation there for more details on exceptions that may be raised if the netrc file is not found, or cannot be parsed. - ## HTTP Proxying HTTPX supports setting up [HTTP proxies](https://en.wikipedia.org/wiki/Proxy_server#Web_proxy_servers) via the `proxy` parameter to be passed on client initialization or top-level API functions like `httpx.get(..., proxy=...)`. @@ -738,153 +694,6 @@ For instance this request sends 2 files, `foo.png` and `bar.png` in one request >>> r = httpx.post("https://httpbin.org/post", files=files) ``` -## Customizing authentication - -When issuing requests or instantiating a client, the `auth` argument can be used to pass an authentication scheme to use. The `auth` argument may be one of the following... - -* A two-tuple of `username`/`password`, to be used with basic authentication. -* An instance of `httpx.BasicAuth()`, `httpx.DigestAuth()`, or `httpx.NetRCAuth()`. -* A callable, accepting a request and returning an authenticated request instance. -* An instance of subclasses of `httpx.Auth`. - -The most involved of these is the last, which allows you to create authentication flows involving one or more requests. A subclass of `httpx.Auth` should implement `def auth_flow(request)`, and yield any requests that need to be made... - -```python -class MyCustomAuth(httpx.Auth): - def __init__(self, token): - self.token = token - - def auth_flow(self, request): - # Send the request, with a custom `X-Authentication` header. - request.headers['X-Authentication'] = self.token - yield request -``` - -If the auth flow requires more than one request, you can issue multiple yields, and obtain the response in each case... - -```python -class MyCustomAuth(httpx.Auth): - def __init__(self, token): - self.token = token - - def auth_flow(self, request): - response = yield request - if response.status_code == 401: - # If the server issues a 401 response then resend the request, - # with a custom `X-Authentication` header. - request.headers['X-Authentication'] = self.token - yield request -``` - -Custom authentication classes are designed to not perform any I/O, so that they may be used with both sync and async client instances. If you are implementing an authentication scheme that requires the request body, then you need to indicate this on the class using a `requires_request_body` property. - -You will then be able to access `request.content` inside the `.auth_flow()` method. - -```python -class MyCustomAuth(httpx.Auth): - requires_request_body = True - - def __init__(self, token): - self.token = token - - def auth_flow(self, request): - response = yield request - if response.status_code == 401: - # If the server issues a 401 response then resend the request, - # with a custom `X-Authentication` header. - request.headers['X-Authentication'] = self.sign_request(...) - yield request - - def sign_request(self, request): - # Create a request signature, based on `request.method`, `request.url`, - # `request.headers`, and `request.content`. - ... -``` - -Similarly, if you are implementing a scheme that requires access to the response body, then use the `requires_response_body` property. You will then be able to access response body properties and methods such as `response.content`, `response.text`, `response.json()`, etc. - -```python -class MyCustomAuth(httpx.Auth): - requires_response_body = True - - def __init__(self, access_token, refresh_token, refresh_url): - self.access_token = access_token - self.refresh_token = refresh_token - self.refresh_url = refresh_url - - def auth_flow(self, request): - request.headers["X-Authentication"] = self.access_token - response = yield request - - if response.status_code == 401: - # If the server issues a 401 response, then issue a request to - # refresh tokens, and resend the request. - refresh_response = yield self.build_refresh_request() - self.update_tokens(refresh_response) - - request.headers["X-Authentication"] = self.access_token - yield request - - def build_refresh_request(self): - # Return an `httpx.Request` for refreshing tokens. - ... - - def update_tokens(self, response): - # Update the `.access_token` and `.refresh_token` tokens - # based on a refresh response. - data = response.json() - ... -``` - -If you _do_ need to perform I/O other than HTTP requests, such as accessing a disk-based cache, or you need to use concurrency primitives, such as locks, then you should override `.sync_auth_flow()` and `.async_auth_flow()` (instead of `.auth_flow()`). The former will be used by `httpx.Client`, while the latter will be used by `httpx.AsyncClient`. - -```python -import asyncio -import threading -import httpx - - -class MyCustomAuth(httpx.Auth): - def __init__(self): - self._sync_lock = threading.RLock() - self._async_lock = asyncio.Lock() - - def sync_get_token(self): - with self._sync_lock: - ... - - def sync_auth_flow(self, request): - token = self.sync_get_token() - request.headers["Authorization"] = f"Token {token}" - yield request - - async def async_get_token(self): - async with self._async_lock: - ... - - async def async_auth_flow(self, request): - token = await self.async_get_token() - request.headers["Authorization"] = f"Token {token}" - yield request -``` - -If you only want to support one of the two methods, then you should still override it, but raise an explicit `RuntimeError`. - -```python -import httpx -import sync_only_library - - -class MyCustomAuth(httpx.Auth): - def sync_auth_flow(self, request): - token = sync_only_library.get_token(...) - request.headers["Authorization"] = f"Token {token}" - yield request - - async def async_auth_flow(self, request): - raise RuntimeError("Cannot use a sync authentication class with httpx.AsyncClient") -``` - ## SSL certificates When making a request over HTTPS, HTTPX needs to verify the identity of the requested host. To do this, it uses a bundle of SSL certificates (a.k.a. CA bundle) delivered by a trusted certificate authority (CA). diff --git a/mkdocs.yml b/mkdocs.yml index 73b9511cca..9055c9e8da 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -26,6 +26,7 @@ nav: - QuickStart: 'quickstart.md' - Clients: - Overview: 'clients/overview.md' + - Authentication: 'clients/authentication.md' - Guides: - Async Support: 'async.md' - HTTP/2 Support: 'http2.md' From 2eed616c92ae9cadadb387396c167ab960e59f43 Mon Sep 17 00:00:00 2001 From: Tom Christie Date: Tue, 9 Jan 2024 12:43:02 +0000 Subject: [PATCH 04/14] Client authentication docs --- docs/clients/authentication.md | 31 +++++++++++++++++++------------ 1 file changed, 19 insertions(+), 12 deletions(-) diff --git a/docs/clients/authentication.md b/docs/clients/authentication.md index f5be1b1af4..ad2a6c6e69 100644 --- a/docs/clients/authentication.md +++ b/docs/clients/authentication.md @@ -1,43 +1,50 @@ +Authentication can either be included on a per-request basis... + ```pycon ->>> auth = ... +>>> auth = httpx.BasicAuthentication(username="username", password="secret") +>>> client = httpx.Client() >>> response = httpx.get("https://www.example.com/", auth=auth) ``` +Or configured on the client instance, ensuring that all outgoing requests will include authentication credentials... + ```pycon ->>> auth = ... +>>> auth = httpx.BasicAuthentication(username="username", password="secret") >>> client = httpx.Client(auth=auth) ->>> response = client.get("https://www.example.com/") +>>> response = httpx.get("https://www.example.com/") ``` ## Basic authentication -HTTP basic authentication is an unencrypted authentication scheme that uses a simple encoding of the username and password in the request `Authorization` header. Since it is unencrypted it should only be used over `https`, although this is not strictly enforced. +HTTP basic authentication is an unencrypted authentication scheme that uses a simple encoding of the username and password in the request `Authorization` header. Since it is unencrypted it should typically only be used over `https`, although this is not strictly enforced. ```pycon ->>> auth = httpx.BasicAuthentication(username="user@example.com", password="password123") ->>> response = httpx.get("https://www.example.com/", auth=auth) +>>> auth = httpx.BasicAuthentication(username="finley", password="secret") +>>> client = httpx.Client(auth=auth) +>>> response = client.get("https://httpbin.org/basic-auth/finley/secret") >>> response ``` ## Digest authentication -HTTP digest authentication is a challenge-response authentication scheme. Unlike basic authentication it provides encryption **TODO** +HTTP digest authentication is a challenge-response authentication scheme. Unlike basic authentication it provides encryption, and can be used over unencrypted `http` connections. It requires an additional round-trip in order to negotiate the authentication. ```pycon ->>> auth = httpx.DigestAuth(username="user@example.com", password="password123") ->>> response = httpx.get("https://example.com/", auth=auth) +>>> auth = httpx.DigestAuth(username="olivia", password="secret") +>>> client = httpx.Client(auth=auth) +>>> response = client.get("https://httpbin.org/digest-auth/auth/olivia/secret") >>> response >>> response.history -... # TODO +[] ``` ## NetRC authentication HTTPX can be configured to use [a `.netrc` config file](https://everything.curl.dev/usingcurl/netrc) for authentication. -The `.netrc` config file allows authentication credentials to be associated with specified hosts. When a request is made to a host that is found in the netrc file, the username and password will be included using HTTP basic auth. +The `.netrc` config file allows authentication credentials to be associated with specified hosts. When a request is made to a host that is found in the netrc file, the username and password will be included using HTTP basic authentication. Example `.netrc` file: @@ -75,7 +82,7 @@ or fallback to the default. >>> client = httpx.Client(auth=auth) ``` -The `NetRCAuth()` class uses [the `netrc.netrc()` function from the Python standard library](https://docs.python.org/3/library/netrc.html). See the documentation there for more details on exceptions that may be raised if the netrc file is not found, or cannot be parsed. +The `NetRCAuth()` class uses [the `netrc.netrc()` function from the Python standard library](https://docs.python.org/3/library/netrc.html). See the documentation there for more details on exceptions that may be raised if the `.netrc` file is not found, or cannot be parsed. ## Custom authentication schemes From 04329d243164313aca0d0657b3cd4a6583ae6d7b Mon Sep 17 00:00:00 2001 From: Tom Christie Date: Tue, 9 Jan 2024 13:12:17 +0000 Subject: [PATCH 05/14] Fix authentication example --- docs/clients/authentication.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/clients/authentication.md b/docs/clients/authentication.md index ad2a6c6e69..edcc15f815 100644 --- a/docs/clients/authentication.md +++ b/docs/clients/authentication.md @@ -3,7 +3,7 @@ Authentication can either be included on a per-request basis... ```pycon >>> auth = httpx.BasicAuthentication(username="username", password="secret") >>> client = httpx.Client() ->>> response = httpx.get("https://www.example.com/", auth=auth) +>>> response = client.get("https://www.example.com/", auth=auth) ``` Or configured on the client instance, ensuring that all outgoing requests will include authentication credentials... @@ -11,7 +11,7 @@ Or configured on the client instance, ensuring that all outgoing requests will i ```pycon >>> auth = httpx.BasicAuthentication(username="username", password="secret") >>> client = httpx.Client(auth=auth) ->>> response = httpx.get("https://www.example.com/") +>>> response = client.get("https://www.example.com/") ``` ## Basic authentication From 2b32fa733e9ce336b9a47c2877d3dbc2a9624936 Mon Sep 17 00:00:00 2001 From: Tom Christie Date: Tue, 9 Jan 2024 13:32:12 +0000 Subject: [PATCH 06/14] SSL Context --- docs/clients/overview.md | 105 --------------------------------------- docs/clients/ssl.md | 100 +++++++++++++++++++++++++++++++++++++ mkdocs.yml | 1 + 3 files changed, 101 insertions(+), 105 deletions(-) create mode 100644 docs/clients/ssl.md diff --git a/docs/clients/overview.md b/docs/clients/overview.md index b3f756ddfb..325a0cb1f6 100644 --- a/docs/clients/overview.md +++ b/docs/clients/overview.md @@ -694,111 +694,6 @@ For instance this request sends 2 files, `foo.png` and `bar.png` in one request >>> r = httpx.post("https://httpbin.org/post", files=files) ``` -## SSL certificates - -When making a request over HTTPS, HTTPX needs to verify the identity of the requested host. To do this, it uses a bundle of SSL certificates (a.k.a. CA bundle) delivered by a trusted certificate authority (CA). - -### Changing the verification defaults - -By default, HTTPX uses the CA bundle provided by [Certifi](https://pypi.org/project/certifi/). This is what you want in most cases, even though some advanced situations may require you to use a different set of certificates. - -If you'd like to use a custom CA bundle, you can use the `verify` parameter. - -```python -import httpx - -r = httpx.get("https://example.org", verify="path/to/client.pem") -``` - -Alternatively, you can pass a standard library `ssl.SSLContext`. - -```pycon ->>> import ssl ->>> import httpx ->>> context = ssl.create_default_context() ->>> context.load_verify_locations(cafile="/tmp/client.pem") ->>> httpx.get('https://example.org', verify=context) - -``` - -We also include a helper function for creating properly configured `SSLContext` instances. - -```pycon ->>> context = httpx.create_ssl_context() -``` - -The `create_ssl_context` function accepts the same set of SSL configuration arguments -(`trust_env`, `verify`, `cert` and `http2` arguments) -as `httpx.Client` or `httpx.AsyncClient` - -```pycon ->>> import httpx ->>> context = httpx.create_ssl_context(verify="/tmp/client.pem") ->>> httpx.get('https://example.org', verify=context) - -``` - -Or you can also disable the SSL verification entirely, which is _not_ recommended. - -```python -import httpx - -r = httpx.get("https://example.org", verify=False) -``` - -### SSL configuration on client instances - -If you're using a `Client()` instance, then you should pass any SSL settings when instantiating the client. - -```python -client = httpx.Client(verify=False) -``` - -The `client.get(...)` method and other request methods *do not* support changing the SSL settings on a per-request basis. If you need different SSL settings in different cases you should use more that one client instance, with different settings on each. Each client will then be using an isolated connection pool with a specific fixed SSL configuration on all connections within that pool. - -### Client Side Certificates - -You can also specify a local cert to use as a client-side certificate, either a path to an SSL certificate file, or two-tuple of (certificate file, key file), or a three-tuple of (certificate file, key file, password) - -```python -import httpx - -r = httpx.get("https://example.org", cert="path/to/client.pem") -``` - -Alternatively, - -```pycon ->>> cert = ("path/to/client.pem", "path/to/client.key") ->>> httpx.get("https://example.org", cert=cert) - -``` - -or - -```pycon ->>> cert = ("path/to/client.pem", "path/to/client.key", "password") ->>> httpx.get("https://example.org", cert=cert) - -``` - -### Making HTTPS requests to a local server - -When making requests to local servers, such as a development server running on `localhost`, you will typically be using unencrypted HTTP connections. - -If you do need to make HTTPS connections to a local server, for example to test an HTTPS-only service, you will need to create and use your own certificates. Here's one way to do it: - -1. Use [trustme](https://github.com/python-trio/trustme) to generate a pair of server key/cert files, and a client cert file. -1. Pass the server key/cert files when starting your local server. (This depends on the particular web server you're using. For example, [Uvicorn](https://www.uvicorn.org) provides the `--ssl-keyfile` and `--ssl-certfile` options.) -1. Tell HTTPX to use the certificates stored in `client.pem`: - -```pycon ->>> import httpx ->>> r = httpx.get("https://localhost:8000", verify="/tmp/client.pem") ->>> r -Response <200 OK> -``` - ## Custom Transports HTTPX's `Client` also accepts a `transport` argument. This argument allows you diff --git a/docs/clients/ssl.md b/docs/clients/ssl.md new file mode 100644 index 0000000000..d96bbe1979 --- /dev/null +++ b/docs/clients/ssl.md @@ -0,0 +1,100 @@ +When making a request over HTTPS, HTTPX needs to verify the identity of the requested host. To do this, it uses a bundle of SSL certificates (a.k.a. CA bundle) delivered by a trusted certificate authority (CA). + +## Changing the verification defaults + +By default, HTTPX uses the CA bundle provided by [Certifi](https://pypi.org/project/certifi/). This is what you want in most cases, even though some advanced situations may require you to use a different set of certificates. + +If you'd like to use a custom CA bundle, you can use the `verify` parameter. + +```python +import httpx + +r = httpx.get("https://example.org", verify="path/to/client.pem") +``` + +Alternatively, you can pass a standard library `ssl.SSLContext`. + +```pycon +>>> import ssl +>>> import httpx +>>> context = ssl.create_default_context() +>>> context.load_verify_locations(cafile="/tmp/client.pem") +>>> httpx.get('https://example.org', verify=context) + +``` + +We also include a helper function for creating properly configured `SSLContext` instances. + +```pycon +>>> context = httpx.create_ssl_context() +``` + +The `create_ssl_context` function accepts the same set of SSL configuration arguments +(`trust_env`, `verify`, `cert` and `http2` arguments) +as `httpx.Client` or `httpx.AsyncClient` + +```pycon +>>> import httpx +>>> context = httpx.create_ssl_context(verify="/tmp/client.pem") +>>> httpx.get('https://example.org', verify=context) + +``` + +Or you can also disable the SSL verification entirely, which is _not_ recommended. + +```python +import httpx + +r = httpx.get("https://example.org", verify=False) +``` + +## SSL configuration on client instances + +If you're using a `Client()` instance, then you should pass any SSL settings when instantiating the client. + +```python +client = httpx.Client(verify=False) +``` + +The `client.get(...)` method and other request methods *do not* support changing the SSL settings on a per-request basis. If you need different SSL settings in different cases you should use more that one client instance, with different settings on each. Each client will then be using an isolated connection pool with a specific fixed SSL configuration on all connections within that pool. + +## Client Side Certificates + +You can also specify a local cert to use as a client-side certificate, either a path to an SSL certificate file, or two-tuple of (certificate file, key file), or a three-tuple of (certificate file, key file, password) + +```python +cert = "path/to/client.pem" +client = httpx.Client(cert=cert) +response = client.get("https://example.org") +``` + +Alternatively... + +```python +cert = ("path/to/client.pem", "path/to/client.key") +client = httpx.Client(cert=cert) +response = client.get("https://example.org") +``` + +Or... + +```python +cert = ("path/to/client.pem", "path/to/client.key", "password") +client = httpx.Client(cert=cert) +response = client.get("https://example.org") +``` + +## Making HTTPS requests to a local server + +When making requests to local servers, such as a development server running on `localhost`, you will typically be using unencrypted HTTP connections. + +If you do need to make HTTPS connections to a local server, for example to test an HTTPS-only service, you will need to create and use your own certificates. Here's one way to do it: + +1. Use [trustme](https://github.com/python-trio/trustme) to generate a pair of server key/cert files, and a client cert file. +1. Pass the server key/cert files when starting your local server. (This depends on the particular web server you're using. For example, [Uvicorn](https://www.uvicorn.org) provides the `--ssl-keyfile` and `--ssl-certfile` options.) +1. Tell HTTPX to use the certificates stored in `client.pem`: + +```python +client = httpx.Client(verify="/tmp/client.pem") +response = client.get("https://localhost:8000") +``` diff --git a/mkdocs.yml b/mkdocs.yml index 9055c9e8da..5c37c1f033 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -27,6 +27,7 @@ nav: - Clients: - Overview: 'clients/overview.md' - Authentication: 'clients/authentication.md' + - SSL Context: 'clients/ssl.md' - Guides: - Async Support: 'async.md' - HTTP/2 Support: 'http2.md' From 3c02725e12e2d1727b8baa805564ef9cc30b9fb6 Mon Sep 17 00:00:00 2001 From: Tom Christie Date: Tue, 9 Jan 2024 13:39:23 +0000 Subject: [PATCH 07/14] Timeouts --- docs/clients/overview.md | 74 ---------------------------------------- docs/clients/timeouts.md | 71 ++++++++++++++++++++++++++++++++++++++ mkdocs.yml | 3 +- 3 files changed, 73 insertions(+), 75 deletions(-) create mode 100644 docs/clients/timeouts.md diff --git a/docs/clients/overview.md b/docs/clients/overview.md index 325a0cb1f6..58ddc9509f 100644 --- a/docs/clients/overview.md +++ b/docs/clients/overview.md @@ -544,80 +544,6 @@ You can now configure a client to make requests via a proxy using the SOCKS prot httpx.Client(proxy='socks5://user:pass@host:port') ``` -## Timeout Configuration - -HTTPX is careful to enforce timeouts everywhere by default. - -The default behavior is to raise a `TimeoutException` after 5 seconds of -network inactivity. - -### Setting and disabling timeouts - -You can set timeouts for an individual request: - -```python -# Using the top-level API: -httpx.get('http://example.com/api/v1/example', timeout=10.0) - -# Using a client instance: -with httpx.Client() as client: - client.get("http://example.com/api/v1/example", timeout=10.0) -``` - -Or disable timeouts for an individual request: - -```python -# Using the top-level API: -httpx.get('http://example.com/api/v1/example', timeout=None) - -# Using a client instance: -with httpx.Client() as client: - client.get("http://example.com/api/v1/example", timeout=None) -``` - -### Setting a default timeout on a client - -You can set a timeout on a client instance, which results in the given -`timeout` being used as the default for requests made with this client: - -```python -client = httpx.Client() # Use a default 5s timeout everywhere. -client = httpx.Client(timeout=10.0) # Use a default 10s timeout everywhere. -client = httpx.Client(timeout=None) # Disable all timeouts by default. -``` - -### Fine tuning the configuration - -HTTPX also allows you to specify the timeout behavior in more fine grained detail. - -There are four different types of timeouts that may occur. These are **connect**, -**read**, **write**, and **pool** timeouts. - -* The **connect** timeout specifies the maximum amount of time to wait until -a socket connection to the requested host is established. If HTTPX is unable to connect -within this time frame, a `ConnectTimeout` exception is raised. -* The **read** timeout specifies the maximum duration to wait for a chunk of -data to be received (for example, a chunk of the response body). If HTTPX is -unable to receive data within this time frame, a `ReadTimeout` exception is raised. -* The **write** timeout specifies the maximum duration to wait for a chunk of -data to be sent (for example, a chunk of the request body). If HTTPX is unable -to send data within this time frame, a `WriteTimeout` exception is raised. -* The **pool** timeout specifies the maximum duration to wait for acquiring -a connection from the connection pool. If HTTPX is unable to acquire a connection -within this time frame, a `PoolTimeout` exception is raised. A related -configuration here is the maximum number of allowable connections in the -connection pool, which is configured by the `limits` argument. - -You can configure the timeout behavior for any of these values... - -```python -# A client with a 60s timeout for connecting, and a 10s timeout elsewhere. -timeout = httpx.Timeout(10.0, connect=60.0) -client = httpx.Client(timeout=timeout) - -response = client.get('http://example.com/') -``` - ## Pool limit configuration You can control the connection pool size using the `limits` keyword diff --git a/docs/clients/timeouts.md b/docs/clients/timeouts.md new file mode 100644 index 0000000000..aedcfb627f --- /dev/null +++ b/docs/clients/timeouts.md @@ -0,0 +1,71 @@ +HTTPX is careful to enforce timeouts everywhere by default. + +The default behavior is to raise a `TimeoutException` after 5 seconds of +network inactivity. + +## Setting and disabling timeouts + +You can set timeouts for an individual request: + +```python +# Using the top-level API: +httpx.get('http://example.com/api/v1/example', timeout=10.0) + +# Using a client instance: +with httpx.Client() as client: + client.get("http://example.com/api/v1/example", timeout=10.0) +``` + +Or disable timeouts for an individual request: + +```python +# Using the top-level API: +httpx.get('http://example.com/api/v1/example', timeout=None) + +# Using a client instance: +with httpx.Client() as client: + client.get("http://example.com/api/v1/example", timeout=None) +``` + +## Setting a default timeout on a client + +You can set a timeout on a client instance, which results in the given +`timeout` being used as the default for requests made with this client: + +```python +client = httpx.Client() # Use a default 5s timeout everywhere. +client = httpx.Client(timeout=10.0) # Use a default 10s timeout everywhere. +client = httpx.Client(timeout=None) # Disable all timeouts by default. +``` + +## Fine tuning the configuration + +HTTPX also allows you to specify the timeout behavior in more fine grained detail. + +There are four different types of timeouts that may occur. These are **connect**, +**read**, **write**, and **pool** timeouts. + +* The **connect** timeout specifies the maximum amount of time to wait until +a socket connection to the requested host is established. If HTTPX is unable to connect +within this time frame, a `ConnectTimeout` exception is raised. +* The **read** timeout specifies the maximum duration to wait for a chunk of +data to be received (for example, a chunk of the response body). If HTTPX is +unable to receive data within this time frame, a `ReadTimeout` exception is raised. +* The **write** timeout specifies the maximum duration to wait for a chunk of +data to be sent (for example, a chunk of the request body). If HTTPX is unable +to send data within this time frame, a `WriteTimeout` exception is raised. +* The **pool** timeout specifies the maximum duration to wait for acquiring +a connection from the connection pool. If HTTPX is unable to acquire a connection +within this time frame, a `PoolTimeout` exception is raised. A related +configuration here is the maximum number of allowable connections in the +connection pool, which is configured by the `limits` argument. + +You can configure the timeout behavior for any of these values... + +```python +# A client with a 60s timeout for connecting, and a 10s timeout elsewhere. +timeout = httpx.Timeout(10.0, connect=60.0) +client = httpx.Client(timeout=timeout) + +response = client.get('http://example.com/') +``` \ No newline at end of file diff --git a/mkdocs.yml b/mkdocs.yml index 5c37c1f033..fdf8ab89a1 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -27,7 +27,8 @@ nav: - Clients: - Overview: 'clients/overview.md' - Authentication: 'clients/authentication.md' - - SSL Context: 'clients/ssl.md' + - SSL: 'clients/ssl.md' + - Timeouts: 'clients/timeouts.md' - Guides: - Async Support: 'async.md' - HTTP/2 Support: 'http2.md' From c82f095f656b522b46ed1961571d1584c0424424 Mon Sep 17 00:00:00 2001 From: Tom Christie Date: Tue, 9 Jan 2024 13:49:32 +0000 Subject: [PATCH 08/14] Event hooks --- docs/clients/event-hooks.md | 65 +++++++++++++++++++++++++++++++++++ docs/clients/overview.md | 68 ------------------------------------- mkdocs.yml | 1 + 3 files changed, 66 insertions(+), 68 deletions(-) create mode 100644 docs/clients/event-hooks.md diff --git a/docs/clients/event-hooks.md b/docs/clients/event-hooks.md new file mode 100644 index 0000000000..28cf353d96 --- /dev/null +++ b/docs/clients/event-hooks.md @@ -0,0 +1,65 @@ +HTTPX allows you to register "event hooks" with the client, that are called +every time a particular type of event takes place. + +There are currently two event hooks: + +* `request` - Called after a request is fully prepared, but before it is sent to the network. Passed the `request` instance. +* `response` - Called after the response has been fetched from the network, but before it is returned to the caller. Passed the `response` instance. + +These allow you to install client-wide functionality such as logging, monitoring or tracing. + +```python +def log_request(request): + print(f"Request event hook: {request.method} {request.url} - Waiting for response") + +def log_response(response): + request = response.request + print(f"Response event hook: {request.method} {request.url} - Status {response.status_code}") + +client = httpx.Client(event_hooks={'request': [log_request], 'response': [log_response]}) +``` + +You can also use these hooks to install response processing code, such as this +example, which creates a client instance that always raises `httpx.HTTPStatusError` +on 4xx and 5xx responses. + +```python +def raise_on_4xx_5xx(response): + response.raise_for_status() + +client = httpx.Client(event_hooks={'response': [raise_on_4xx_5xx]}) +``` + +!!! note + Response event hooks are called before determining if the response body + should be read or not. + + If you need access to the response body inside an event hook, you'll + need to call `response.read()`, or for AsyncClients, `response.aread()`. + +The hooks are also allowed to modify `request` and `response` objects. + +```python +def add_timestamp(request): + request.headers['x-request-timestamp'] = datetime.now(tz=datetime.utc).isoformat() + +client = httpx.Client(event_hooks={'request': [add_timestamp]}) +``` + +Event hooks must always be set as a **list of callables**, and you may register +multiple event hooks for each type of event. + +As well as being able to set event hooks on instantiating the client, there +is also an `.event_hooks` property, that allows you to inspect and modify +the installed hooks. + +```python +client = httpx.Client() +client.event_hooks['request'] = [log_request] +client.event_hooks['response'] = [log_response, raise_on_4xx_5xx] +``` + +!!! note + If you are using HTTPX's async support, then you need to be aware that + hooks registered with `httpx.AsyncClient` MUST be async functions, + rather than plain functions. diff --git a/docs/clients/overview.md b/docs/clients/overview.md index 58ddc9509f..b6b7bc4615 100644 --- a/docs/clients/overview.md +++ b/docs/clients/overview.md @@ -303,74 +303,6 @@ with httpx.Client(headers=headers) as client: ... ``` -## Event Hooks - -HTTPX allows you to register "event hooks" with the client, that are called -every time a particular type of event takes place. - -There are currently two event hooks: - -* `request` - Called after a request is fully prepared, but before it is sent to the network. Passed the `request` instance. -* `response` - Called after the response has been fetched from the network, but before it is returned to the caller. Passed the `response` instance. - -These allow you to install client-wide functionality such as logging, monitoring or tracing. - -```python -def log_request(request): - print(f"Request event hook: {request.method} {request.url} - Waiting for response") - -def log_response(response): - request = response.request - print(f"Response event hook: {request.method} {request.url} - Status {response.status_code}") - -client = httpx.Client(event_hooks={'request': [log_request], 'response': [log_response]}) -``` - -You can also use these hooks to install response processing code, such as this -example, which creates a client instance that always raises `httpx.HTTPStatusError` -on 4xx and 5xx responses. - -```python -def raise_on_4xx_5xx(response): - response.raise_for_status() - -client = httpx.Client(event_hooks={'response': [raise_on_4xx_5xx]}) -``` - -!!! note - Response event hooks are called before determining if the response body - should be read or not. - - If you need access to the response body inside an event hook, you'll - need to call `response.read()`, or for AsyncClients, `response.aread()`. - -The hooks are also allowed to modify `request` and `response` objects. - -```python -def add_timestamp(request): - request.headers['x-request-timestamp'] = datetime.now(tz=datetime.utc).isoformat() - -client = httpx.Client(event_hooks={'request': [add_timestamp]}) -``` - -Event hooks must always be set as a **list of callables**, and you may register -multiple event hooks for each type of event. - -As well as being able to set event hooks on instantiating the client, there -is also an `.event_hooks` property, that allows you to inspect and modify -the installed hooks. - -```python -client = httpx.Client() -client.event_hooks['request'] = [log_request] -client.event_hooks['response'] = [log_response, raise_on_4xx_5xx] -``` - -!!! note - If you are using HTTPX's async support, then you need to be aware that - hooks registered with `httpx.AsyncClient` MUST be async functions, - rather than plain functions. - ## Monitoring download progress If you need to monitor download progress of large responses, you can use response streaming and inspect the `response.num_bytes_downloaded` property. diff --git a/mkdocs.yml b/mkdocs.yml index fdf8ab89a1..8672dd4364 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -29,6 +29,7 @@ nav: - Authentication: 'clients/authentication.md' - SSL: 'clients/ssl.md' - Timeouts: 'clients/timeouts.md' + - Event Hooks: 'clients/event-hooks.md' - Guides: - Async Support: 'async.md' - HTTP/2 Support: 'http2.md' From c25315ccf09e42fa5f51fd7cac81bbc9f6cb0886 Mon Sep 17 00:00:00 2001 From: Tom Christie Date: Tue, 9 Jan 2024 14:04:36 +0000 Subject: [PATCH 09/14] Proxies, Transports --- docs/clients/overview.md | 434 ------------------------------------- docs/clients/proxies.md | 83 +++++++ docs/clients/transports.md | 344 +++++++++++++++++++++++++++++ mkdocs.yml | 2 + 4 files changed, 429 insertions(+), 434 deletions(-) create mode 100644 docs/clients/proxies.md create mode 100644 docs/clients/transports.md diff --git a/docs/clients/overview.md b/docs/clients/overview.md index b6b7bc4615..cbaaccc51c 100644 --- a/docs/clients/overview.md +++ b/docs/clients/overview.md @@ -227,49 +227,6 @@ print(response.text) --- -## Calling into Python Web Apps - -You can configure an `httpx` client to call directly into a Python web application using the WSGI protocol. - -This is particularly useful for two main use-cases: - -* Using `httpx` as a client inside test cases. -* Mocking out external services during tests or in dev/staging environments. - -Here's an example of integrating against a Flask application: - -```python -from flask import Flask -import httpx - - -app = Flask(__name__) - -@app.route("/") -def hello(): - return "Hello World!" - -with httpx.Client(app=app, base_url="http://testserver") as client: - r = client.get("/") - assert r.status_code == 200 - assert r.text == "Hello World!" -``` - -For some more complex cases you might need to customize the WSGI transport. This allows you to: - -* Inspect 500 error responses rather than raise exceptions by setting `raise_app_exceptions=False`. -* Mount the WSGI application at a subpath by setting `script_name` (WSGI). -* Use a given client address for requests by setting `remote_addr` (WSGI). - -For example: - -```python -# Instantiate a client that makes WSGI requests with a client IP of "1.2.3.4". -transport = httpx.WSGITransport(app=app, remote_addr="1.2.3.4") -with httpx.Client(transport=transport, base_url="http://testserver") as client: - ... -``` - ## Request instances For maximum control on what gets sent over the wire, HTTPX supports building explicit [`Request`](api.md#request) instances: @@ -390,92 +347,6 @@ httpx.post("https://httpbin.org/post", content=gen()) ![tqdm progress bar](img/tqdm-progress.gif) -## HTTP Proxying - -HTTPX supports setting up [HTTP proxies](https://en.wikipedia.org/wiki/Proxy_server#Web_proxy_servers) via the `proxy` parameter to be passed on client initialization or top-level API functions like `httpx.get(..., proxy=...)`. - -
- -
Diagram of how a proxy works (source: Wikipedia). The left hand side "Internet" blob may be your HTTPX client requesting example.com through a proxy.
-
- -### Example - -To route all traffic (HTTP and HTTPS) to a proxy located at `http://localhost:8030`, pass the proxy URL to the client... - -```python -with httpx.Client(proxy="http://localhost:8030") as client: - ... -``` - -For more advanced use cases, pass a mounts `dict`. For example, to route HTTP and HTTPS requests to 2 different proxies, respectively located at `http://localhost:8030`, and `http://localhost:8031`, pass a `dict` of proxy URLs: - -```python -proxy_mounts = { - "http://": httpx.HTTPTransport(proxy="http://localhost:8030"), - "https://": httpx.HTTPTransport(proxy="http://localhost:8031"), -} - -with httpx.Client(mounts=proxy_mounts) as client: - ... -``` - -For detailed information about proxy routing, see the [Routing](#routing) section. - -!!! tip "Gotcha" - In most cases, the proxy URL for the `https://` key _should_ use the `http://` scheme (that's not a typo!). - - This is because HTTP proxying requires initiating a connection with the proxy server. While it's possible that your proxy supports doing it via HTTPS, most proxies only support doing it via HTTP. - - For more information, see [FORWARD vs TUNNEL](#forward-vs-tunnel). - -### Authentication - -Proxy credentials can be passed as the `userinfo` section of the proxy URL. For example: - -```python -with httpx.Client(proxy="http://username:password@localhost:8030") as client: - ... -``` - -### Proxy mechanisms - -!!! note - This section describes **advanced** proxy concepts and functionality. - -#### FORWARD vs TUNNEL - -In general, the flow for making an HTTP request through a proxy is as follows: - -1. The client connects to the proxy (initial connection request). -2. The proxy transfers data to the server on your behalf. - -How exactly step 2/ is performed depends on which of two proxying mechanisms is used: - -* **Forwarding**: the proxy makes the request for you, and sends back the response it obtained from the server. -* **Tunnelling**: the proxy establishes a TCP connection to the server on your behalf, and the client reuses this connection to send the request and receive the response. This is known as an [HTTP Tunnel](https://en.wikipedia.org/wiki/HTTP_tunnel). This mechanism is how you can access websites that use HTTPS from an HTTP proxy (the client "upgrades" the connection to HTTPS by performing the TLS handshake with the server over the TCP connection provided by the proxy). - -### Troubleshooting proxies - -If you encounter issues when setting up proxies, please refer to our [Troubleshooting guide](troubleshooting.md#proxies). - -## SOCKS - -In addition to HTTP proxies, `httpcore` also supports proxies using the SOCKS protocol. -This is an optional feature that requires an additional third-party library be installed before use. - -You can install SOCKS support using `pip`: - -```shell -$ pip install httpx[socks] -``` - -You can now configure a client to make requests via a proxy using the SOCKS protocol: - -```python -httpx.Client(proxy='socks5://user:pass@host:port') -``` - ## Pool limit configuration You can control the connection pool size using the `limits` keyword @@ -551,308 +422,3 @@ For instance this request sends 2 files, `foo.png` and `bar.png` in one request ('images', ('bar.png', open('bar.png', 'rb'), 'image/png'))] >>> r = httpx.post("https://httpbin.org/post", files=files) ``` - -## Custom Transports - -HTTPX's `Client` also accepts a `transport` argument. This argument allows you -to provide a custom Transport object that will be used to perform the actual -sending of the requests. - -### Usage - -For some advanced configuration you might need to instantiate a transport -class directly, and pass it to the client instance. One example is the -`local_address` configuration which is only available via this low-level API. - -```pycon ->>> import httpx ->>> transport = httpx.HTTPTransport(local_address="0.0.0.0") ->>> client = httpx.Client(transport=transport) -``` - -Connection retries are also available via this interface. Requests will be retried the given number of times in case an `httpx.ConnectError` or an `httpx.ConnectTimeout` occurs, allowing smoother operation under flaky networks. If you need other forms of retry behaviors, such as handling read/write errors or reacting to `503 Service Unavailable`, consider general-purpose tools such as [tenacity](https://github.com/jd/tenacity). - -```pycon ->>> import httpx ->>> transport = httpx.HTTPTransport(retries=1) ->>> client = httpx.Client(transport=transport) -``` - -Similarly, instantiating a transport directly provides a `uds` option for -connecting via a Unix Domain Socket that is only available via this low-level API: - -```pycon ->>> import httpx ->>> # Connect to the Docker API via a Unix Socket. ->>> transport = httpx.HTTPTransport(uds="/var/run/docker.sock") ->>> client = httpx.Client(transport=transport) ->>> response = client.get("http://docker/info") ->>> response.json() -{"ID": "...", "Containers": 4, "Images": 74, ...} -``` - -### urllib3 transport - -This [public gist](https://gist.github.com/florimondmanca/d56764d78d748eb9f73165da388e546e) provides a transport that uses the excellent [`urllib3` library](https://urllib3.readthedocs.io/en/latest/), and can be used with the sync `Client`... - -```pycon ->>> import httpx ->>> from urllib3_transport import URLLib3Transport ->>> client = httpx.Client(transport=URLLib3Transport()) ->>> client.get("https://example.org") - -``` - -### Writing custom transports - -A transport instance must implement the low-level Transport API, which deals -with sending a single request, and returning a response. You should either -subclass `httpx.BaseTransport` to implement a transport to use with `Client`, -or subclass `httpx.AsyncBaseTransport` to implement a transport to -use with `AsyncClient`. - -At the layer of the transport API we're using the familiar `Request` and -`Response` models. - -See the `handle_request` and `handle_async_request` docstrings for more details -on the specifics of the Transport API. - -A complete example of a custom transport implementation would be: - -```python -import json -import httpx - - -class HelloWorldTransport(httpx.BaseTransport): - """ - A mock transport that always returns a JSON "Hello, world!" response. - """ - - def handle_request(self, request): - message = {"text": "Hello, world!"} - content = json.dumps(message).encode("utf-8") - stream = httpx.ByteStream(content) - headers = [(b"content-type", b"application/json")] - return httpx.Response(200, headers=headers, stream=stream) -``` - -Which we can use in the same way: - -```pycon ->>> import httpx ->>> client = httpx.Client(transport=HelloWorldTransport()) ->>> response = client.get("https://example.org/") ->>> response.json() -{"text": "Hello, world!"} -``` - -### Mock transports - -During testing it can often be useful to be able to mock out a transport, -and return pre-determined responses, rather than making actual network requests. - -The `httpx.MockTransport` class accepts a handler function, which can be used -to map requests onto pre-determined responses: - -```python -def handler(request): - return httpx.Response(200, json={"text": "Hello, world!"}) - - -# Switch to a mock transport, if the TESTING environment variable is set. -if os.environ.get('TESTING', '').upper() == "TRUE": - transport = httpx.MockTransport(handler) -else: - transport = httpx.HTTPTransport() - -client = httpx.Client(transport=transport) -``` - -For more advanced use-cases you might want to take a look at either [the third-party -mocking library, RESPX](https://lundberg.github.io/respx/), or the [pytest-httpx library](https://github.com/Colin-b/pytest_httpx). - -### Mounting transports - -You can also mount transports against given schemes or domains, to control -which transport an outgoing request should be routed via, with [the same style -used for specifying proxy routing](#routing). - -```python -import httpx - -class HTTPSRedirectTransport(httpx.BaseTransport): - """ - A transport that always redirects to HTTPS. - """ - - def handle_request(self, method, url, headers, stream, extensions): - scheme, host, port, path = url - if port is None: - location = b"https://%s%s" % (host, path) - else: - location = b"https://%s:%d%s" % (host, port, path) - stream = httpx.ByteStream(b"") - headers = [(b"location", location)] - extensions = {} - return 303, headers, stream, extensions - - -# A client where any `http` requests are always redirected to `https` -mounts = {'http://': HTTPSRedirectTransport()} -client = httpx.Client(mounts=mounts) -``` - -A couple of other sketches of how you might take advantage of mounted transports... - -Disabling HTTP/2 on a single given domain... - -```python -mounts = { - "all://": httpx.HTTPTransport(http2=True), - "all://*example.org": httpx.HTTPTransport() -} -client = httpx.Client(mounts=mounts) -``` - -Mocking requests to a given domain: - -```python -# All requests to "example.org" should be mocked out. -# Other requests occur as usual. -def handler(request): - return httpx.Response(200, json={"text": "Hello, World!"}) - -mounts = {"all://example.org": httpx.MockTransport(handler)} -client = httpx.Client(mounts=mounts) -``` - -Adding support for custom schemes: - -```python -# Support URLs like "file:///Users/sylvia_green/websites/new_client/index.html" -mounts = {"file://": FileSystemTransport()} -client = httpx.Client(mounts=mounts) -``` - -### Routing - -HTTPX provides a powerful mechanism for routing requests, allowing you to write complex rules that specify which transport should be used for each request. - -The `mounts` dictionary maps URL patterns to HTTP transports. HTTPX matches requested URLs against URL patterns to decide which transport should be used, if any. Matching is done from most specific URL patterns (e.g. `https://:`) to least specific ones (e.g. `https://`). - -HTTPX supports routing requests based on **scheme**, **domain**, **port**, or a combination of these. - -#### Wildcard routing - -Route everything through a transport... - -```python -mounts = { - "all://": httpx.HTTPTransport(proxy="http://localhost:8030"), -} -``` - -#### Scheme routing - -Route HTTP requests through one transport, and HTTPS requests through another... - -```python -mounts = { - "http://": httpx.HTTPTransport(proxy="http://localhost:8030"), - "https://": httpx.HTTPTransport(proxy="http://localhost:8031"), -} -``` - -#### Domain routing - -Proxy all requests on domain "example.com", let other requests pass through... - -```python -mounts = { - "all://example.com": httpx.HTTPTransport(proxy="http://localhost:8030"), -} -``` - -Proxy HTTP requests on domain "example.com", let HTTPS and other requests pass through... - -```python -mounts = { - "http://example.com": httpx.HTTPTransport(proxy="http://localhost:8030"), -} -``` - -Proxy all requests to "example.com" and its subdomains, let other requests pass through... - -```python -mounts = { - "all://*example.com": httpx.HTTPTransport(proxy="http://localhost:8030"), -} -``` - -Proxy all requests to strict subdomains of "example.com", let "example.com" and other requests pass through... - -```python -mounts = { - "all://*.example.com": httpx.HTTPTransport(proxy="http://localhost:8030"), -} -``` - -#### Port routing - -Proxy HTTPS requests on port 1234 to "example.com"... - -```python -mounts = { - "https://example.com:1234": httpx.HTTPTransport(proxy="http://localhost:8030"), -} -``` - -Proxy all requests on port 1234... - -```python -mounts = { - "all://*:1234": httpx.HTTPTransport(proxy="http://localhost:8030"), -} -``` - -#### No-proxy support - -It is also possible to define requests that _shouldn't_ be routed through the transport. - -To do so, pass `None` as the proxy URL. For example... - -```python -mounts = { - # Route requests through a proxy by default... - "all://": httpx.HTTPTransport(proxy="http://localhost:8031"), - # Except those for "example.com". - "all://example.com": None, -} -``` - -#### Complex configuration example - -You can combine the routing features outlined above to build complex proxy routing configurations. For example... - -```python -mounts = { - # Route all traffic through a proxy by default... - "all://": httpx.HTTPTransport(proxy="http://localhost:8030"), - # But don't use proxies for HTTPS requests to "domain.io"... - "https://domain.io": None, - # And use another proxy for requests to "example.com" and its subdomains... - "all://*example.com": httpx.HTTPTransport(proxy="http://localhost:8031"), - # And yet another proxy if HTTP is used, - # and the "internal" subdomain on port 5550 is requested... - "http://internal.example.com:5550": httpx.HTTPTransport(proxy="http://localhost:8032"), -} -``` - -#### Environment variables - -There are also environment variables that can be used to control the dictionary of the client mounts. -They can be used to configure HTTP proxying for clients. - -See documentation on [`HTTP_PROXY`, `HTTPS_PROXY`, `ALL_PROXY`](environment_variables.md#http_proxy-https_proxy-all_proxy) for more information. - diff --git a/docs/clients/proxies.md b/docs/clients/proxies.md new file mode 100644 index 0000000000..d51b241a2c --- /dev/null +++ b/docs/clients/proxies.md @@ -0,0 +1,83 @@ +HTTPX supports setting up [HTTP proxies](https://en.wikipedia.org/wiki/Proxy_server#Web_proxy_servers) via the `proxy` parameter to be passed on client initialization or top-level API functions like `httpx.get(..., proxy=...)`. + +
+ +
Diagram of how a proxy works (source: Wikipedia). The left hand side "Internet" blob may be your HTTPX client requesting example.com through a proxy.
+
+ +## HTTP Proxies + +To route all traffic (HTTP and HTTPS) to a proxy located at `http://localhost:8030`, pass the proxy URL to the client... + +```python +with httpx.Client(proxy="http://localhost:8030") as client: + ... +``` + +For more advanced use cases, pass a mounts `dict`. For example, to route HTTP and HTTPS requests to 2 different proxies, respectively located at `http://localhost:8030`, and `http://localhost:8031`, pass a `dict` of proxy URLs: + +```python +proxy_mounts = { + "http://": httpx.HTTPTransport(proxy="http://localhost:8030"), + "https://": httpx.HTTPTransport(proxy="http://localhost:8031"), +} + +with httpx.Client(mounts=proxy_mounts) as client: + ... +``` + +For detailed information about proxy routing, see the [Routing](#routing) section. + +!!! tip "Gotcha" + In most cases, the proxy URL for the `https://` key _should_ use the `http://` scheme (that's not a typo!). + + This is because HTTP proxying requires initiating a connection with the proxy server. While it's possible that your proxy supports doing it via HTTPS, most proxies only support doing it via HTTP. + + For more information, see [FORWARD vs TUNNEL](#forward-vs-tunnel). + +## Authentication + +Proxy credentials can be passed as the `userinfo` section of the proxy URL. For example: + +```python +with httpx.Client(proxy="http://username:password@localhost:8030") as client: + ... +``` + +## Proxy mechanisms + +!!! note + This section describes **advanced** proxy concepts and functionality. + +### FORWARD vs TUNNEL + +In general, the flow for making an HTTP request through a proxy is as follows: + +1. The client connects to the proxy (initial connection request). +2. The proxy transfers data to the server on your behalf. + +How exactly step 2/ is performed depends on which of two proxying mechanisms is used: + +* **Forwarding**: the proxy makes the request for you, and sends back the response it obtained from the server. +* **Tunnelling**: the proxy establishes a TCP connection to the server on your behalf, and the client reuses this connection to send the request and receive the response. This is known as an [HTTP Tunnel](https://en.wikipedia.org/wiki/HTTP_tunnel). This mechanism is how you can access websites that use HTTPS from an HTTP proxy (the client "upgrades" the connection to HTTPS by performing the TLS handshake with the server over the TCP connection provided by the proxy). + +### Troubleshooting proxies + +If you encounter issues when setting up proxies, please refer to our [Troubleshooting guide](troubleshooting.md#proxies). + +## SOCKS + +In addition to HTTP proxies, `httpcore` also supports proxies using the SOCKS protocol. +This is an optional feature that requires an additional third-party library be installed before use. + +You can install SOCKS support using `pip`: + +```shell +$ pip install httpx[socks] +``` + +You can now configure a client to make requests via a proxy using the SOCKS protocol: + +```python +httpx.Client(proxy='socks5://user:pass@host:port') +``` diff --git a/docs/clients/transports.md b/docs/clients/transports.md new file mode 100644 index 0000000000..100b562469 --- /dev/null +++ b/docs/clients/transports.md @@ -0,0 +1,344 @@ +HTTPX's `Client` also accepts a `transport` argument. This argument allows you +to provide a custom Transport object that will be used to perform the actual +sending of the requests. + +## HTTPTransport + +For some advanced configuration you might need to instantiate a transport +class directly, and pass it to the client instance. One example is the +`local_address` configuration which is only available via this low-level API. + +```pycon +>>> import httpx +>>> transport = httpx.HTTPTransport(local_address="0.0.0.0") +>>> client = httpx.Client(transport=transport) +``` + +Connection retries are also available via this interface. Requests will be retried the given number of times in case an `httpx.ConnectError` or an `httpx.ConnectTimeout` occurs, allowing smoother operation under flaky networks. If you need other forms of retry behaviors, such as handling read/write errors or reacting to `503 Service Unavailable`, consider general-purpose tools such as [tenacity](https://github.com/jd/tenacity). + +```pycon +>>> import httpx +>>> transport = httpx.HTTPTransport(retries=1) +>>> client = httpx.Client(transport=transport) +``` + +Similarly, instantiating a transport directly provides a `uds` option for +connecting via a Unix Domain Socket that is only available via this low-level API: + +```pycon +>>> import httpx +>>> # Connect to the Docker API via a Unix Socket. +>>> transport = httpx.HTTPTransport(uds="/var/run/docker.sock") +>>> client = httpx.Client(transport=transport) +>>> response = client.get("http://docker/info") +>>> response.json() +{"ID": "...", "Containers": 4, "Images": 74, ...} +``` + +## WSGI Transport + +You can configure an `httpx` client to call directly into a Python web application using the WSGI protocol. + +This is particularly useful for two main use-cases: + +* Using `httpx` as a client inside test cases. +* Mocking out external services during tests or in dev/staging environments. + +Here's an example of integrating against a Flask application: + +```python +from flask import Flask +import httpx + + +app = Flask(__name__) + +@app.route("/") +def hello(): + return "Hello World!" + +with httpx.Client(app=app, base_url="http://testserver") as client: + r = client.get("/") + assert r.status_code == 200 + assert r.text == "Hello World!" +``` + +For some more complex cases you might need to customize the WSGI transport. This allows you to: + +* Inspect 500 error responses rather than raise exceptions by setting `raise_app_exceptions=False`. +* Mount the WSGI application at a subpath by setting `script_name` (WSGI). +* Use a given client address for requests by setting `remote_addr` (WSGI). + +For example: + +```python +# Instantiate a client that makes WSGI requests with a client IP of "1.2.3.4". +transport = httpx.WSGITransport(app=app, remote_addr="1.2.3.4") +with httpx.Client(transport=transport, base_url="http://testserver") as client: + ... +``` + +## urllib3 transport + +This [public gist](https://gist.github.com/florimondmanca/d56764d78d748eb9f73165da388e546e) provides a transport that uses the excellent [`urllib3` library](https://urllib3.readthedocs.io/en/latest/), and can be used with the sync `Client`... + +```pycon +>>> import httpx +>>> from urllib3_transport import URLLib3Transport +>>> client = httpx.Client(transport=URLLib3Transport()) +>>> client.get("https://example.org") + +``` + +## Custom transports + +A transport instance must implement the low-level Transport API, which deals +with sending a single request, and returning a response. You should either +subclass `httpx.BaseTransport` to implement a transport to use with `Client`, +or subclass `httpx.AsyncBaseTransport` to implement a transport to +use with `AsyncClient`. + +At the layer of the transport API we're using the familiar `Request` and +`Response` models. + +See the `handle_request` and `handle_async_request` docstrings for more details +on the specifics of the Transport API. + +A complete example of a custom transport implementation would be: + +```python +import json +import httpx + + +class HelloWorldTransport(httpx.BaseTransport): + """ + A mock transport that always returns a JSON "Hello, world!" response. + """ + + def handle_request(self, request): + message = {"text": "Hello, world!"} + content = json.dumps(message).encode("utf-8") + stream = httpx.ByteStream(content) + headers = [(b"content-type", b"application/json")] + return httpx.Response(200, headers=headers, stream=stream) +``` + +Which we can use in the same way: + +```pycon +>>> import httpx +>>> client = httpx.Client(transport=HelloWorldTransport()) +>>> response = client.get("https://example.org/") +>>> response.json() +{"text": "Hello, world!"} +``` + +## Mock transports + +During testing it can often be useful to be able to mock out a transport, +and return pre-determined responses, rather than making actual network requests. + +The `httpx.MockTransport` class accepts a handler function, which can be used +to map requests onto pre-determined responses: + +```python +def handler(request): + return httpx.Response(200, json={"text": "Hello, world!"}) + + +# Switch to a mock transport, if the TESTING environment variable is set. +if os.environ.get('TESTING', '').upper() == "TRUE": + transport = httpx.MockTransport(handler) +else: + transport = httpx.HTTPTransport() + +client = httpx.Client(transport=transport) +``` + +For more advanced use-cases you might want to take a look at either [the third-party +mocking library, RESPX](https://lundberg.github.io/respx/), or the [pytest-httpx library](https://github.com/Colin-b/pytest_httpx). + +## Mounting transports + +You can also mount transports against given schemes or domains, to control +which transport an outgoing request should be routed via, with [the same style +used for specifying proxy routing](#routing). + +```python +import httpx + +class HTTPSRedirectTransport(httpx.BaseTransport): + """ + A transport that always redirects to HTTPS. + """ + + def handle_request(self, method, url, headers, stream, extensions): + scheme, host, port, path = url + if port is None: + location = b"https://%s%s" % (host, path) + else: + location = b"https://%s:%d%s" % (host, port, path) + stream = httpx.ByteStream(b"") + headers = [(b"location", location)] + extensions = {} + return 303, headers, stream, extensions + + +# A client where any `http` requests are always redirected to `https` +mounts = {'http://': HTTPSRedirectTransport()} +client = httpx.Client(mounts=mounts) +``` + +A couple of other sketches of how you might take advantage of mounted transports... + +Disabling HTTP/2 on a single given domain... + +```python +mounts = { + "all://": httpx.HTTPTransport(http2=True), + "all://*example.org": httpx.HTTPTransport() +} +client = httpx.Client(mounts=mounts) +``` + +Mocking requests to a given domain: + +```python +# All requests to "example.org" should be mocked out. +# Other requests occur as usual. +def handler(request): + return httpx.Response(200, json={"text": "Hello, World!"}) + +mounts = {"all://example.org": httpx.MockTransport(handler)} +client = httpx.Client(mounts=mounts) +``` + +Adding support for custom schemes: + +```python +# Support URLs like "file:///Users/sylvia_green/websites/new_client/index.html" +mounts = {"file://": FileSystemTransport()} +client = httpx.Client(mounts=mounts) +``` + +### Routing + +HTTPX provides a powerful mechanism for routing requests, allowing you to write complex rules that specify which transport should be used for each request. + +The `mounts` dictionary maps URL patterns to HTTP transports. HTTPX matches requested URLs against URL patterns to decide which transport should be used, if any. Matching is done from most specific URL patterns (e.g. `https://:`) to least specific ones (e.g. `https://`). + +HTTPX supports routing requests based on **scheme**, **domain**, **port**, or a combination of these. + +### Wildcard routing + +Route everything through a transport... + +```python +mounts = { + "all://": httpx.HTTPTransport(proxy="http://localhost:8030"), +} +``` + +### Scheme routing + +Route HTTP requests through one transport, and HTTPS requests through another... + +```python +mounts = { + "http://": httpx.HTTPTransport(proxy="http://localhost:8030"), + "https://": httpx.HTTPTransport(proxy="http://localhost:8031"), +} +``` + +### Domain routing + +Proxy all requests on domain "example.com", let other requests pass through... + +```python +mounts = { + "all://example.com": httpx.HTTPTransport(proxy="http://localhost:8030"), +} +``` + +Proxy HTTP requests on domain "example.com", let HTTPS and other requests pass through... + +```python +mounts = { + "http://example.com": httpx.HTTPTransport(proxy="http://localhost:8030"), +} +``` + +Proxy all requests to "example.com" and its subdomains, let other requests pass through... + +```python +mounts = { + "all://*example.com": httpx.HTTPTransport(proxy="http://localhost:8030"), +} +``` + +Proxy all requests to strict subdomains of "example.com", let "example.com" and other requests pass through... + +```python +mounts = { + "all://*.example.com": httpx.HTTPTransport(proxy="http://localhost:8030"), +} +``` + +### Port routing + +Proxy HTTPS requests on port 1234 to "example.com"... + +```python +mounts = { + "https://example.com:1234": httpx.HTTPTransport(proxy="http://localhost:8030"), +} +``` + +Proxy all requests on port 1234... + +```python +mounts = { + "all://*:1234": httpx.HTTPTransport(proxy="http://localhost:8030"), +} +``` + +### No-proxy support + +It is also possible to define requests that _shouldn't_ be routed through the transport. + +To do so, pass `None` as the proxy URL. For example... + +```python +mounts = { + # Route requests through a proxy by default... + "all://": httpx.HTTPTransport(proxy="http://localhost:8031"), + # Except those for "example.com". + "all://example.com": None, +} +``` + +### Complex configuration example + +You can combine the routing features outlined above to build complex proxy routing configurations. For example... + +```python +mounts = { + # Route all traffic through a proxy by default... + "all://": httpx.HTTPTransport(proxy="http://localhost:8030"), + # But don't use proxies for HTTPS requests to "domain.io"... + "https://domain.io": None, + # And use another proxy for requests to "example.com" and its subdomains... + "all://*example.com": httpx.HTTPTransport(proxy="http://localhost:8031"), + # And yet another proxy if HTTP is used, + # and the "internal" subdomain on port 5550 is requested... + "http://internal.example.com:5550": httpx.HTTPTransport(proxy="http://localhost:8032"), +} +``` + +### Environment variables + +There are also environment variables that can be used to control the dictionary of the client mounts. +They can be used to configure HTTP proxying for clients. + +See documentation on [`HTTP_PROXY`, `HTTPS_PROXY`, `ALL_PROXY`](environment_variables.md#http_proxy-https_proxy-all_proxy) for more information. diff --git a/mkdocs.yml b/mkdocs.yml index 8672dd4364..049b90518e 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -28,8 +28,10 @@ nav: - Overview: 'clients/overview.md' - Authentication: 'clients/authentication.md' - SSL: 'clients/ssl.md' + - Proxies: 'clients/proxies.md' - Timeouts: 'clients/timeouts.md' - Event Hooks: 'clients/event-hooks.md' + - Transports: 'clients/transports.md' - Guides: - Async Support: 'async.md' - HTTP/2 Support: 'http2.md' From 6cb21f4832360e3e6a81055b4ef777f349158c6b Mon Sep 17 00:00:00 2001 From: Tom Christie Date: Tue, 9 Jan 2024 14:34:45 +0000 Subject: [PATCH 10/14] Text encodings --- docs/clients/overview.md | 88 ++-------------------------------- docs/clients/text-encodings.md | 75 +++++++++++++++++++++++++++++ mkdocs.yml | 1 + 3 files changed, 79 insertions(+), 85 deletions(-) create mode 100644 docs/clients/text-encodings.md diff --git a/docs/clients/overview.md b/docs/clients/overview.md index cbaaccc51c..f6b65394c3 100644 --- a/docs/clients/overview.md +++ b/docs/clients/overview.md @@ -1,6 +1,4 @@ -# Advanced Usage - -## Client Instances +# Client Instances !!! hint If you are coming from Requests, `httpx.Client()` is what you can use instead of `requests.Session()`. @@ -10,7 +8,7 @@ !!! note "TL;DR" If you do anything more than experimentation, one-off scripts, or prototypes, then you should use a `Client` instance. -#### More efficient usage of network resources +**More efficient usage of network resources** When you make requests using the top-level API as documented in the [Quickstart](quickstart.md) guide, HTTPX has to establish a new connection _for every single request_ (connections are not reused). As the number of requests to a host increases, this quickly becomes inefficient. @@ -22,7 +20,7 @@ This can bring **significant performance improvements** compared to using the to - Reduced CPU usage and round-trips. - Reduced network congestion. -#### Extra features +**Extra features** `Client` instances also support features that aren't available at the top-level API, such as: @@ -147,86 +145,6 @@ For a list of all available client parameters, see the [`Client`](api.md#client) --- -## Character set encodings and auto-detection - -When accessing `response.text`, we need to decode the response bytes into a unicode text representation. - -By default `httpx` will use `"charset"` information included in the response `Content-Type` header to determine how the response bytes should be decoded into text. - -In cases where no charset information is included on the response, the default behaviour is to assume "utf-8" encoding, which is by far the most widely used text encoding on the internet. - -### Using the default encoding - -To understand this better let's start by looking at the default behaviour for text decoding... - -```python -import httpx -# Instantiate a client with the default configuration. -client = httpx.Client() -# Using the client... -response = client.get(...) -print(response.encoding) # This will either print the charset given in - # the Content-Type charset, or else "utf-8". -print(response.text) # The text will either be decoded with the Content-Type - # charset, or using "utf-8". -``` - -This is normally absolutely fine. Most servers will respond with a properly formatted Content-Type header, including a charset encoding. And in most cases where no charset encoding is included, UTF-8 is very likely to be used, since it is so widely adopted. - -### Using an explicit encoding - -In some cases we might be making requests to a site where no character set information is being set explicitly by the server, but we know what the encoding is. In this case it's best to set the default encoding explicitly on the client. - -```python -import httpx -# Instantiate a client with a Japanese character set as the default encoding. -client = httpx.Client(default_encoding="shift-jis") -# Using the client... -response = client.get(...) -print(response.encoding) # This will either print the charset given in - # the Content-Type charset, or else "shift-jis". -print(response.text) # The text will either be decoded with the Content-Type - # charset, or using "shift-jis". -``` - -### Using character set auto-detection - -In cases where the server is not reliably including character set information, and where we don't know what encoding is being used, we can enable auto-detection to make a best-guess attempt when decoding from bytes to text. - -To use auto-detection you need to set the `default_encoding` argument to a callable instead of a string. This callable should be a function which takes the input bytes as an argument and returns the character set to use for decoding those bytes to text. - -There are two widely used Python packages which both handle this functionality: - -* [`chardet`](https://chardet.readthedocs.io/) - This is a well established package, and is a port of [the auto-detection code in Mozilla](https://www-archive.mozilla.org/projects/intl/chardet.html). -* [`charset-normalizer`](https://charset-normalizer.readthedocs.io/) - A newer package, motivated by `chardet`, with a different approach. - -Let's take a look at installing autodetection using one of these packages... - - ```shell -$ pip install httpx -$ pip install chardet - ``` - -Once `chardet` is installed, we can configure a client to use character-set autodetection. - -```python -import httpx -import chardet - -def autodetect(content): - return chardet.detect(content).get("encoding") - -# Using a client with character-set autodetection enabled. -client = httpx.Client(default_encoding=autodetect) -response = client.get(...) -print(response.encoding) # This will either print the charset given in - # the Content-Type charset, or else the auto-detected - # character set. -print(response.text) -``` - ---- - ## Request instances For maximum control on what gets sent over the wire, HTTPX supports building explicit [`Request`](api.md#request) instances: diff --git a/docs/clients/text-encodings.md b/docs/clients/text-encodings.md new file mode 100644 index 0000000000..5565f02695 --- /dev/null +++ b/docs/clients/text-encodings.md @@ -0,0 +1,75 @@ +When accessing `response.text`, we need to decode the response bytes into a unicode text representation. + +By default `httpx` will use `"charset"` information included in the response `Content-Type` header to determine how the response bytes should be decoded into text. + +In cases where no charset information is included on the response, the default behaviour is to assume "utf-8" encoding, which is by far the most widely used text encoding on the internet. + +## Using the default encoding + +To understand this better let's start by looking at the default behaviour for text decoding... + +```python +import httpx +# Instantiate a client with the default configuration. +client = httpx.Client() +# Using the client... +response = client.get(...) +print(response.encoding) # This will either print the charset given in + # the Content-Type charset, or else "utf-8". +print(response.text) # The text will either be decoded with the Content-Type + # charset, or using "utf-8". +``` + +This is normally absolutely fine. Most servers will respond with a properly formatted Content-Type header, including a charset encoding. And in most cases where no charset encoding is included, UTF-8 is very likely to be used, since it is so widely adopted. + +## Using an explicit encoding + +In some cases we might be making requests to a site where no character set information is being set explicitly by the server, but we know what the encoding is. In this case it's best to set the default encoding explicitly on the client. + +```python +import httpx +# Instantiate a client with a Japanese character set as the default encoding. +client = httpx.Client(default_encoding="shift-jis") +# Using the client... +response = client.get(...) +print(response.encoding) # This will either print the charset given in + # the Content-Type charset, or else "shift-jis". +print(response.text) # The text will either be decoded with the Content-Type + # charset, or using "shift-jis". +``` + +## Using auto-detection + +In cases where the server is not reliably including character set information, and where we don't know what encoding is being used, we can enable auto-detection to make a best-guess attempt when decoding from bytes to text. + +To use auto-detection you need to set the `default_encoding` argument to a callable instead of a string. This callable should be a function which takes the input bytes as an argument and returns the character set to use for decoding those bytes to text. + +There are two widely used Python packages which both handle this functionality: + +* [`chardet`](https://chardet.readthedocs.io/) - This is a well established package, and is a port of [the auto-detection code in Mozilla](https://www-archive.mozilla.org/projects/intl/chardet.html). +* [`charset-normalizer`](https://charset-normalizer.readthedocs.io/) - A newer package, motivated by `chardet`, with a different approach. + +Let's take a look at installing autodetection using one of these packages... + +```shell +$ pip install httpx +$ pip install chardet +``` + +Once `chardet` is installed, we can configure a client to use character-set autodetection. + +```python +import httpx +import chardet + +def autodetect(content): + return chardet.detect(content).get("encoding") + +# Using a client with character-set autodetection enabled. +client = httpx.Client(default_encoding=autodetect) +response = client.get(...) +print(response.encoding) # This will either print the charset given in + # the Content-Type charset, or else the auto-detected + # character set. +print(response.text) +``` diff --git a/mkdocs.yml b/mkdocs.yml index 049b90518e..3e89edd7ec 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -32,6 +32,7 @@ nav: - Timeouts: 'clients/timeouts.md' - Event Hooks: 'clients/event-hooks.md' - Transports: 'clients/transports.md' + - Text Encodings: 'clients/text-encodings.md' - Guides: - Async Support: 'async.md' - HTTP/2 Support: 'http2.md' From 3b03a1c170bcfaf0e05f5ea79460767d3f73418b Mon Sep 17 00:00:00 2001 From: Tom Christie Date: Thu, 11 Jan 2024 11:18:44 +0000 Subject: [PATCH 11/14] Resource limits --- docs/clients/overview.md | 20 ++------------------ docs/clients/resource-limits.md | 13 +++++++++++++ mkdocs.yml | 1 + 3 files changed, 16 insertions(+), 18 deletions(-) create mode 100644 docs/clients/resource-limits.md diff --git a/docs/clients/overview.md b/docs/clients/overview.md index f6b65394c3..7ab0d31cb9 100644 --- a/docs/clients/overview.md +++ b/docs/clients/overview.md @@ -231,7 +231,7 @@ with tempfile.NamedTemporaryFile() as download_file: progress.update(download_task, completed=response.num_bytes_downloaded) ``` -![rich progress bar](img/rich-progress.gif) +![rich progress bar](../img/rich-progress.gif) ## Monitoring upload progress @@ -263,23 +263,7 @@ def gen(): httpx.post("https://httpbin.org/post", content=gen()) ``` -![tqdm progress bar](img/tqdm-progress.gif) - -## Pool limit configuration - -You can control the connection pool size using the `limits` keyword -argument on the client. It takes instances of `httpx.Limits` which define: - -- `max_keepalive_connections`, number of allowable keep-alive connections, or `None` to always -allow. (Defaults 20) -- `max_connections`, maximum number of allowable connections, or `None` for no limits. -(Default 100) -- `keepalive_expiry`, time limit on idle keep-alive connections in seconds, or `None` for no limits. (Default 5) - -```python -limits = httpx.Limits(max_keepalive_connections=5, max_connections=10) -client = httpx.Client(limits=limits) -``` +![tqdm progress bar](../img/tqdm-progress.gif) ## Multipart file encoding diff --git a/docs/clients/resource-limits.md b/docs/clients/resource-limits.md new file mode 100644 index 0000000000..2002428326 --- /dev/null +++ b/docs/clients/resource-limits.md @@ -0,0 +1,13 @@ +You can control the connection pool size using the `limits` keyword +argument on the client. It takes instances of `httpx.Limits` which define: + +- `max_keepalive_connections`, number of allowable keep-alive connections, or `None` to always +allow. (Defaults 20) +- `max_connections`, maximum number of allowable connections, or `None` for no limits. +(Default 100) +- `keepalive_expiry`, time limit on idle keep-alive connections in seconds, or `None` for no limits. (Default 5) + +```python +limits = httpx.Limits(max_keepalive_connections=5, max_connections=10) +client = httpx.Client(limits=limits) +``` \ No newline at end of file diff --git a/mkdocs.yml b/mkdocs.yml index 3e89edd7ec..9dd4a93696 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -30,6 +30,7 @@ nav: - SSL: 'clients/ssl.md' - Proxies: 'clients/proxies.md' - Timeouts: 'clients/timeouts.md' + - Resource Limits: 'clients/resource-limits.md' - Event Hooks: 'clients/event-hooks.md' - Transports: 'clients/transports.md' - Text Encodings: 'clients/text-encodings.md' From e0fdc696c4818494ff715821520b465856c243df Mon Sep 17 00:00:00 2001 From: Tom Christie Date: Thu, 11 Jan 2024 11:34:44 +0000 Subject: [PATCH 12/14] 'Clients' -> 'Advanced' --- docs/{clients => advanced}/authentication.md | 0 docs/{clients/overview.md => advanced/clients.md} | 0 docs/{clients => advanced}/event-hooks.md | 0 docs/{clients => advanced}/proxies.md | 0 docs/{clients => advanced}/resource-limits.md | 0 docs/{clients => advanced}/ssl.md | 0 docs/{clients => advanced}/text-encodings.md | 0 docs/{clients => advanced}/timeouts.md | 0 docs/{clients => advanced}/transports.md | 0 mkdocs.yml | 4 ++-- 10 files changed, 2 insertions(+), 2 deletions(-) rename docs/{clients => advanced}/authentication.md (100%) rename docs/{clients/overview.md => advanced/clients.md} (100%) rename docs/{clients => advanced}/event-hooks.md (100%) rename docs/{clients => advanced}/proxies.md (100%) rename docs/{clients => advanced}/resource-limits.md (100%) rename docs/{clients => advanced}/ssl.md (100%) rename docs/{clients => advanced}/text-encodings.md (100%) rename docs/{clients => advanced}/timeouts.md (100%) rename docs/{clients => advanced}/transports.md (100%) diff --git a/docs/clients/authentication.md b/docs/advanced/authentication.md similarity index 100% rename from docs/clients/authentication.md rename to docs/advanced/authentication.md diff --git a/docs/clients/overview.md b/docs/advanced/clients.md similarity index 100% rename from docs/clients/overview.md rename to docs/advanced/clients.md diff --git a/docs/clients/event-hooks.md b/docs/advanced/event-hooks.md similarity index 100% rename from docs/clients/event-hooks.md rename to docs/advanced/event-hooks.md diff --git a/docs/clients/proxies.md b/docs/advanced/proxies.md similarity index 100% rename from docs/clients/proxies.md rename to docs/advanced/proxies.md diff --git a/docs/clients/resource-limits.md b/docs/advanced/resource-limits.md similarity index 100% rename from docs/clients/resource-limits.md rename to docs/advanced/resource-limits.md diff --git a/docs/clients/ssl.md b/docs/advanced/ssl.md similarity index 100% rename from docs/clients/ssl.md rename to docs/advanced/ssl.md diff --git a/docs/clients/text-encodings.md b/docs/advanced/text-encodings.md similarity index 100% rename from docs/clients/text-encodings.md rename to docs/advanced/text-encodings.md diff --git a/docs/clients/timeouts.md b/docs/advanced/timeouts.md similarity index 100% rename from docs/clients/timeouts.md rename to docs/advanced/timeouts.md diff --git a/docs/clients/transports.md b/docs/advanced/transports.md similarity index 100% rename from docs/clients/transports.md rename to docs/advanced/transports.md diff --git a/mkdocs.yml b/mkdocs.yml index 9dd4a93696..975ed4df6b 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -24,8 +24,8 @@ edit_uri: "" nav: - Introduction: 'index.md' - QuickStart: 'quickstart.md' - - Clients: - - Overview: 'clients/overview.md' + - Advanced: + - Clients: 'clients/clients.md' - Authentication: 'clients/authentication.md' - SSL: 'clients/ssl.md' - Proxies: 'clients/proxies.md' From 89a7bf16b43da236d677eb351a586853c5a3201d Mon Sep 17 00:00:00 2001 From: Tom Christie Date: Thu, 11 Jan 2024 11:56:51 +0000 Subject: [PATCH 13/14] 'Clients' -> 'Advanced' --- mkdocs.yml | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/mkdocs.yml b/mkdocs.yml index 975ed4df6b..7c6fcbd56d 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -25,15 +25,15 @@ nav: - Introduction: 'index.md' - QuickStart: 'quickstart.md' - Advanced: - - Clients: 'clients/clients.md' - - Authentication: 'clients/authentication.md' - - SSL: 'clients/ssl.md' - - Proxies: 'clients/proxies.md' - - Timeouts: 'clients/timeouts.md' - - Resource Limits: 'clients/resource-limits.md' - - Event Hooks: 'clients/event-hooks.md' - - Transports: 'clients/transports.md' - - Text Encodings: 'clients/text-encodings.md' + - Clients: 'advanced/clients.md' + - Authentication: 'advanced/authentication.md' + - SSL: 'advanced/ssl.md' + - Proxies: 'advanced/proxies.md' + - Timeouts: 'advanced/timeouts.md' + - Resource Limits: 'advanced/resource-limits.md' + - Event Hooks: 'advanced/event-hooks.md' + - Transports: 'advanced/transports.md' + - Text Encodings: 'advanced/text-encodings.md' - Guides: - Async Support: 'async.md' - HTTP/2 Support: 'http2.md' From fb6adcd4adeb307091df0612a0d9dd963156d1b4 Mon Sep 17 00:00:00 2001 From: Tom Christie Date: Mon, 15 Jan 2024 09:43:44 +0000 Subject: [PATCH 14/14] Add client docs --- docs/advanced/clients.md | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/docs/advanced/clients.md b/docs/advanced/clients.md index 7ab0d31cb9..6905724dcd 100644 --- a/docs/advanced/clients.md +++ b/docs/advanced/clients.md @@ -1,9 +1,7 @@ -# Client Instances - !!! hint If you are coming from Requests, `httpx.Client()` is what you can use instead of `requests.Session()`. -### Why use a Client? +## Why use a Client? !!! note "TL;DR" If you do anything more than experimentation, one-off scripts, or prototypes, then you should use a `Client` instance. @@ -31,7 +29,7 @@ This can bring **significant performance improvements** compared to using the to The other sections on this page go into further detail about what you can do with a `Client` instance. -### Usage +## Usage The recommended way to use a `Client` is as a context manager. This will ensure that connections are properly cleaned up when leaving the `with` block: @@ -50,7 +48,7 @@ finally: client.close() ``` -### Making requests +## Making requests Once you have a `Client`, you can send requests using `.get()`, `.post()`, etc. For example: @@ -75,7 +73,7 @@ For example, to send a request with custom headers: 'value' ``` -### Sharing configuration across requests +## Sharing configuration across requests Clients allow you to apply configuration to all outgoing requests by passing parameters to the `Client` constructor. @@ -91,7 +89,7 @@ For example, to apply a set of custom headers _on every request_: 'my-app/0.0.1' ``` -### Merging of configuration +## Merging of configuration When a configuration option is provided at both the client-level and request-level, one of two things can happen: @@ -127,7 +125,7 @@ b'alice:ecila123' If you need finer-grained control on the merging of client-level and request-level parameters, see [Request instances](#request-instances). -### Other Client-only configuration options +## Other Client-only configuration options Additionally, `Client` accepts some configuration options that aren't available at the request level.