Skip to content

Repository navigation

Response cookies are not ingested properly in 0.14.0 #1151

Closed
#1156
Closed
Response cookies are not ingested properly in 0.14.0#1151
#1156
Labels
bugSomething isn't working

Description

@djigit0s
djigit0s
opened on Aug 8, 2020

Hi. After last updatet when loading cookies from a file, it gives an error

http/cookiejar.py", line 1947, in _really_load
domain_specified = domain.startswith(".")
AttributeError: 'NoneType' object has no attribute 'startswith'

import os
from http.cookiejar import LWPCookieJar
import httpx

h = httpx.Client(cookies=LWPCookieJar('test'))
if os.path.isfile('test'):
    h.cookies.jar.load()
h.get('https://www.google.ru')
h.cookies.jar.save()

Cookies file - httpx 0.13.3
#LWP-Cookies-2.0
Set-Cookie3: 1P_JAR="2020-08-08-21"; path="/"; domain=".google.ru"; path_spec; domain_dot; secure; expires="2020-09-07 21:47:19Z"; version=0
Set-Cookie3: NID="204=CQ898UjmES*****************************ctxJAE99ngLlZcojaaWLrqex-td7X00PooPgmVnW61VrtwCHIPSb_CPR8Ic6dpD_gRY_pBteV_4YSGU7Vu58V7uHPwNcPxZ6PMCAHt-RZSPgnxcoM"; path="/"; domain=".google.ru"; path_spec; domain_dot; expires="2021-02-07 21:47:19Z"; HttpOnly=None; version=0

Cookies file - httpx 0.14.0
#LWP-Cookies-2.0
Set-Cookie3: 1P_JAR="2020-08-08-21"; path="/"; domain=".google.ru"; path_spec; domain_dot; expires="2020-09-07 21:50:09Z"; HttpOnly=None; Secure, NID="204=NCk0**********************xnr8aaHwCrC-zMdQQ9bAS5QAp6rfv-yOiuwkYDhiN7kTyDASjlZVIJ0Zase-JQMH_YwCvaiS9I232kIRu36O684kwHD_nRXHeUVpTJTlS7mbniFmebf5nhE"; version=0

Activity

  1. lovelydinosaur commented on Aug 9, 2020

    @lovelydinosaur
    lovelydinosaur
    on Aug 9, 2020
    Member

    Heya, first things...

    1. Could you provide the full traceback?
    2. Could you double check and confirm the exact behaviour on 0.13?
    3. I don't recall us making any changes in our cookie handling? Both versions are tagged on GitHub so you could review if there are any changes?

  2. djigit0s commented on Aug 9, 2020

    @djigit0s
    djigit0s
    on Aug 9, 2020
    Author

    1. Traceback, script runned on version 0.14.0
    (manual demand for payment) djigit0s@R:/mnt/d/prj/manual demand for payment$ python test.py
    (manual demand for payment) djigit0s@R:/mnt/d/prj/manual demand for payment$ python test.py
    /usr/lib/python3.7/http/cookiejar.py:1966: UserWarning: http.cookiejar bug!
    Traceback (most recent call last):
    File "/usr/lib/python3.7/http/cookiejar.py", line 1947, in _really_load
    domain_specified = domain.startswith(".")
    AttributeError: 'NoneType' object has no attribute 'startswith'

    _warn_unhandled_exception()
    Traceback (most recent call last):
    File "/usr/lib/python3.7/http/cookiejar.py", line 1947, in _really_load
    domain_specified = domain.startswith(".")
    AttributeError: 'NoneType' object has no attribute 'startswith'

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
    File "test.py", line 7, in
    h.cookies.jar.load()
    File "/usr/lib/python3.7/http/cookiejar.py", line 1794, in load
    self._really_load(f, filename, ignore_discard, ignore_expires)
    File "/usr/lib/python3.7/http/cookiejar.py", line 1968, in _really_load
    (filename, line))
    http.cookiejar.LoadError: invalid Set-Cookie3 format file 'test': '1P_JAR="2020-08-09-09"; path="/"; domain=".google.ru"; path_spec; domain_dot; expires="2020-09-08 09:44:05Z"; HttpOnly=None; Secure, NID="204=------------------------------yl_8tbSzcfb2Y6lmG7j8kLKLigynEVHW8PqDowWASEG_tCeKBBwx_qpTfUIt6klL4UoSaX6uVK32R9mIcXDK5kN2DRIgu07yYVJB3-CSq5SEoA"; version=0'
    (manual demand for payment) djigit0s@R:/mnt/d/prj/manual demand for payment$

    2. Checked on version 0.13.0 and 0.13.3, script work.

    pay attention in the first message I indicated the results of saving cookies they are different, as if deleted some cookies

  3. florimondmanca
    added
    questionFurther information is requested
    on Aug 9, 2020

  4. florimondmanca commented on Aug 9, 2020

    @florimondmanca
    florimondmanca
    on Aug 9, 2020
    Contributor

    Yeah, something's weird.

    I ran your script against 0.14.0 (master), and 0.13.0, printing the contents dict(h.cookies) after the first run of the script.

    Looks like in 0.14.0 we're conflating the two cookies into one upon receiving the response...

    0.14.0:

    $ python debug/cookies.py
{'1P_JAR': '2020-08-09-18'}
    # test-0-14.log
#LWP-Cookies-2.0
Set-Cookie3: 1P_JAR="2020-08-09-18"; path="/"; domain=".google.com"; path_spec; domain_dot; expires="2020-09-08 18:18:25Z"; HttpOnly=None; Secure, NID="204=jte-Aw2Xlnx61T6Jtp7pDe8FnyHBrbm4SOr3zAw0ew_P-Terd-JUiALnKz4dJZKaLrb3lg-qbzFdTUedj6tDARlSymHiE2rOXGghxG7P7Ca0FlgbMa6BEJdfBb1DUUvI5FTnchaPytDbLoDjsLJJckAld0fix9c8GAGCTeM8OrQ"; version=0

    0.13.0:

    $ python debug/cookies.py
{'1P_JAR': '2020-08-09-18', 'NID': '204=za_rWjCBE8fM88zGZ_XZSVy6nUG99DpCv5BUxwGPBmiIQdXwEJMOjy7kteacGRw9doy-PIf1wxJXcn4tFHy_tZBtvKUbyJA1ewCDmbalWFUagYHJuzGD9SLxSw6sHZKggf33xy57-sQtBR3jQUWkTrhP4L4jQ0NrO1QHQDgGB54'}
    # test-0-13.log
#LWP-Cookies-2.0
Set-Cookie3: 1P_JAR="2020-08-09-18"; path="/"; domain=".google.com"; path_spec; domain_dot; secure; expires="2020-09-08 18:19:10Z"; version=0
Set-Cookie3: NID="204=fd_FJ8M55DBEyI76sXNFuLw8xWoBOBnfTBRxvy5Nz_hUGjWtA-XSWOI5MIT5JSIY78g2K_yvYokJuS8ZgoISf8oxX2vzzwSkSXeE-jM8FXH-UCTjs87T3i3pGdX8KVBbPjNk5hUUBuWRiB5ocxwqtBZaSupp0Tsc32usj9qc6NA"; path="/"; domain=".google.com"; path_spec; domain_dot; expires="2021-02-08 18:19:10Z"; HttpOnly=None; version=0

  5. florimondmanca commented on Aug 9, 2020

    @florimondmanca
    florimondmanca
    on Aug 9, 2020
    Contributor

    Doesn't seem to come from httpcore (tested against 0.9.1 with a bit of patching for the API changes…), and the git blame on Cookies shows no code changes, so… 🤷‍♂️

  6. florimondmanca commented on Aug 9, 2020

    @florimondmanca
    florimondmanca
    on Aug 9, 2020
    Contributor

    Ah-ha! Found it :-) When extracting cookies we're calling into response.headers.items(), which used to return all headers, but now returns de-duplicated headers (via #1089).

    Yet there seems to be a problem... Here's what response.headers.items() contains:

    ('set-cookie', '1P_JAR=2020-08-09-18; expires=Tue, 08-Sep-2020 18:33:35 GMT; path=/; domain=.google.com; Secure, NID=204=KWdXOuypc86YvRfBSiWoW1dEXfSl_5qI7sxZY4umlk4J35yNTeNEkw15MRaujK6uYCwkrtjihTTXZPp285z_xDOUzrdHt4dj0Z5C0VOpbvdLwRdHatHAzQs77TsaiWY78a3qU9r7KP_RbSLvLl2hlhnWFR2Hp5nWKPsAcOhQgSg; expires=Mon, 08-Feb-2021 18:33:35 GMT; path=/; domain=.google.com; HttpOnly')

    Notice how this is…?

    • "1P_JAR=2020-08-09-18; expires=Tue, 08-Sep-2020 18:33:35 GMT; path=/; domain=.google.com; Secure"
    • Plus ", "
    • Plus "NID=204=KWdXOuypc86YvRfBSiWoW1dEXfSl_5qI7sxZY4umlk4J35yNTeNEkw15MRaujK6uYCwkrtjihTTXZPp285z_xDOUzrdHt4dj0Z5C0VOpbvdLwRdHatHAzQs77TsaiWY78a3qU9r7KP_RbSLvLl2hlhnWFR2Hp5nWKPsAcOhQgSg; expires=Mon, 08-Feb-2021 18:33:35 GMT; path=/; domain=.google.com; HttpOnly"

    The values from the two Set-Cookie are indeed conflated into a single Set-Cookie: <value_1>, <value_2>, as per this line…

    httpx/httpx/_models.py

    Line 367 in 0e73be8

    self._dict[key] = self._dict[key] + b", " + value

    This is not really the problem that's causing this bug though, but rather that we're still calling .multi_items() when processing response cookies, instead of calling into response.headers.multi_items() in order to properly get all separate Set-Cookie headers:

    httpx/httpx/_models.py

    Line 1199 in 0e73be8

    for key, value in self.response.headers.items():

    (Still not sure if concatenating headers into comma-separated lists shouldn't include some conditions on what header is being flattened. Eg for Set-Cookie we probably want to only keep the latest value, instead of concatenating all values which results in an invalid Set-Cookie value syntax.)

  7. florimondmanca
    added
    bugSomething isn't working
    and removed
    questionFurther information is requested
    on Aug 9, 2020
  8. florimondmanca
    changed the title [-]Save/Load cookies from file[/-] [+]Response cookies are not ingested properly in 0.14.0[/+] on Aug 9, 2020

  9. florimondmanca commented on Aug 9, 2020

    @florimondmanca
    florimondmanca
    on Aug 9, 2020
    Contributor

    So to be clear, the fix is to change this line to use .multi_items():

    httpx/httpx/_models.py

    Line 1199 in 0e73be8

    for key, value in self.response.headers.items():

    @djigit0s Does this sound legit to you? If you're up for throwing a PR together for this feel free to give it a go!

  10. lovelydinosaur
    closed this as completedin #1156on Aug 10, 2020

  11. lovelydinosaur commented on Aug 11, 2020

    @lovelydinosaur
    lovelydinosaur
    on Aug 11, 2020
    Member

    Now resolved in 0.14.1.
    Thanks for raising this!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions