Skip to content

Storage: TypeError: request() takes at least 3 arguments (3 given) #3666

Closed
Closed
Storage: TypeError: request() takes at least 3 arguments (3 given)#3666
Assignees
dhermes
Labels
api: storageIssues related to the Cloud Storage API.

Description

@kipi78
kipi78
opened on Jul 25, 2017

Hi,

I'm having trouble while using google.cloud for Storage. I start by trying to list blobs in a bucket.
I'm trying to get threw the proxy of my organization. So I try to create my own http connection in order to specify the proxy, but even with a simple urllib3.PoolManager(), I keep getting the error and stacktrace bellow:

Error:
TypeError: request() takes at least 3 arguments (3 given)

Stacktrace:

Traceback (most recent call last)
<ipython-input-9-a7bd2d0fcc13> in <module>()
      3 authed_http = AuthorizedHttp(credentials, http)
      4 storage_client = storage.Client(_http=http)
----> 5 bucket = storage_client.get_bucket('mybucket')
      6 blobs = bucket.list_blobs()
      7 

/gpfs/user/u515866/env_python2/lib/python2.7/site-packages/google/cloud/storage/client.pyc in get_bucket(self, bucket_name)
    171         """
    172         bucket = Bucket(self, name=bucket_name)
--> 173         bucket.reload(client=self)
    174         return bucket
    175 

/gpfs/user/u515866/env_python2/lib/python2.7/site-packages/google/cloud/storage/_helpers.pyc in reload(self, client)
     97         api_response = client._connection.api_request(
     98             method='GET', path=self.path, query_params=query_params,
---> 99             _target_object=self)
    100         self._set_properties(api_response)
    101 

/gpfs/user/u515866/env_python2/lib/python2.7/site-packages/google/cloud/_http.pyc in api_request(self, method, path, query_params, data, content_type, headers, api_base_url, api_version, expect_json, _target_object)
    297         response, content = self._make_request(
    298             method=method, url=url, data=data, content_type=content_type,
--> 299             headers=headers, target_object=_target_object)
    300 
    301         if not 200 <= response.status < 300:

/gpfs/user/u515866/env_python2/lib/python2.7/site-packages/google/cloud/_http.pyc in _make_request(self, method, url, data, content_type, headers, target_object)
    191         headers['User-Agent'] = self.USER_AGENT
    192 
--> 193         return self._do_request(method, url, headers, data, target_object)
    194 
    195     def _do_request(self, method, url, headers, data,

/gpfs/user/u515866/env_python2/lib/python2.7/site-packages/google/cloud/_http.pyc in _do_request(self, method, url, headers, data, target_object)
    221         """
    222         return self.http.request(uri=url, method=method, headers=headers,
--> 223                                  body=data)
    224 
    225     def api_request(self, method, path, query_params=None,

TypeError: request() takes at least 3 arguments (3 given)

My code is the following:

from google.oauth2 import service_account
from google.cloud import storage
import urllib3
from google.auth.transport.urllib3 import AuthorizedHttp

json="pathtojson"

credentials = service_account.Credentials.from_service_account_file(json)
http = urllib3.PoolManager()
#authed_http = AuthorizedHttp(credentials, http)
storage_client = storage.Client(_http=http)
bucket = storage_client.get_bucket('mybucket')
blobs = bucket.list_blobs()

for blob in blobs:
    print(blob.name)

Python is 2.7.11.
I'm using Jupyter (notebook server is 5.0.0).

pip freeze:

Click to expand 
asn1crypto==0.22.0
backports-abc==0.5
backports.shutil-get-terminal-size==1.0.0
bleach==2.0.0
cachetools==2.0.0
certifi==2017.4.17
cffi==1.10.0
chardet==3.0.4
configparser==3.5.0
cx-Oracle==5.3
decorator==4.0.11
dill==0.2.7.1
entrypoints==0.2.3
enum34==1.1.6
functools32==3.2.3.post2
future==0.16.0
futures==3.1.1
gapic-google-cloud-datastore-v1==0.15.3
gapic-google-cloud-error-reporting-v1beta1==0.15.3
gapic-google-cloud-logging-v2==0.91.3
gapic-google-cloud-pubsub-v1==0.15.4
gapic-google-cloud-spanner-admin-database-v1==0.15.3
gapic-google-cloud-spanner-admin-instance-v1==0.15.3
gapic-google-cloud-spanner-v1==0.15.3
gapic-google-cloud-speech-v1==0.15.3
gapic-google-logging-v2==0.10.1
gapic-google-pubsub-v1==0.10.1
google-api-python-client==1.6.2
google-auth==1.0.1
google-auth-httplib2==0.0.2
google-cloud==0.26.1
google-cloud-bigquery==0.25.0
google-cloud-bigtable==0.25.0
google-cloud-core==0.25.0
google-cloud-datastore==1.1.0
google-cloud-dns==0.25.0
google-cloud-error-reporting==0.25.1
google-cloud-happybase==0.21.0
google-cloud-language==0.25.0
google-cloud-logging==1.1.0
google-cloud-monitoring==0.25.0
google-cloud-pubsub==0.26.0
google-cloud-resource-manager==0.25.0
google-cloud-runtimeconfig==0.25.0
google-cloud-spanner==0.25.0
google-cloud-speech==0.26.0
google-cloud-storage==1.2.0
google-cloud-translate==0.25.0
google-cloud-videointelligence==0.25.0
google-cloud-vision==0.25.0
google-gax==0.15.13
google-resumable-media==0.2.1
googleapis-common-protos==1.5.2
grpc-google-iam-v1==0.11.1
grpc-google-logging-v2==0.10.1
grpc-google-pubsub-v1==0.10.1
grpcio==1.4.0
html5lib==0.999999999
httplib2==0.10.3
idna==2.5
ipaddress==1.0.18
ipykernel==4.6.1
ipython==5.4.1
ipython-genutils==0.2.0
ipywidgets==6.0.0
Jinja2==2.9.6
jsonschema==2.6.0
jupyter==1.0.0
jupyter-client==5.1.0
jupyter-console==5.1.0
jupyter-core==4.3.0
MarkupSafe==1.0
mistune==0.7.4
nbconvert==5.2.1
nbformat==4.3.0
notebook==5.0.0
numpy==1.13.1
oauth2client==3.0.0
oauthclient==1.0.3
pandas==0.20.3
pandocfilters==1.4.1
pathlib2==2.3.0
pexpect==4.2.1
pickleshare==0.7.4
ply==3.8
prompt-toolkit==1.0.14
proto-google-cloud-datastore-v1==0.90.4
proto-google-cloud-error-reporting-v1beta1==0.15.3
proto-google-cloud-logging-v2==0.91.3
proto-google-cloud-pubsub-v1==0.15.4
proto-google-cloud-spanner-admin-database-v1==0.15.3
proto-google-cloud-spanner-admin-instance-v1==0.15.3
proto-google-cloud-spanner-v1==0.15.3
proto-google-cloud-speech-v1==0.15.3
protobuf==3.3.0
ptyprocess==0.5.2
pyasn1==0.2.3
pyasn1-modules==0.0.9
pycparser==2.18
Pygments==2.2.0
pysqlite==2.8.3
python-dateutil==2.6.1
pytz==2017.2
pyzmq==16.0.2
qtconsole==4.3.0
requests==2.18.1
rsa==3.4.2
scandir==1.5
simplegeneric==0.8.1
singledispatch==3.4.0.3
six==1.10.0
SQLAlchemy==1.1.11
terminado==0.6
testpath==0.3.1
tornado==4.5.1
traitlets==4.3.2
uritemplate==3.0.0
urllib3==1.21.1
wcwidth==0.1.7
webencodings==0.5.1
widgetsnbextension==2.0.0

Please feel free to reformat this issue if that could help.

Many thanks.

Activity

  1. dhermes
    added
    api: storageIssues related to the Cloud Storage API.
    on Jul 25, 2017

  2. dhermes commented on Jul 25, 2017

    @dhermes
    dhermes
    on Jul 25, 2017
    Contributor

    @kipi78 The issue is your transport:

    http = urllib3.PoolManager()
#authed_http = AuthorizedHttp(credentials, http)
storage_client = storage.Client(_http=http)

    Until #1998 is closed, you must use an httplib2.Http transport. From the docstring

    Can be any object that defines request() with the same interface as httplib2.Http.request

    Is there any reason you passed an explicit _http to the Client constructor? The leading underscore is a sign that _http isn't part of the public API.

  3. kipi78 commented on Jul 26, 2017

    @kipi78
    kipi78
    on Jul 26, 2017
    Author

    Thank you for your response.

    I wrote _http according to this doc:
    http://google-cloud-python.readthedocs.io/en/latest/storage/client.html

    Okay, I read that it was not an httplib2 object but an urllib3 object, but I tried with a httplib2.Http.
    As I said I want to get threw my proxy. Here is my code with both "simple" http transport and one with proxy settings:

    httpStorage = httplib2.Http(proxy_info = httplib2.ProxyInfo(httplib2.socks.PROXY_TYPE_HTTP,proxyUrl, int(proxyPort), True,proxyUser,proxyPassword),ca_certs = pathCertificat)
#httpStorage = httplib2.Http()
authed_http = AuthorizedHttp(credentials, httpStorage)
storage_client = storage.Client(credentials=credentials, _http=httpStorage)

    How am I supposed to use that httplib2.Http transport if not passed with the explicit _http arg ?

  4. dhermes commented on Jul 26, 2017

    @dhermes
    dhermes
    on Jul 26, 2017
    Contributor

    It seems to me that you should be passing _http=authed_http.

    I am pre-emptively closing this out, it seems you've figure out how to configure against your proxy?

    While waiting on #3674 / #1998, you may also want to check out https://github.com/GoogleCloudPlatform/httplib2shim

  5. dhermes
    closed this as completedon Jul 26, 2017

  6. dhermes commented on Jul 26, 2017

    @dhermes
    dhermes
    on Jul 26, 2017
    Contributor

    Let me know if there are still concerns and we can re-open and discuss if need be.

  7. kipi78 commented on Jul 27, 2017

    @kipi78
    kipi78
    on Jul 27, 2017
    Author

    No, my comments meant what I tried to simplify the case. I already tried with _http=authed_http which leads to the TypeError: request() takes at least 3 arguments (3 given) error. With the bellow code:

    credentials = service_account.Credentials.from_service_account_file(json)
httpStorage = httplib2.Http(proxy_info = httplib2.ProxyInfo(httplib2.socks.PROXY_TYPE_HTTP,proxyUrl, int(proxyPort), True,proxyUser,proxyPassword),ca_certs = pathCertificat)
authed_http = AuthorizedHttp(credentials, httpStorage)
storage_client = storage.Client(credentials=credentials, _http=authed_http)
bucket = storage_client.get_bucket('mybucket')

    I will have a try with httplib2shim, thank you.

  8. kipi78 commented on Jul 27, 2017

    @kipi78
    kipi78
    on Jul 27, 2017
    Author

    So, with httplib2shim, with the sample code from the doc which uses GoogleCredentials from oauth2client.client, the problem is that google-cloud-python Client only supports credentials from google-auth-library-python. But authorize is not a method of credentials from google-cloud-python Client.

    My code:

    credentials = GoogleCredentials.get_application_default()
httpStorage = httplib2.Http(proxy_info = httplib2.ProxyInfo(httplib2.socks.PROXY_TYPE_HTTP,proxyUrl, int(proxyPort), True,proxyUser,proxyPassword),ca_certs = pathCertificat)
httpStorage = credentials.authorize(httpStorage)
storage_client = storage.Client(credentials=credentials, _http=httpStorage)
bucket = storage_client.get_bucket('mybucket')

    As I need to use ca_certs, I tried to implement this:

    Instead, pass a urllib3.Pool instance http = httplib2shim.Http(pool=my_pool)

    with the bellow code:

    credentials = service_account.Credentials.from_service_account_file(json)
http = ProxyManager(proxy_url=proxyUrl, ca_certs=pathCertificat)
http = httplib2shim.Http(pool=http)
authed_http = AuthorizedHttp(credentials, http)
storage_client = storage.Client(_http=authed_http, credentials=credentials)
bucket = storage_client.get_bucket('mybucket')

    Which still leads to the same error, whether I try with a simple PoolManager or with a ProxyManager.

  9. dhermes commented on Jul 27, 2017

    @dhermes
    dhermes
    on Jul 27, 2017
    Contributor

    @jonparrott Can you weigh in here?

  10. theacodes commented on Jul 27, 2017

    @theacodes
    theacodes
    on Jul 27, 2017
    Contributor

    I'm not able to reproduce, here's the code I used:

    >>> from google.cloud import storage
>>> import google.auth
>>> import google.auth.transport.urllib3
>>> import urllib3
>>> import httplib2shim
>>>
>>> pool = urllib3.PoolManager()
>>> credentials, project = google.auth.default()
>>> authed_pool = google.auth.transport.urllib3.AuthorizedHttp(credentials, http=pool)
>>> http = httplib2shim.Http(pool=authed_pool)
>>>
>>> client = storage.Client(project=project, _http=http)
>>> list(client.list_buckets())
[<Bucket: ...>, ...]
>>>

    I would strongly encourage waiting until #1998 is closed as this will all be significantly easier then.

  11. kipi78 commented on Jul 28, 2017

    @kipi78
    kipi78
    on Jul 28, 2017
    Author

    Okay, the thing is that you passed an AuthorizedHttp to httplib2shim.Http(), while I passed the ProxyManager and applied the AuthorizedHttp on httplib2shim.Http(). So, I don't have this error anymore with your code :)

    The thing is now to use the ProxyManager, it looks like I don't manage to do it. I need to configure a proxy with user/password and a certificate. My code bellow leads to this error:

    ProxyError: ('Cannot connect to proxy.', error('Tunnel connection failed: 407 Proxy Authentication Required',))

    proxyUrl="http://host:80"
scopes=["https://www.googleapis.com/auth/devstorage.read_write"]
credentials = service_account.Credentials.from_service_account_file(json, scopes=scopes)
default_headers = make_headers(proxy_basic_auth='user:password')
http = ProxyManager(proxy_url=proxyUrl, ca_certs=pathCertificat, headers=default_headers)
authed_http = AuthorizedHttp(credentials, http)
http = httplib2shim.Http(pool=authed_http)
storage_client = storage.Client(_http=http, credentials=credentials)
bucket = storage_client.get_bucket('mybucket')

    I will certainly wait, many thanks.

  12. theacodes commented on Jul 28, 2017

    @theacodes
    theacodes
    on Jul 28, 2017 via email
    Contributor

  13. kipi78 commented on Jul 31, 2017

    @kipi78
    kipi78
    on Jul 31, 2017
    Author

    Yes it looks like the ProxyManager alone is working:

    http = ProxyManager(proxy_url=proxyUrl, ca_certs=pathCertificat, headers=default_headers)
a = http.request('GET','http://httpbin.org')
print(a.data)

    The expected HTML is printed.

  14. theacodes commented on Jul 31, 2017

    @theacodes
    theacodes
    on Jul 31, 2017
    Contributor

    I wonder if the headers are being overridden. Why happens if you make a request and specify some random headers? http.request('GET','http://httpbin.org', headers={'foo': 'bar'})?

  15. kipi78 commented on Aug 1, 2017

    @kipi78
    kipi78
    on Aug 1, 2017
    Author

    As soon as I insert an error in the login/password in the headers, this small request does not work. Same thing if I try with random headers. So, I guess the headers are not overridden.

  16. 8 remaining items

  17. theacodes commented on Aug 3, 2017

    @theacodes
    theacodes
    on Aug 3, 2017
    Contributor

    In the meantime, requests should honor the HTTP_PROXY and HTTPS_PROXY environment variables: http://docs.python-requests.org/en/master/user/advanced/#proxies

  18. kipi78 commented on Aug 4, 2017

    @kipi78
    kipi78
    on Aug 4, 2017
    Author

    Uploading and downloading is now working using the HTTP_PROXY and HTTPS_PROXY environment variables :) .
    But I still need the proxy headers to perform get_bucket.
    It would be great to have the ability to configure everything once. I'm looking forward to the next release :)

  19. theacodes commented on Aug 4, 2017

    @theacodes
    theacodes
    on Aug 4, 2017
    Contributor

    Good deal. Glad we could help. :)

  20. kipi78 commented on Aug 8, 2017

    @kipi78
    kipi78
    on Aug 8, 2017
    Author

    I have installed the new release (pip install --upgrade google-cloud) so that I now have google-cloud==0.27.0, google-cloud-core==0.26.0 and google-cloud-storage==1.3.1.

    I don't need httplib2shim anymore, right ? I can't perform get_bucket. My code:

    credentials = service_account.Credentials.from_service_account_file(json, scopes=scopes)
default_headers = make_headers(proxy_basic_auth='user:pass')
http = ProxyManager(proxy_url=proxyUrl, ca_certs=pathCertificat, proxy_headers=default_headers)
authed_http = AuthorizedHttp(credentials, http)
storage_client = storage.Client(_http=authed_http, credentials=credentials)
bucket = storage_client.get_bucket('mybucket')

    I got the following traceback:

    TypeErrorTraceback (most recent call last)
    in ()
    23 #storage_client = storage.Client(credentials=credentials, _http=httpStorage)
    24
    ---> 25 bucket = storage_client.get_bucket('mybucket')
    26 blobs = bucket.list_blobs()
    27

    /gpfs/user/u515866/env_python2/lib/python2.7/site-packages/google/cloud/storage/client.pyc in get_bucket(self, bucket_name)
    171 """
    172 bucket = Bucket(self, name=bucket_name)
    --> 173 bucket.reload(client=self)
    174 return bucket
    175

    /gpfs/user/u515866/env_python2/lib/python2.7/site-packages/google/cloud/storage/_helpers.pyc in reload(self, client)
    97 api_response = client._connection.api_request(
    98 method='GET', path=self.path, query_params=query_params,
    ---> 99 _target_object=self)
    100 self._set_properties(api_response)
    101

    /gpfs/user/u515866/env_python2/lib/python2.7/site-packages/google/cloud/_http.pyc in api_request(self, method, path, query_params, data, content_type, headers, api_base_url, api_version, expect_json, _target_object)
    288 response = self._make_request(
    289 method=method, url=url, data=data, content_type=content_type,
    --> 290 headers=headers, target_object=_target_object)
    291
    292 if not 200 <= response.status_code < 300:

    /gpfs/user/u515866/env_python2/lib/python2.7/site-packages/google/cloud/_http.pyc in _make_request(self, method, url, data, content_type, headers, target_object)
    181 headers['User-Agent'] = self.USER_AGENT
    182
    --> 183 return self._do_request(method, url, headers, data, target_object)
    184
    185 def _do_request(self, method, url, headers, data,

    /gpfs/user/u515866/env_python2/lib/python2.7/site-packages/google/cloud/_http.pyc in _do_request(self, method, url, headers, data, target_object)
    210 """
    211 return self.http.request(
    --> 212 url=url, method=method, headers=headers, data=data)
    213
    214 def api_request(self, method, path, query_params=None,

    /gpfs/user/u515866/env_python2/lib/python2.7/site-packages/urllib3/request.pyc in request(self, method, url, fields, headers, **urlopen_kw)
    64 return self.request_encode_url(method, url, fields=fields,
    65 headers=headers,
    ---> 66 **urlopen_kw)
    67 else:
    68 return self.request_encode_body(method, url, fields=fields,

    /gpfs/user/u515866/env_python2/lib/python2.7/site-packages/urllib3/request.pyc in request_encode_url(self, method, url, fields, headers, **urlopen_kw)
    85 url += '?' + urlencode(fields)
    86
    ---> 87 return self.urlopen(method, url, **extra_kw)
    88
    89 def request_encode_body(self, method, url, fields=None, headers=None,

    /gpfs/user/u515866/env_python2/lib/python2.7/site-packages/google/auth/transport/urllib3.pyc in urlopen(self, method, url, body, headers, **kwargs)
    212
    213 response = self.http.urlopen(
    --> 214 method, url, body=body, headers=request_headers, **kwargs)
    215
    216 # If the response indicated that the credentials needed to be

    /gpfs/user/u515866/env_python2/lib/python2.7/site-packages/urllib3/poolmanager.pyc in urlopen(self, method, url, redirect, **kw)
    434 kw['headers'] = self._set_proxy_headers(url, headers)
    435
    --> 436 return super(ProxyManager, self).urlopen(method, url, redirect=redirect, **kw)
    437
    438

    /gpfs/user/u515866/env_python2/lib/python2.7/site-packages/urllib3/poolmanager.pyc in urlopen(self, method, url, redirect, **kw)
    319 response = conn.urlopen(method, url, **kw)
    320 else:
    --> 321 response = conn.urlopen(method, u.request_uri, **kw)
    322
    323 redirect_location = redirect and response.get_redirect_location()

    /gpfs/user/u515866/env_python2/lib/python2.7/site-packages/urllib3/connectionpool.pyc in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
    615 connection=response_conn,
    616 retries=retries,
    --> 617 **response_kw)
    618
    619 # Everything went great!

    /gpfs/user/u515866/env_python2/lib/python2.7/site-packages/urllib3/response.pyc in from_httplib(ResponseCls, r, **response_kw)
    465 strict=strict,
    466 original_response=r,
    --> 467 **response_kw)
    468 return resp
    469

    TypeError: init() got an unexpected keyword argument 'data'

  21. dhermes commented on Aug 8, 2017

    @dhermes
    dhermes
    on Aug 8, 2017
    Contributor

    @kipi78 That's correct. Now the httplib2.Http.request interface is wrong, i.e. it doesn't accept the keyword data

  22. kipi78 commented on Aug 8, 2017

    @kipi78
    kipi78
    on Aug 8, 2017
    Author

    So, I don't understand. Am I doing something wrong here ?

  23. dhermes commented on Aug 8, 2017

    @dhermes
    dhermes
    on Aug 8, 2017
    Contributor

    Yes / no. You no longer need to use httplib2shim and using it is now an error.

  24. kipi78 commented on Aug 8, 2017

    @kipi78
    kipi78
    on Aug 8, 2017
    Author

    I agree with that but I'm no longer using it as you can see in the code.

  25. dhermes commented on Aug 8, 2017

    @dhermes
    dhermes
    on Aug 8, 2017
    Contributor

    Sorry I'm not familiar, where does ProxyManager come from?

  26. kipi78 commented on Aug 8, 2017

    @kipi78
    kipi78
    on Aug 8, 2017
    Author

    ProxyManager is from urllib3 import ProxyManager. It is supposed to act like a PoolManager. And I guess it is the best way for me to configure my proxy, but I'm opened to a better solution :)

  27. theacodes commented on Aug 8, 2017

    @theacodes
    theacodes
    on Aug 8, 2017
    Contributor

    @kipi78 now that requests is our default transport you don't need to use httplib2 or urllib3. You should be able to just set the HTTP_PROXY and HTTPS_PROXY environment variables and not pass anything special at all into _http.

  28. kipi78 commented on Aug 9, 2017

    @kipi78
    kipi78
    on Aug 9, 2017
    Author

    Hi,

    Alright, I was wondering how to specify the certificate but it is a bundled feature of requests, we just need to export REQUESTS_CA_BUNDLE. It is now working fine with a very simple storage_client = storage.Client() :)

    Thank you !

  29. theacodes commented on Aug 9, 2017

    @theacodes
    theacodes
    on Aug 9, 2017 via email
    Contributor
  30. JustinBeckwith
    assigned
    dhermes
    on Feb 1, 2021
  31. parthea
    added a commit that references this issue on Oct 21, 2023

    chore(deps): update dependency cryptography to v2.9.2 [(#3666)](Googl…

    d8c5141
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

api: storageIssues related to the Cloud Storage API.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions