This repository was archived by the owner on Mar 6, 2026. It is now read-only.
Repository navigation
This repository was archived by the owner on Mar 6, 2026. It is now read-only.
The magics API do not support setting client options #160
Description
Environment details
- OS type and version: jupyter notebook
- Python version: Python 3.7.3
- pip version: pip 20.1.1
- google-cloud-bigquery version: 1.25.0
Steps to reproduce
This is a missing option that do exists in google bigquery client - "https://googleapis.dev/python/bigquery/latest/generated/google.cloud.bigquery.client.Client.html#google.cloud.bigquery.client.Client"
But is missing from "https://github.com/googleapis/python-bigquery/blob/master/google/cloud/bigquery/magics.py"
Code example
client = bigquery.Client(
project=project,
credentials=context.credentials,
default_query_job_config=context.default_query_job_config,
client_info=client_info.ClientInfo(user_agent=IPYTHON_USER_AGENT),
)Note the the client_option is missing in creating the client in the code above
The patch to apply to fix this flow:
diff --git a/google/cloud/bigquery/magics.py b/google/cloud/bigquery/magics.py
index 7128e32..00e26d0 100644
--- a/google/cloud/bigquery/magics.py
+++ b/google/cloud/bigquery/magics.py
@@ -140,6 +140,10 @@
Out[7]: num
...: -------
...: 0 17
+ Executing query with job ID: bf633912-af2c-4780-b568-5d868058632b
+ Query executing: 2.61s
+ Query complete after 2.92s
+
"""
from __future__ import print_function
@@ -183,6 +187,7 @@ class Context(object):
self._project = None
self._connection = None
self._default_query_job_config = bigquery.QueryJobConfig()
+ self._client_options = google.api_core.client_options.ClientOptions()
@property
def credentials(self):
@@ -244,6 +249,30 @@ class Context(object):
def project(self, value):
self._project = value
+ @property
+ def client_options(self):
+ """ google.api_core.client_options.ClientOptions: client options to be used through IPython
+ Magics
+
+ Note::
+ The client options does not need to be explicitly defined if you have an
+ no special network conenctions are required. Normally you would
+ be using https://www.googleapis.com/ end point
+
+ Example:
+ Manually setting the endpoint:
+
+ >>> from google.cloud.bigquery import magics
+ >>> client_option = {}
+ >>> client_option['api_endpoint'] = "https://some.special.url"
+ >>> magics.context.client_options = client_option
+ """
+ return self._client_options
+
+ @client_options.setter
+ def client_options(self, value):
+ self._client_options = value
+
@property
def default_query_job_config(self):
"""google.cloud.bigquery.job.QueryJobConfig: Default job
@@ -459,6 +488,18 @@ def _create_dataset_if_necessary(client, dataset_id):
"name (ex. $my_dict_var)."
),
)
+@magic_arguments.argument(
+ "--options",
+ nargs="+",
+ default=None,
+ help=(
+ "Parameters to format the query string. If present, the --options "
+ "flag should be followed by a string representation of a dictionary "
+ "in the format (Union[google.api_core.client_options.ClientOptions, Dict])"
+ "or a reference to a dictionary in the same format. The dictionary "
+ "key/value matches does at google.api_core.client_options.ClientOptions"
+ ),
+)
def _cell_magic(line, query):
"""Underlying function for bigquery cell magic
@@ -501,6 +542,7 @@ def _cell_magic(line, query):
credentials=context.credentials,
default_query_job_config=context.default_query_job_config,
client_info=client_info.ClientInfo(user_agent=IPYTHON_USER_AGENT),
+ client_options=context.client_options
)
if context._connection:
client._connection = context._connection
Activity
plamut commentedon Jul 11, 2020
@boazsade Thanks for the suggestion and the patch!
I'll just check if Product has any objections (cc: @shollyman), and if not, we'll add this.
P.S.: If confirmed, would you mind opening an actual PR with added tests, or are you fine with us taking the patch as a base and create a PR ourselves?
(if significant portions of the patch are used, it would probably still require signing the CLA, though)
boazsade commentedon Jul 11, 2020
You can take this pach, this is something that we really need to be part of the code base, thanks for the reply
boazsade commentedon Aug 2, 2020
@plamut is there any progress with this issue? I was unable to create pull request, what about applying the patch? was that approved? thanks
plamut commentedon Aug 3, 2020
@boazsade Not at this point, unfortunately, but I can re-check at this week's meeting. Lately one of the primary things in focus are preparations to dropping Python 2 support and moving them to the new code generator (for those who have some auto-generated parts).
yoavcloud commentedon Oct 11, 2020
Hi @plamut. I wanted to check in and see if there's an update regarding including this patch in an upcoming release. One of our customers is in the process of migrating their data environment to BigQuery and this has become a priority. Thanks in advance.
cguardia commentedon Oct 11, 2020
@yoavcloud Hi, I'm working on a PR to add your patch. Should be ready for review in a couple of days.
yoavcloud commentedon Oct 11, 2020
Thank you so much @cguardia!
cguardia commentedon Oct 12, 2020
@yoavcloud Well, I've been looking at this and have a couple of questions. While
api_endpointdoes seem like a useful option to have here, I don't think the rest of the client options are practical for magics, with the possible exception ofquota_project_id. Passing in callables in the command line is just not possible, plus getting the lexer to parse the dictionary format in the first place would not be trivial. Have you tried the patch? I don't believe it can work as it is. We could probably add the api endpoint as an option, though. Would that cover your use case?boaz-work commentedon Oct 12, 2020
The patch file was generated by me (@boazsade), this patch was generate after we tested this to work. This was the reason we wanted it - after verify it. What we really need is the option to set the end point value (so that the traffic can pass not directly to google but through other authenticated host. As for the other parameters (like quota_project_id) I didn't test that, the reason I wrote it the way I did was to keep it with the same as the internal calls, but this can be dropped - again we really need the option to set the api_endpoint, other parameters as less important for us.
tswast commentedon Oct 12, 2020
We don't necessarily need command-line options for all of these (though it wouldn't hurt to add). The most important thing is to make sure it is settable from
google.cloud.bigquery.magics.context, as this object sets default values for all calls to the magics.cguardia commentedon Oct 13, 2020
@boazsade sorry about my confusion with the patch's authorship. Setting the client options directly on the context worked, but the command line options did not, so I changed the patch to add an
--api_endpointparameter instead. This is under review now.boaz-work commentedon Oct 13, 2020
good to here it, looking forward for this
yoavcloud commentedon Oct 15, 2020
Thank you everyone!