You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Mar 6, 2026. It is now read-only.
Repository navigation
This repository was archived by the owner on Mar 6, 2026. It is now read-only.
Resource objects such as *JobConfig do not error or warn when setting unknown properties #303
Thanks for stopping by to let us know something could be better!
PLEASE READ: If you have a support contract with Google, please create an issue in the support console instead of filing on GitHub. This will ensure a timely response.
Please run down the following list and make sure you've tried the usual "quick fixes":
If you are still having issues, please be sure to include as much information as possible:
Environment details
OS type and version: MACOS Catalina 10.15.5
Python version: 3.7.6
pip version: 20.0.2
google-cloud-bigquery version: 1.28.0
Steps to reproduce
Step 1. Extract BQ tabke data to bucket
Code example
# from google.cloud import bigquery
# client = bigquery.Client()
# bucket_name = 'my-bucket'
destination_uri = "gs://{}/{}".format(bucket_name, "shakespeare.json")
dataset_ref = bigquery.DatasetReference(project, dataset_id)
table_ref = dataset_ref.table("shakespeare")
job_config = bigquery.job.ExtractJobConfig()
job_config.destinationFormat = bigquery.DestinationFormat.NEWLINE_DELIMITED_JSON
job_config.compression = bigquery.Compression.GZIP
extract_job = client.extract_table(
table_ref,
destination_uri,
# Location must match that of the source table.
location="US",
job_config=job_config,
) # API request
extract_job.result() # Waits for job to complete.
Fetch the table from bucket and load into bq
Code example
from google.cloud import bigquery
# Construct a BigQuery client object.
client = bigquery.Client()
# TODO(developer): Set table_id to the ID of the table to create.
# table_id = "your-project.your_dataset.your_table_name"
job_config = bigquery.LoadJobConfig(
schema=[
bigquery.SchemaField("name", "STRING"),
bigquery.SchemaField("post_abbr", "STRING"),
], #This is not shakespeare schema, as just copied pasted the code provided on google docs
source_format=bigquery.SourceFormat.NEWLINE_DELIMITED_JSON,
)
uri = "gs://{}/{}".format(bucket_name, "shakespeare.json")
load_job = client.load_table_from_uri(
uri,
table_id,
location="US", # Must match the destination dataset location.
job_config=job_config,
) # Make an API request.
load_job.result() # Waits for the job to complete.
destination_table = client.get_table(table_id)
print("Loaded {} rows.".format(destination_table.num_rows))
Stack trace
Error while reading data, error message: JSON table encountered too many errors, giving up. Rows: 1; errors: 1. Please look into the errors[] collection for more details.
Error while reading data, error message: JSON processing encountered too many errors, giving up. Rows: 1; errors: 1; max bad: 0; error percent: 0
Error while reading data, error message: JSON parsing error in row starting at position 0: Parser terminated before end of string
However when I extract the table using bq command line and follow the example of:
changed the title [-]Extracting bq table as NEWLINE_DELIMITED_JSON may not be a correct format[/-][+]Resource objects such as *JobConfig do not error or warn when setting unknown properties[/+]on Oct 6, 2020
Activity
tswast commentedon Oct 6, 2020
I see a typo. There is no
destinationFormatproperty onExtractJobConfigThe correct name is
destination_formathttps://googleapis.dev/python/bigquery/latest/generated/google.cloud.bigquery.job.ExtractJobConfig.html#google.cloud.bigquery.job.ExtractJobConfig.destination_format
Perhaps we need ways of warning about unknown properties?
[-]Extracting bq table as NEWLINE_DELIMITED_JSON may not be a correct format[/-][+]Resource objects such as *JobConfig do not error or warn when setting unknown properties[/+]tswast commentedon Oct 6, 2020
On internal Q&A forum question 5801905264132096, this same issue was encountered, but with setting Table properties on a TableReference object.
Nakulbajaj101 commentedon Oct 6, 2020
Ah thanks, that might be it, a warning would be lovely