How do I write code that talks to two different datasets with two different sets of credentials?

@pcostell

Let's say I have:

/creds1.json
/creds2.json
dataset1
dataset2
gcloud.datastore

How do I pull down Person:1 from dataset1, retrieve the 'name' property, and write it back to Log:2 in dataset2?

Here's my best guess so far:

from gcloud import credentials
from gcloud import datastore

creds1 = credentials.get_for_service_account_json('/creds1.json')
creds2 = credentials.get_for_service_account_json('/creds2.json')

connection1 = datastore.Connection(credentials=creds1)
connection2 = datastore.Connection(credentials=creds1)

person1_key = datastore.Key('Person', 1, dataset_id='dataset1')
log2_key = datastore.Key('Log', 2, dataset_id='dataset2')

person1 = datastore.get(datastore.Key(person1_key), connection=connection1, dataset_id='dataset1')
log2 = datastore.get(datastore.Key(log2_key), connection=connection2, dataset_id='dataset2')

log2['data'] = person1['name']
datastore.put(log2, connection=connection2, dataset_id='dataset2')

I only got that by digging through tons of code. It made me sad.

What I want to write:

from gcloud.datastore import get_connection

dataset1 = get_connection(credentials_json='/creds1.json').get_dataset('dataset1')
dataset2 = get_connection(credentials_json='/creds2.json').get_dataset('dataset2')

person1 = dataset1.get('Person', 1)
log2 = dataset2.get('Log', 2)

log2['data'] = person1['name']
log2.put()

It seems that somewhere along the way, we lost the "hierarchy" of high-level concepts (Datastore -> Connection -> Dataset -> Entity) so that things don't seem to know who their "parent" is in the tree on the way up.

This means things like dataset.get() because it needs to be provided it's connection and credentials. It seems we've tried to overcome this by storing defaults, but that blows up when you have more than one set of credentials...

Maybe I'm totally misunderstanding?

I'm thinking that it'd be cool if we could allow three things:

datastore. that accepts all the parameters to be absurdly specific (here is the connection, here is the dataset_it, etc)
datastore. that has lots of Nones as default parameters, and we "go get the default" if you left it as None (ie, connection=None -> get_default_connection())
Datastore -> Connection -> Dataset -> Entity drill down that "pre-fills" these things going up the chain. That is, I can say:

connection = datastore.get_connection(...)
dataset = connection.get_dataset(...)
entity = dataset.get(...)

This means that the datastore module, Connection, and Dataset all would likely have the same methods, just with fewer things you can specify because some of those fieldsare specified when you "ask for" the next level down (ie, a Dataset knows it's dataset_id, so you don't have an option to provide that) .

Example

datastore.py:

def get(key, connection=None, dataset_id=None):
  connection = connection or get_default_connection()
  dataset_id = dataset_id or get_default_dataset_id()

# Means I can do: datastore.get(Key('Person', 1), dataset_id='dataset1')

dataset.py:

class Dataset(object):

  def __init__(self, dataset_id, connection=None):
    self.dataset_id = dataset_id
    self.connection = connection

  def get(self, key):
    return datastore.get(key, dataset_id=self.dataset_id, connection=self.connection)

/cc @pcostell @dhermes @tseaver

@jgeewax

@jgeewax I'm working on documenting this in a branch:
https://github.com/dhermes/gcloud-python/tree/add-gcloud-auth-docs

This is easy to do currently, as your first code snippet shows.

Let's not lose sight of the fact that

very few users will ever talk to multiple datasets simultaneously or require multiple connections
auth code is "one and done", so reducing your snippet from 6 lines to 8 lines is not really that meaningful when the actual code would be more like an entire code base and the auth code will still be just 4 lines

I think "bundling" defaults via an Environment is a good concept (somewhat back to a Dataset) but not one an average user should be worried with. (More like 99% of users than average.)

We had a massive discussion before (were you not cc'ed?):
https://docs.google.com/document/d/1xKw8Tz6lfN5uOcv0my4uUgFSeDoRzwCLho2UGlBpqDU/edit

Part of the removal of Dataset was also the removal of all the factory methods like Connection.get_dataset, Connection.query, etc.

The Dataset class was really just a string with factory methods.

The philosophy behind this was

If we need factory methods, our constructors need more work.

and also in the spirit of being idiomatic Python (https://www.python.org/dev/peps/pep-0020/)

There should be one-- and preferably only one --obvious way to do it.

This is easy to do currently, as your first code snippet shows.

I ... really do not think what I wrote was easy. :(

very few users will ever talk to multiple datasets simultaneously or require multiple connection

This will become less and less true over time. Right now people are tossing everything into "their datastore" because there was .. only one "datastore" (from google.appengine.ext import db). That is not going to be the case anymore, people will want to store certain data in certain regions and be able to pull from their US-based region, and put that data in their European region...

This certainly won't be "the most common thing", but I don't think that changes much.

We had a massive discussion before (were you not cc'ed?):

and

If we need factory methods, our constructors need more work.

I definitely wasn't ... and I disagree...

DI works really well when you don't need to change something mid-flight in the dependency chain (ie, if you only ever need one connection, one set of credentials, one dataset at a time). If you need to change one piece, you have to manually create the entire dependency chain and put the final product into the function call. That sucks.

Since we do need to support multiple dependency chains in the same code at the same time... I think we need concepts to help craft all of that.

There should be one-- and preferably only one --obvious way to do it.

I agree. I don't think this breaks with that. (I'm saying these aren't the same "it" -- these are two different things, each with their own obvious way to do them.)

If you need to store data (one set of credentials, one dataset, one project) the obvious way is datastore.put().

If you need to interact with multiple datasets, use multiple sets of credentials, or multiple connections, the obvious way is to drill down and create things that you need.

Most important thing which we can all agree on

If you need to change one piece, you have to manually create the entire dependency chain and
put the final product into the function call. That sucks.