Improving the "Hidden Div" Solution #290

Currently, there are several ways to share data between callbacks. These are outlined here: https://dash.plot.ly/sharing-data-between-callbacks

Some users find these methods complex or clunky and others have trouble understanding which method is best for them.

Let's discuss some ways that we can improve this aspect of Dash.

How Are Hidden Divs Being Used?
A1. To store intermediate state that is expensive to generate (in time, # of connections to e.g. a database, in CPU) and to use that intermediate state in several callbacks
- e.g. 5 graphs generated from the same expensive SQL query
A2. To store data for controls that are being used across multiple pages or tabs
- e.g. Tab 1 is the "controls" page and Tab 2 has several graphs that depend on those controls. Since the controls aren't in the component tree in Tab 2, they are stored in a global div that can be updated from Tab 1 and referenced in Tab 2.
A3. As a mutable, derived-state store
- e.g. Appending click data from a graph into a click data store that contains all of the previous click events (https://community.plot.ly/t/solution-persistent-click-events/6590)

How Is Memoization Caching Being Used?
B1. Like A1, except persisting the data across all user sessions, perhaps with a time expiring cache
- e.g. There is a dropdown with 5 values and it makes 5 expensive unique SQL queries. These get cached so that all users retrieve the same cached value.
B2. A filesystem/redis based version of A1. User sessions IDs are created in the def layout function and used as the memoization key

How Could These Methods Be Improved?

• For A1 - Allowing Multiple Outputs in a callback would remedy many (all?) of these issues. It's also the easiest to grok as there are no intermediate steps.

• For A2 and B1 - Easier Serialization would reduce friction. This is the Refactor #1 issue that I hear encounter: confusion around serializing and deserializing JSON or even inability to do so (with complex data structures).

  • Perhaps we can just use pickle instead so that serialization and deseralization "just works"?
    • We would need to sign it to prevent untrusted arbitrary python code from executing
    • And we should encrypt it to prevent users from unknowingly shipping secret intermediate data to the front end
    • I believe that pickle is quite a bit slower, almost unusable: https://community.plot.ly/t/working-on-large-datasets-comparison-with-shiny/6199/6
    • If we had per-component-serialization, our Store component could handle all types of objects:
      • If it was a dataframe, it could serialize it with arrow: https://community.plot.ly/t/apache-arrow-and-dash-community-thread/7381
      • If it was a primative type, we could use our existing JSON serializer
      • If it was something else, we could use pickle
      • Currently, we don't have a way for components to have a custom Python serializer/deserializer. But it would be a good idea to incorporate this into the component architecture. I outlined how to do this here: [WIP] encrypted_ props #254 (comment)
  • The Redis memoization is hard to wrap your head around.
    • Decorators are confusing, require a fair amount of refactoring: instead of df = df.read_csv it's df = get_data() and get_data is a complex decorated function (see the solution in https://dash.plot.ly/sharing-data-between-callbacks)
    • Perhaps we can create an object that looks and feels like a global dict but is in fact saving to Redis
  • The per-user Redis cache is also confusing as it requires unique user IDs (see solution in https://dash.plot.ly/sharing-data-between-callbacks)
    • Perhaps we can leverage flask session IDs or roll our own and just make them available to everyone
    • If there was a nice global dict object, then users could figure out how to assign items on a per-user basis to it, e.g. dash.datastore[dash.user_id] = some_dataframe(dash.user_id)

• For A2 and B1 - Faster Serialization

  • See Apache Arrow serialization above
  • If the issue is network delay, then we could come up with a better solution with Redis.
    • Redis Cache Solution will act differently than dcc.Store as the dcc.Store would be part of the Dash DAG: updating it would trigger other callbacks. The Redis Cache Solution wouldn't, it would just memoize

There's some more stuff to discuss here, but this is as far as I've gotten in exploring this problem space today.

cc @plotly/dash