Dynamic Layouts (Wildcard Props)

It would be really powerful if we could have the layout of an app, not just individual component properties, respond to state changes. Some use cases that come to mind from previous (non-dash) apps I've made:

Creating an arbitrary number of graphs, based on a multi-select dropdown
Chained filters to a database query, where all previous filters determine the options available to the next filter
You might want a new component (or set of components) for each item in some query results. Perhaps each items gets its own graph, or each gets a full report page.

There would be one new piece of user API: wildcards in property callbacks. But first some notes about callbacks affecting the layout (via children)

Thoughts @plotly/dash ?

Layout callbacks

(edit: we can already do this by having a callback output the children of some component - So these are just regular app.callback, I've updated the pseudocode to reflect this but there are some notes I wanted to add)

app.layout = html.Div([
    dcc.Dropdown(id='dd', multi=True, ...),
    html.Div(id='graphs')
])

@app.callback(Output('graphs', 'children'), [Input('dd', 'value')])
def graphs(selected):
    return html.Div([graph(val) for val in selected])

def graph(val):
    return dcc.Graph(id='graph-{}'.format(val), figure={...})

That example uses val to construct the id. Importantly, Graph elements set the key prop to match id independent of their original location, which means an existing Graph will be reused independent of its position among siblings, which both improves performance and would preserve any GUI interactions on the graph for the original item even if a new item is added before it.

Two related thoughts:

For dynamic layout support, should we extend key=id to all core components? It doesn't look like most of them do this yet, though one or two do. id is tied to the meaning of the component via callbacks so I can't see any downside to keeping this correspondence everywhere, anyone disagree?
dash_html_components allows users to separately specify id and key for all elements. That's probably fine, but the dynamic layout docs should highlight importance key. For example if you make a larger html.Div of a graph and some controls surrounding it, you'd like that whole item to be reused.

There's at least one very common use case that on its surface requires a circular dependency, which would be nice to avoid: "remove this item", like an ❎ button to remove a pane. For the case of creating panes from a dropdown or other list, we could just say "don't do that" (since this would be two separate controls for the same thing, and you can always deselect the item in the list where you selected it). But in other cases you just have an "add new item" button (so you'd use n_clicks for that button, and State for the existing list, to add new items). I suppose we could have the ❎ button just permanently hide the item, and you need to check the item visibility in the rest of your callbacks? Any better solution to this case? I suppose it's possible that we could just plumb it up to have the ❎ delete its own parent... maybe that won't get flagged as a circular dep, though in principle it is one.

Wildcards in regular (property) callbacks

When we have these repeating items, a single callback will need to manage a whole class of output, and some callbacks will want to use an entire class or a subset of that class as an input. I propose a wildcard syntax based on str.format that can handle integers and strings, and returns the matched values in kwargs.

Single items can be used in Output, this creates one set of dependencies for each matching item. If an identifier has already been used in Output, that same identifier can be used in Input or State too.
- Single integer: Output('graph-{n:d}', 'figure') -> kwargs['n'] = int
- Single string: Output('info-{item:s}-summary', 'children') -> kwargs['item'] = str
  (should we make :s the default, so you only need to specify it if you want an integer match?)
Multi-item matches can be used only in Input and State, and the corresponding args (and kwargs) will be lists. I suppose if you have two multi-item matches in a single id you would get a nested list...
- All integers: Input('text-input-{*m:d}', 'value') -> kwargs['m'] = list(int)
- All previous integers: Input('dropdown-{*m<n:d}', 'value) -> kwargs['m'] = list(int)
  (requires n to be used in the Output)
- All strings: Input('options-{*v:s}', 'value') -> kwargs['v'] = list(str)

Some examples:

Interdependencies within one dynamically-created pane:

@app.callback(
    Output('graph-{name:s}', 'figure'),
    [Input('log-scale-{name:s}', 'value')]
)
def make_graph(is_log_scale, name):
    return {'data': get_data(name), 'layout': {'yaxis': {'type': 'log' if is_log_scale else 'linear'}}

Use a list of filters to construct a query:

@app.callback(
    Output('results', 'children'),
    [Input('filter-field-{*n:d}', 'value'), Input('filter-values-{*n:d}', 'value')]
)
def get_count(fields, values, n):
    # we don't use n but our function needs to accept it - it's a list of integers
    # or could just take **kwargs
    filters = ['{} in ("{}")'.format(field, '","'.join(vals)) for field, vals in zip(fields, values)]
    res = run_query('select count(*) from users where ' + ' and '.join(filters))[0][0]
    return 'Found {} matching users'.format(res)

Use all previous filters to find values available to this filter:

@app.callback(
    Output('filter-values-{n:d}', 'options'),
    [Input('filter-field-{n:d}', 'value'),
     Input('filter-field-{*m<n:d}', 'value'),
     Input('filter-values-{*m<n:d}', 'value')]
)
def get_values(field, prev_fields, prev_vals, n, m):
    filters = ['{} in ("{}")'.format(f, '","'.join(v)) for f, v in zip(prev_fields, prev_vals)]
    res = run_query('select distinct(' + field + ') from users where ' + ' and '.join(filters))
    return [{
        'label': capitalize(v[0]),
        'value': v[0]
    } for v in res]

@nicolaskruchten

@nicolaskruchten reminded me we can already alter the layout by including components in the children property - thanks, that certainly simplifies things! updated the above to reflect this, but the notes about these callbacks still apply.

Use a list of filters to construct a query:
[...]

@app.callback(
    Output('results', 'children'),
    [Input('filter-field-{*n:d}', 'value'), Input('filter-values-{*n:d}', 'value')]
)
def get_count(fields, values, n):
    # we don't use n but our function needs to accept it - it's a list of integers

For n, if the layout was a list of inputs that couldn't be deleted, then it'd be like [0, 1, 2, 3, 4]. But, if the inputs could each be deleted, then it might end up being like [1, 4, 6]. Am I understanding this correctly?

I'm going to try to write the "todo mvc" app with this framework (example app: http://todomvc.com/examples/react/#/, specifications: https://github.com/tastejs/todomvc/blob/master/app-spec.md#functionality)

OK, here's a stab at writing the todo mvc. Things to look out for in the code:

Deleting nodes is certainly interesting. I've added some comments in the code about some different possible approaches
Passing components (children) as State is new for me. I've only done this when children is a string, not when it's a list of components. I'm not sure how we handle this currently (i.e. do we convert the component json back into a proper Dash Component?)
Appending components won't scale well when we have a large number of components (you have to pass all of the components as state, and then append to that list, and then return the list). perhaps, like delete, there is an "operations" api for "append", "extend", "insert", and "delete" that doesn't require passing back the entire list. Haven't really thought this one through.
I had a couple of places where I needed to use wildcard multiple outputs. Wasn't clear to me if this was in the spec or not.
All in all, only 150 lines, which is less than the pure React version (https://github.com/tastejs/todomvc/blob/gh-pages/examples/react/js/app.jsx) and it felt pretty natural to write: each little feature of the todo list has its own self-contained function. pretty modular.

app.layout = html.Div([
    dcc.Input(
        placeholder='What needs to be done?',
        id='add-todo'
    ),

    html.Div(id='todo-container', children=[]),

    html.Div([
        html.Span(id='counter'),
        html.RadioItems(
            id='todo-filter',
            options=[
                {'label': i, 'value': i} for i in [
                    'All', 'Active', 'Completed'
                ]
            ],
            value='All'
        ),
        html.Span(id='clear-todos')
    ])
])

def create_todo_item(todo_text, todo_number):
    return html.Div(
        id='item-container-{}'.format(todo_number),
        children=[
            dcc.CheckList(
                options=[{'label': '', 'value': 'selected'}],
                values=[],
                id='item-selected-{}'.format(todo_number)
            ),
            html.Span(todo_text),
            html.Span(
                'Remove',
                id='item-remove-{}'.format(todo_number)
            )
        ]
    )


# Add a todo to the end of the list
@app.callback(Output('todo-container', 'children'),
              [Input('add-todo', 'n_enter_timestamp')],  
              [State('add-todo', 'value'),
               State('todo-container', 'children'),
def append_todo(n_enter_timestamp, todo_text, existing_todos):
    # If children is a component, do we convert it
    # before passing it into a callback?
    # I don't think I've ever tested this, I've only
    # used `children` when the content is a string
    existing_todos.append(create_todo_item(
        todo_text, len(existing_todos)
    ))
    return existing_todos


# Hide/show items depending on the toggle buttons on the bottom
@app.callback(Output('item-container-{*n:d}', 'style'),
              [Input('todo-filter', 'value')],
              [State('item-selected-{*n:d}', 'value')])
def toggle_items(todo_filter, values, item_numerical_ids):
    todo_styles = {}
    for (item_id, value) in zip(item_numerical_ids, values):
        if todo_filter == 'All':
            todo_styles[item_id] = {'display': 'block'}
        elif todo_filter == 'Active':
            if 'selected' in value:
                todo_styles[item_id] = {'display': 'block'}
            else:
                todo_styles[item_id] = {'display': 'none'}
        else:
            if 'selected' in value:
                todo_styles[item_id] = {'display': 'none'}
            else:
                todo_styles[item_id] = {'display': 'block'}
    return todo_styles


# Update the counter at the bottom of the page
@app.callback(Output('counter', 'children'),
              [Input('item-selected-{*n:d}', 'value')])
def update_counter(values, item_numerical_ids):
    # values for a single checklist is an array
    # in this case, [] or ['selected'].
    # since there are multiple inputs, this would be an array of arrays?
    # e.g. `[ [], ['selected'], [], [], ['selected'] ]`
    return '{} items completed'.format(len([
        item if 'selected' in item for item in values
    ]))


# Display "Clear completed" if there are completed tasks
@app.callback(Output('clear-completed', 'children'),
              [Input('item-selected-{*n:d}', 'value')])
def display_clear_completed(values, item_numerical_ids):
    if len([item if 'selected' in item for item in values]):
        return 'Clear completed'
    else:
        return None


# Strike-out the todo text when the item is selected
@app.callback(Output('item-container-{n:d}', 'style'))
              [Input('item-selected-{n:d}', 'values')])
def update_todo_style(value):
    if 'selected' in value:
        return {
            'text-decoration': 'line-through',
            'color': 'grey'
        }
    else:
        return {}


# Remove a todo item
@app.callback(Output('item-container-{n:d}', 'children')
              [Input('item-remove-{n:d}', 'n_clicks')])
def remove_item(n_clicks):
    # OK, so returning `None` doesn't actually remove the item
    # And if there is CSS applied, the empty content might actually be visible
    # (although with multiple outputs, the callback could update `style` to
    #  {'display': 'none'} too)
    # How do we solve this?
    # - Return a special value that actually deletes the item?
    #   `return dash.operations.Remove`
    # - Instead of `dash.dependencies.Output`, it's `dash.dependencies.Remove`?
    #   and it's `Remove('item-container-{n:d}')` (no property needed).
    # - Pass all of the todos back (`Output('todo-container', 'children')`)
    #   into the callback and filter out the one that was clicked on.
    #   This doesn't scale well if the content is large (e.g. a list of graphs)
    # - Pass the todos container back but delete it with a clientside update:
    #   `Output('todo-container', 'children')`, `R.filter(R.not(R.equals(n)), children)`
    return None


# Remove all of the todo items
# Not sure if this is possible: I want to delete a set of components
# when I click on Clear completed.
# So, I think I need a wildcard output mapping, with checkboxes as state,
# and triggered by a click.
# If we have `return dash.operations.Remove`, then perhaps this could be
# conditional on the checkboxes. I'll try writing this out below.
@app.callback(Remove('item-container-{*n:d}'),
              [Input('clear-todos', 'n_clicks')],
              [State('item-selected-{*n:d}', 'values')])
def remove_selected_items(n_clicks, item_values, item_numerical_ids):
    # Since the "Output" (er... the `Remove`) is a wildcard, we'd
    # need some way to map our return value to each of the items.
    output = {}
    for (selected_values, numerical_id) in zip(item_values, item_numerical_ids):
        if 'selected' in selected_values:
            output[numerical_id] = dash.operations.Remove()
        else:
            output[numerical_id] = dash.operations.Keep()

Next up, I'd like try to create the late elasticsearch falcon UI. This'll be more of a challenge of hidden/dynamic inputs

I started writing the elasticsearch UI in the screnshot above and I found that if I prefixed all of my controls with the same string, then I could write a single callback that would handle an arbitrary set of inputs (many inputs hide/show depending on which elasticsearch columns are available or which elasticsearch aggregation options are selected). Really sweet!

One thing that I'm finding a little tedious is the string concatenation and constants management (reusing the same prefix across callbacks and components). Also, I find the number casting syntax a little tough to read.

I'm curious if we could expand out our regex groups to be stricter key-value pairs (key representing the "group"). An input could belong to multiple groups with multiple key-value pairs.

This is inspired by html's form attribute (https://stackoverflow.com/a/6644180/4142536).

In the examples below, we have:

OutputPattern / InputPattern / StatePattern: Used to "match" repeated sets of inputs and outputs by a certain key in group. In the wildcard pattern above, this was e.g. city-{s}
InputGroup / StateGroup - A way to get a set of inputs/states that match a certain filter. In the wildcard pattern above, this was e.g. city-{*:s}

Multiple 1-1 relationships.

html.Div([
    html.Div([

        dcc.Input(group={'type': 'input', 'city': i}),
        dcc.Graph(group={'type': 'graph', 'city': i})

    ]) for i in cities # e.g. ['nyc', 'mtl', 'la']
])

@app.callback(
    OutputPattern(group='city', filter={'type': 'graph'}, property='figure'),
    [InputPattern(group='city', filter={'type': 'input'}, property='value')])
def update(value):
    # value is e.g. 'nyc', 'mtl', 'la'
    return {
        'data': [{
            'y': compute_something(value)
        }]
    }

Single N-1 relationship

html.Div([
    html.Div([
        dcc.Input(group={'city': i}),
        for i in city
    ]), # e.g. ['a', 'b', 'c']
    dcc.Graph(id='graph')
])

@app.callback(
    Output('graph', 'figure'),
    [InputGroup(filter={'city': '*'}, property='value')])
def update_graph(cities, values):
    # e.g. cities=['nyc', 'mtl'], values=['foo', 'bar']
    return {
        'data': [{'y': compute_something(cities, values)}]
    }

Multiple N-M relationships

html.Div([
    html.Div([

        html.Div([
            html.Label(experiment_id),
            dcc.Input(group={
                'experiment_id': experiment_id, 'setting': setting
            })

        # variable number of settings for each experiment
        ] for setting in get_settings(experiment_id)),
        dcc.Graph(group={
            'experiment_id': experiment_id, 'computation': 'voltage'
        }),
        dcc.Graph(group={
            'experiment_id': experiment_id, 'computation': 'current'
        })

    ] for experiment_id in experiments),
])


@app.callback(
    OutputPattern(group='experiment_id', filter={'computation': '*'}, property='figure'),
    [InputPattern(group='experiment_id', filter={'setting': '*'}, property='value')])
def update(experiment_id, computations, settings, setting_values):
    # experiment_id: e.g. 'experiment-2015-01-40'
    # computations: ['voltage', 'current']
    # settings: e.g. ['pressure', 'temperature']
    # settings_values: e.g. [1, 4]
    experiment_data = get_experiment_data(experiment_id)
    model_data = compute_data(experiment_data, zip(settings, setting_values))

    # return multiple outputs, keyed by their filter group
    return {
        'voltage': {
            'data': [{'y': model_data['voltage']}],
            'layout': {'title': 'voltage'}
        },
        'current': {
            'data': [{'y': model_data['current']}],
            'layout': {'title': 'current'}
        }
    }

Outputs targeting different groups

html.Div([
    html.H1('Inputs'),
    html.Div([
        html.Label(vegetable),
        html.Div([
            html.Label(attribute)
            dcc.Input(group={
                'attribute': attribute,
                'vegetable': vegetable
            }),
            # e.g. weight, length, color
            for attribute in get_food_attributes()
        ])
        # e.g. 'lettuce', 'tomato'
        for vegetable in get_vegetables()
    ]),

    html.Hr(),

    html.H1('Computations'),

    html.Div([
        html.Label('Total weight'),
        html.Div(id='weight')
    ]),

    html.Div([
        html.Div([
            html.Label('Cost of {}'.format(vegetable)),
            html.Div(group={'vegetable': vegetable, 'computation': 'cost'})
        for vegetable in get_vegetables()
    ]),

    html.Div([
        html.Div([
            html.Label('Average {}'.format(attribute)),
            html.Div(group={'attribute': attribute, 'computation': 'average'})
        for vegetable in get_vegetables()
    ]),

])


@app.callback(Output('weight', 'children'),
              [InputGroup(filter={'vegetable': '*', 'attribute': 'weight'},
                          property='value')])
def compute_weight(vegetables, weights):
    return sum(weights)


@app.callback(OutputPattern(group='vegetable', filter={'computation': 'cost'}),
              [InputPattern(group='vegetable',
                            filter={'attribute': '*'})])
def compute_cost(vegetable, attributes, attribute_values):
    # vegetable: e.g. 'lettuce'
    # attributes: e.g. ['weight', 'length', ...]
    # attribute_values: e.g. [3, 15, ...]
    return compute_vege_cost(vegetable, attributes, attribute_values)


@app.callback(OutputPattern(group='attribute', filter={'computation': 'average'}),
              [InputPattern(group='attribute',
                            filter={'vegetable': '*'})])
def compute_average(attribute, vegetables, vegetable_values):
    # attribute: e.g. 'weight'
    # vegetables: e.g. ['lettuce', 'tomato', ...]
    # vegetable_values: e.g. [3, 15, ...]
    return mean(vegetable_values)

A single 1-Many update

html.Div([
    dcc.Input(id='alpha'),
    dcc.Graph(group={'id': 'graph-1'}),
    dcc.Graph(group={'id': 'graph-2'}),
    dcc.Graph(group={'id': 'graph-3'}),
    dcc.Graph(group={'id': 'graph-4'}),
])

@app.callback(OutputPattern(group='id', property='figure'), [Input('alpha', 'value')])
def update_graphs(output_ids):
    # output_ids: [1, 2, 3, 4]
    return {
        'graph-1': {
            'data': [{...}],
            'layout': {...}
        },
        'graph-2': {
            'data': [{...}],
            'layout': {...}
        },
        'graph-3': {
            'data': [{...}],
            'layout': {...}
        },
        'graph-4': {
            'data': [{...}],
            'layout': {...}
        }        
    }

Some additional notes:

The values in the group keyword (key-value pairs that classify an input) can be strings or numbers. This simplifies the wildcard casting a bit.
They group keys can be descriptive, making the callback signatures pretty easy to read
In OutputPattern/InputPattern, group keyword and value is repeated. Feels too repetitive.
We're applying filters over the components but not the properties. Something still feels off here: with control groups, we're basically getting lucky that every control has the same property name (value).
I just realized that this general problem could also solve:
- Multiple outputs
- Callbacks where not all of the inputs are defined at once

The positional call signature doesn't feel write to me. In flask, the flask.request import is this variable that flask writes on every request and it's packed with all of the meta info about the request (e.g. flask.request.headers or flask.request.params). Maybe we could do the same thing in Dash as a backwards compatible way to provide all of this info. With group, we could provide nice key-value pairs in this object.
Pulling from an example below, we have:

@app.callback(
    OutputPattern(group='experiment_id', filter={'computation': '*'}, property='figure'),
    [InputPattern(group='experiment_id', filter={'setting': '*'}, property='value')])
def update(experiment_id, computations, settings, setting_values):
    # experiment_id: e.g. 'experiment-2015-01-40'
    # computations: ['voltage', 'current']
    # settings: e.g. ['pressure', 'temperature']
    # settings_values: e.g. [1, 4]

but with a global object, this could be:

@app.callback(
    OutputPattern(group='experiment_id', filter={'computation': '*'}, property='figure'),
    [InputPattern(group='experiment_id', filter={'setting': '*'}, property='value')])
def update():
    # dash.request.group: e.g. 'experiment-2015-01-40'
    # dash.request.outputs: ['voltage', 'current']
    # dash.request.inputs:
    # {'computation': [['pressure', 1], ['temperature', 4]}

We could end up stuffing other data in there in the future, like which input has changed or when the last input changed. We'd be able to add stuff without changing backwards compatibility too. There was an original discussion about this call signature here (never resolved or really investigated): Determining Which Input Has Been Fired #291

Here's a rough draft at the elasticsearch UI. Seems possible! I was particularly curious about the "range aggregation" type: for that type, the UI has an arbitrary number of min/max inputs. So, not only is that UI component dynamic, it's also variable. Seems like all of the controls can inerhit the same prefix/control group and be used in a single callback to create the query. Really awesome!
I think that the amount of code to create this UI would be comparable to my original react-redux implementation and as easy to grok.

ID_PATTERNS = {
    'aggregation': 'aggregation_option_',
    'aggregation_range_min_': 'aggregation_range_min_',
    'aggregation_range_max_': 'aggregation_range_max_',
}

IDS = {
    'aggregate_type' '{}_type'.format(ID_PREFIXES['aggregation']),
    'aggregate_column' '{}_column'.format(ID_PREFIXES['aggregation']),
    'aggregation_order': '{}_order'.format(ID_PREFIXES['aggregation']),
    'aggregation_size': '{}_size'.format(ID_PREFIXES['aggregation']),
    'aggregation_date_size': '{}_date_size'.format(ID_PREFIXES['aggregation']),
    'aggregation_histogram_size': '{}_histogram_size'.format(ID_PREFIXES['aggregation']),
}


def layout():
    mappings = query_elasticsearch_mappings()
    return html.Div([
        dcc.Dropdown(
            id='elasticsearch_mappings',
            options=mappings
        ),

        html.Label('Search'),
        dcc.Input(id='search'),

        # Aggregate dynamic control container
        html.Div([
            html.Label('Aggregate By'),
            dcc.Dropdown(id=IDS['aggregate_type'], options=[]),

            html.Label('Over Column'),
            dcc.Dropdown(id=IDS['aggregate_column'], options=[]),

            html.Div('aggregation_controls')
        ]),

        html.Details([
            html.Summary('Preview Query Body'),
            html.Pre(id='query')
        ])

    ])


@app.callback(Output('aggregate_type', 'options'),
              [Input('elasticsearch_mappings', 'value')])
def aggregation_type_options(elastic_mappings):
    return [
        {
            'label': i,
            'value': i,
            'disabled': not elastic_has_column(i, elastic_mappings)
        } for i in [
            'none',
            'terms', 'histogram', 'date_histogram',
            'range', 'date_range', 'ip_range',
            'significant_terms'
        ]
    ]


@app.callback(Output('aggregate_column', 'options'),
              [Input('aggregate_type', 'value')],
              [State('elasticsearch_mappings', 'value')])
def aggregation_column_options(aggregation_type, elastic_mappings):
    return [
        {
            'label': i,
            'value': i,
            'disabled': not elastic_type_supports_aggregation(name, type)
        } for (name, type) in zip(get_column_names(elastic_mappings), get_column_types(elastic_mappings))
    ]


@app.callback(Output('aggregation_controls', 'children'),
              [Input('aggregation_type', 'value')])
def display_aggregation_controls(aggregation_type):
    if aggregation_type == 'none':
        return None

    elif aggregation_type == 'terms':
        return html.Div([
            html.Label('Order (Optional)'),
            dcc.Dropdown(
                id=IDS['aggregation_order'],
                options=[
                    {'label': 'Ascending by count', 'value': 'asc-count'},
                    {'label': 'Descending by count', 'value': 'desc-count'},
                    {'label': 'Ascending alphanumerically', 'value': 'asc-alphanumerically'},
                    {'label': 'Descending alphanumerically', 'value': 'desc-alphanumerically'},
                ]
            )
        ])
    elif aggregation_type == 'histogram':
        return html.Div([
            html.Label('Interval Size'),
            dcc.Input(id=IDS['aggregation_histogram_size'])
        ])
    elif aggregation_type == 'date_histogram':
        # similar sort of thing here
        pass
    elif aggregation_type == 'range':
        # this one is unique as there could be N number
        # of ranges. these are basically custom, user-defined bins
        return html.Div([
            html.Button(id='add-range'),
            html.Div(id='range-container', children=[])
        ])


# add N number of ranges.
# this is similar to the "todo-mvc" problem
@app.callback(Output('range-container', 'children'),
              [Input('add-range', 'n_clicks')],
              [State('range-container', 'children')])
def add_range(n_clicks, existing_ranges):
    existing_ranges.append(html.Div([
        html.Label('Min'),
        dcc.Input(
            id='{}{}'.format(
                IDS['aggregation_range_min_'],
                len(existing_ranges)
            )
        ),
        html.Label('Max'),
        dcc.Input(
            id='{}{}'.format(
                IDS['aggregation_range_max_'],
                len(existing_ranges)
            )
        ),
    ]))
    return existing_ranges


@app.callback(Output('query', 'children'),
              [Input('search', 'value'),
               Input('aggregation_type', 'value'),
               Input('aggregation_column', 'value'),
               Input(ID_PATTERNS['aggregation'] + '_{*:s}', 'value')])
def create_query(search, aggregation_type, property_ids, property_values):
    '''
    `property_ids` and `property_values` are dynamic and will
    depend on which aggregation options are selected (if any)

    Some examples:
    - `aggregation_type='none'`
        - `property_ids=[]`
        - `property_values=[]`
    - `aggregation_type='terms'`
        - `property_ids=['order']`
        - `property_values=['asc-count']`
    - `aggregation_type='range'`
        - `property_ids=[]`
        - `property_values=[]`
    - `aggregation_type='range'`
        - `property_ids=[
            'range_min_0', 'range_max_0',
            'range_min_1', 'range_max_1',
        ]`
        - `property_values=[0, 1, 4, 6]`
    '''
    return generate_query(search, aggregation_type, property_ids, property_values)

Also, it seems like this problem might solve an issue with "partially defined" inputs/outputs. In the current version of Dash, there is this limitation that either all of the inputs/outputs need to be present on the page or none of them. So, if you wanted to share inputs across multiple tabs (via a store), it wouldn't be possible as the Store would need to be omnipresent, while it's inputs and outputs would hide and show as the tabs render.

Here is a quick sketch on how this might be done with dynamic callbacks:

DEFAULT_CONTROLS = {
    'weight': 5,
    'length': 10,
    'width': 4
}

app.layout = html.Div([

    dcc.Store(id='store', value=DEFAULT_CONTROLS),

    dcc.Tabs(
        options=[
            {'label': i, 'value': i}
            for i in ['Controls', 'Results 1', 'Results 2']
        ],
        id='tabs',
        value='Controls'
    ),

    html.Div(id='container')
])


@app.callback(Output('container', 'children'),
              [Input('tabs', 'value')],
              [State('store', 'value')])
def display_container(tab, controls):
    if tab == 'Controls':
        return html.Div([
            html.Div([
                html.Label(i),
                dcc.Input(value=controls[i], id='controls-{}'.format(i)),
            ]) for i in ['weight', 'length', 'width'])
    elif tab == 'Results 1':
        return computed_results_1(controls)
    elif tab == 'Results 2':
        return computed_results_2(controls)


# In the current version of Dash, this isn't possible because
# on the second tab, `Store` is visible but the inputs aren't.
# In Dash, either _all_ of the inputs and outputs need to be visible
# or _none_ of them.
@app.callback(Output('store', 'value'),
              [Input('controls-{*}', 'value')])
def store_controls(control_names, control_values):
    return {i: v for (i, v) in zip(control_names, control_values)}

@chriddyp

Looking at duplicate IDs: #299 #320 - validating the initial layout is great but I suspect we'll want a dynamic uniqueness test, particularly when we have wildcard components. For example @chriddyp's TODO app would break in a very confusing way: it sets the id of an added item based on the length of the list, but if you delete from the middle of the list, the last item may have an id greater than the length 💥 (the easy solution for this, I think, would just be to use n_submit as the number in the new item ID)

One thing that I'm finding a little tedious is the string concatenation and constants management (reusing the same prefix across callbacks and components). Also, I find the number casting syntax a little tough to read.

I'm curious if we could expand out our regex groups to be stricter key-value pairs

Interesting idea - then we can construct an id / React key out of these values for a specific item, for uniqueness & performance, but that need not be visible to the user. The pattern Component(group={...}) seems funny to me though, as group there really functions as the id - it's the grouping variables AND their values which together become its unique identifier. So can we just keep the name id? ie if id is a string the component is a regular non-grouped item, but if it's a dict the component is addressable as part of a group?

The complexity of OutputPattern, InputGroup etc is a bit confusing to me - could we not just use Output and Input plus the same dict structure with special entries for "one arbitrary value", "all values", and "all (numerically) smaller values"? Since this would be a value that already has a key, it wouldn't need any more information than that. Value type would only matter for "all smaller", we could just throw an error in that case if we encountered a non-numeric value.

from dash.dependencies import ALL, ANY, ALLSMALLER  # or just '*', '?', '<' perhaps?

app.layout = html.Div([
  dcc.Input(id={'city': 'Boston', 'measure': 'Population'}),
  dcc.Input(id={'city': 'Boston', 'measure': 'Area'}),
  html.P(id={'city': 'Boston', 'measure': 'Density'}),
  dcc.Input(id={'city': 'Montreal', 'measure': 'Population'}),
  dcc.Input(id={'city': 'Montreal', 'measure': 'Area'}),
  html.P(id={'city': 'Montreal', 'measure': 'Density'}),
  html.P(id='total-pop'),
  html.Pre(id='restatement'),

  dcc.Input(id={'item': 0, 'type': 'val'}),
  html.P(id={'item': 1, 'type': 'sum'}),
  dcc.Input(id={'item': 1, 'type': 'val'}),
  html.P(id={'item': 2, 'type': 'sum'}),
  dcc.Input(id={'item': 2, 'type': 'val'}),
  html.P(id={'item': 3, 'type': 'sum'}),
  dcc.Input(id={'item': 3, 'type': 'val'}),
  html.P(id={'item': 4, 'type': 'sum'})
])

@app.callback(Output({'city': ANY, 'measure': 'Density'}, 'children'),
              [Input({'city': ANY, 'measure': 'Population'}, 'value'),
               Input({'city': ANY, 'measure': 'Area'}, 'value')])
def density(population, area, city):
    # city would be a kwarg here, so its name matters but not its position?
    # or, with inspection of the function signature, we could support omitting it and
    # optionally using the global request object idea
    return population / area

@app.callback(Output('total-pop', 'children'),
              [Input({'city': ALL, 'measure': 'Population'}, 'value')])
def total(populations, city_list):
    # populations and city_list are both lists
    return sum(populations)

@app.callback(Output('restatement', 'children'),
              [Input({'city': ALL, 'measure': ALL}, 'value')])
def restate_info(values, city_list, measure_list):
    # note we have 2 independent wildcards on the same input here, but we'll
    # give 1D arrays in response, with values, city_list, and measure_list matching by index:
    # city_list = ['Boston', 'Boston', 'Boston', 'Montreal', 'Montreal', 'Montreal']
    # measure_list = ['Population', 'Area', 'Density', 'Population', 'Area', 'Density']
    # BUT this will break per @chriddyp's comment that we're "getting lucky that every
    # control has the same property name (value)." - for html.P it would need to be children.
    return '\n'.join('{} of {}: {}'.format(*i) for i in zip(measure_list, city_list, values))

@app.callback(Output({'item': ANY, 'type': 'sum'}, 'children'),
              [Input({'item': ALLSMALLER, 'type': 'val'}, 'value')])
def running_sum(entries, item, item_list):
    # item is the single 'item' value for the output
    # item_list is the list of all 'item' values matching 'type': 'val'
    # entries is a list of all values for the components in item_list
    return sum(entries)

Revision: move matching IDs info from args to a global var

I really like the idea of a global variable containing the match information! For a lot of simpler applications you don't need this information, so this would limit the function signature to just the explicit inputs and state. And it avoids sticky questions about whether you make these positional args (in what order, if there are multiple wildcards?), or kwargs that you need to know the naming convention for...

I think the most straightforward and complete way to do this would be to exactly match the signature of app.callback for the non-wildcard cases, and expand to the actual items found when you have wildcards. So for example:

# no wildcards
@app.callback(Output('id1, 'children'),
              [Input('id2', 'value'), Input('id3', 'value')],
              [State('id4', 'children')])
def update1(val2, val3, children4):
    # in this case it's normally useless, since it's just copying the info directly above, but
    # you could imagine a reusable function that gets decorated by various inputs & outputs
    # needing to know which instance is being called
    # dash.request.output = Output('id1, 'children')
    # dash.request.inputs = [Input('id2', 'value'), Input('id3', 'value')]
    # dash.request.state = [State('id4', 'children')]

# with wildcards
@app.callback(Output({'item': ANY, 'type': 'sum'}, 'children'),
              [Input({'item': ALLSMALLER, 'type': 'val'}, 'value')])
def running_sum(entries):
    # dash.request.output = Output({'item': 2, 'type': 'sum'}, 'children')
    # NOTE the nested array for inputs, tells you the arg (entries) is an array
    # dash.request.inputs = [[Input({'item': 0, 'type': 'val'}, 'value'),
    #                         Input({'item': 1, 'type': 'val'}, 'value')]]
    # dash.request.state = []

# with multiple outputs - still call it output? or outputs?
@app.callback(Output({'item': ALL, 'type': 'sum'}, 'children'),
              [Input({'item': ALL, 'type': 'val'}, 'value')])
def running_sum(entries):
    # dash.request.output = [Output({'item': 1, 'type': 'sum'}, 'children'),
    #                        Output({'item': 2, 'type': 'sum'}, 'children'),
    #                        Output({'item': 3, 'type': 'sum'}, 'children'),
    #                        Output({'item': 4, 'type': 'sum'}, 'children')]
    # dash.request.inputs = [[Input({'item': 0, 'type': 'val'}, 'value'),
    #                         Input({'item': 1, 'type': 'val'}, 'value'),
    #                         Input({'item': 2, 'type': 'val'}, 'value'),
    #                         Input({'item': 3, 'type': 'val'}, 'value')]]
    # dash.request.state = []

Prop mismatch

Now, what about the problem of different component_prop for different items in the set? I'm tempted to say "just make a new set" for each distinct prop. For example change

html.P(id={'city': 'Boston', 'measure': 'Density'})
# part of the (broken) set Input({'city': ALL, 'measure': ALL}, 'value')

to

html.P(id={'city': 'Boston', 'calculated_measure': 'Density'})
# part of the valid set Input({'city': ALL, 'calculated_measure': ALL}, 'children')

I suppose we could invent a way to distinguish between them when creating the Input though, something like:

Input({'city': ALL, 'measure': ALL}, (('children', {'measure': 'density'}), ('value', {})))

But that seems ugly and hard to use, for not that much benefit. Or perhaps we augment the component itself with an extra mapping of props? Like:

html.P(id={'city': 'Boston', 'measure': 'Density'}, propMap: {'value': 'children'})

Then this particular component could use 'value' as a synonym for 'children' in callback signatures. Or we could augment both the Input and the P components to use a totally new prop in callbacks.

Anyway either of these features could be layered on later, they wouldn't be required for the rest of dynamic callbacks to work.

@chriddyp

Following on from the discussion on the forum I've had a go at implementing the todoMVC example, adapting the @chriddyp version above

This first attempt is the minimum possible changes I could do to the original while making use of my proposed new way to link to callbacks. I have still required wildcard inputs/state in this version and have had to introduce a few tricks/new features. They feel logical to me as functional programming concepts if that is the direction you want to take things:

Dynamically generated callback functions (not sure if these are currently possible):

# The component declaration:

    return html.Div(
        id='item-container-{}'.format(todo_number),
        style=app.compute(todo_style_completed(itemSelectedId)),

# The callback factory:

# Strike-out the todo text when the item is selected
def todo_style_completed(itemSelectedId):
    
    #This could alternatively be declared right alongside the component
    #when being built. I may explore that in the next version
    @app.callback([Input(itemSelectedId, 'value')])
    def inner(value):
        if 'selected' in value:
            return {
                'text-decoration': 'line-through',
                'color': 'grey'
            }
        else:
            return {}
    
    return inner

Chained generation of properties (a bit like redux reducers), with Current() being a special case of State() that can be accessed by any function in the chain.

# The component declaration:
# in this case the order doesn't matter, but in other cases it might

    html.Div(
        id='todo-container', 
        children=app.compute(
                    add_item, 
                    remove_item, 
                    remove_selected_items, 
                    default=[]
                )
    )

#The callback:

# Remove a todo item
@app.callback([Input('item-remove-{n:d}', 'n_clicks')],[Current()])
def remove_item(n_clicks, id, existing_todos):
    # It might be inefficient to bring in the full children tree
    # but at least the calls can be chained together so that they are only
    # brought back once for a given change and then passed through each 
    # function in python
    return [todo for todo in existing_todos if todo.id != 'item-container-{}'.format(id)]

Next I would like to try refactoring it to remove some setting of ids and explore how it would work if Event() and maybe Input() were both declared at the point of declaration. Although maybe we always want id to be set if it is going to be used to derive key in future.

I've settled on my preferred balance where Event() and Output() links are moved to the components, while Input() and State() are left as callback decorators with the option for patterns to select multiple inputs.

You can see what this looks like.

I have also added some other new features that make this work nicely for dynamic layouts such as a Source() function on callbacks that are Event triggered which tends to be how event handlers work elsewhere.

The best example is the removal buttons against each item:

html.Span(
            'Remove',
            click=app.trigger(remove_item),

            #Don't know if custom props are supported, but it would
            #make things a bit easier for linking related components
            todo_number=todo_number
        )

# Remove a todo item, which now just needs to know the 
# which specific item got clicked
@app.callback(state=[Source("todo_number"), Current()])
def remove_item(todo_number, existing_todos):
    return [todo for todo in existing_todos if todo.todo_number != todo_number]

Guys, what's the status of this issue? Is there anything dev-related you need help with?

It just hasn't bubbled up to the top of the TODO list yet. Feel free to take a stab at it, though I suspect it will be a fairly involved effort. Otherwise, we hope to get to it within the next couple of months.

@christianwengert

@christianwengert we’re always looking for organizations to fund & prioritize our open source work. could you get in touch with us here? https://plot.ly/products/consulting-and-oem/

For dynamic layout support, should we extend key=id to all core components?

I would encourage it, but let me ask a slightly different question: is there a compelling reason for prohibiting an application from setting arbitrary properties? By "arbitrary properties" I mean any valid html attribute that isn't explicitly defined by a component.

I ask because, in the absence of a common spec, raising a ValueError when an application tries to annotate a node with custom key/value pairs seems awfully perverse. I would understand if that were something that maybe, as web developers, we just learn to live without because transpiling a Python application to React is hard and there's a few tradeoffs. For example, if you try to monkey-patch component instances with a bunch of random props at instantiation time, Dash spits out an error and refuses to build the app until you remove the offending property names.

Why though? If doing so inevitably caused horrible runtime errors, it would make sense that Dash should not only warn but prevent you from shooting yourself in the foot. But when I suppressed those compile-time checks in my local fork (which requires also tweaking the definition of dash.development.base_component.Component.to_plotly_json(), but it's a trivial fix) the app works beautifully. It works exactly as expected. The foreign props are even visible to the dev_tools_ui debugger's callback graph, which was pleasantly surprising.

Originally I had only intended to modify Component.__init__() to add aria- and data- to a component's list of valid prefixes (tangential rant: the base class defers to its subclasses to define _valid_wildcard_prefixes, which is something that's already been codified in the W3C spec. Why? Lots of any third-party components don't define any ._valid_wildcard_prefixes at all, which makes extending them impossible -- and imho they shouldn't have to). But once I saw how to accomplish that, I realized that it would be just easy to abolish the property check completely to support any property. And boom, suddenly I became 2-3x more productive in Dash. It's helpful for so many use cases. Yesterday, for example, I was annoyed that my dbc.Input component applied spellchecking to a name field and didn't expose an option in its signature that to disable that. But since spellcheck is a valid HTML attribute that is parsed by the browser, setting that attribute to "false" on the Python object directly disables the red squigglies where they're unwanted. This would be impossible under the current regime of whitelisted attributes.

@alexcjohnson

From chatting with @alexcjohnson today... even with wildcard props implemented as in #1103, to make a nice Todo app we need array operations callbacks described in #968

HI, I am a user waiting on this. I'd like to dynamically change the label in a button to create a single button that changes function based on state. At a minimum two states, like a toggle but with text inside.

change build (plotly#475)

change build (#475)

Navigation Menu

Repository navigation

Dynamic Layouts (Wildcard Props) #475

Layout callbacks

Wildcards in regular (property) callbacks

Revision: move matching IDs info from args to a global var

Prop mismatch

5 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Dynamic Layouts (Wildcard Props) #475

Description

Layout callbacks

Wildcards in regular (property) callbacks

Activity

alexcjohnson commented on Nov 30, 2018

chriddyp commented on Dec 14, 2018

chriddyp commented on Dec 14, 2018

chriddyp commented on Dec 15, 2018

chriddyp commented on Dec 15, 2018

chriddyp commented on Dec 18, 2018

chriddyp commented on Dec 18, 2018

chriddyp commented on Dec 18, 2018

alexcjohnson commented on Dec 19, 2018

alexcjohnson commented on Dec 22, 2018

Revision: move matching IDs info from args to a global var

Prop mismatch

mungojam commented on Dec 22, 2018

mungojam commented on Dec 23, 2018

deemson commented on Mar 20, 2019

alexcjohnson commented on Mar 20, 2019

5 remaining items

alexcjohnson commented on Sep 19, 2019

ProbonoBonobo commented on Nov 21, 2019

nicolaskruchten commented on Jan 31, 2020

schaefer01 commented on Feb 24, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions