Skip to content

DataFusion Configuration Consolidation #4349

Closed

Description

@alamb

Related

apache/datafusion-ballista#479
#3885

TLDR Recommendations

This is a complicated issue and I don't have a magic answer. However I have some concrete suggestions

Some suggested steps:

I think consolidating SessionConfig/Config options is likely to be the most controversial / cause the most chrun but it will provide immense benefits I think (like runtime visibility into the current settings)

Then we can further improve from there

Introduction

"Configuration" in DataFusion has a few usecases:

There are also two overlapping "levels" of configuration that are needed

  • Session level (e.g. that can be reused from one query execution to the next)
  • Statement/Task level (e.g. that is needed to plan a query and doesn't change for the duration of a statement such as "the value of now()" and target batch sizes, etc). Statement level configuration is typically a superset of the session level configuration

Current state of configuration in DataFusion

The current state is .... inconsistent to put it mildly.

The core structure is SessionContext which is the final glue and entry point to interacting with datafusion (e.g. tables provided, etc).

Within the SessionContext there is the some combination of SessionState, SessionConfig, ConfigOptions. Part of the hierarchy is like this:

SessionContext
 -- Has a SessionState
 -- SessionStartTime
 -- SessionConfig
   -- ConfigOptions

SessionConfig is effectively the Session level configuration I describe above.

TaskContext is the statement level (aka per task / per query) level context. If you look hard you can see has a copy of the SessionConfig (buried in TaskPropertoes) or also maybe is backed by KVPairs.

Desire

I would like to have a clear configuration system that cleanly separates the statement level config from the task level config and allows configuration values to be set in a uniform manner and that are easy to view programmatically

Activity

  1. alamb commented on Nov 23, 2022

    @alamb
    ContributorAuthor

    However, one question, raised by @avantgardnerio in apache/datafusion-ballista#479, is how to structure the interactions with the configuration code.

    It turns out that SessionConfig has a more rust struct style and ConfigOptions is a more dynamic hashmap style. I honestly don't have a preference as long as the introspection (with docstrings) features of ConfigOptions are preserved. Maybe this is possible via macro magic

  2. alamb commented on Nov 23, 2022

    @alamb
    ContributorAuthor

    🤔 there is also ExecutionProps which is some subset of the TaskProperties 🤔

  3. mingmwang commented on Nov 28, 2022

    @mingmwang
    Contributor

    I think consolidating SessionConfig/Config options is likely to be the most controversial / cause the most chrun but it will provide immense benefits I think (like runtime visibility into the current settings)

    I think consolidating SessionConfig/ConfigOptions is definitely the right direction. The debate should be how to consolidate them, either keep SessionConfig or keep ConfigOptions. I do not have a clear preference on this either. maybe we can have a vote.

  4. mingmwang commented on Nov 28, 2022

    @mingmwang
    Contributor

    🤔 there is also ExecutionProps which is some subset of the TaskProperties 🤔

    We should consolidate ExecutionProps and TaskProperties also.

  5. mingmwang commented on Nov 28, 2022

    @mingmwang
    Contributor

    I'm OK with the change in this PR. Ballista should still work after this PR.

    #4382

  6. alamb commented on Nov 29, 2022

    @alamb
    ContributorAuthor

    Here is my next contribution to clean up configuration: #4427 (slowly consolidating the configurations)

  7. kamilkonior commented on Dec 14, 2022

    @kamilkonior
    Contributor

    If a TaskContext represents a query/task, then could TableProvider receive in scan method TaskContext as replacement for SessionState ? I would like to have a way to identify task in TableProvider scan fn. If I am not wrong, the only way right now to identify task in TableProvider is to create new SessionContext for each query and use session_id from SessionState.

  8. tustvold commented on Dec 14, 2022

    @tustvold
    Contributor

    If a TaskContext represents a query/task

    I think it might be more precise to describe TaskContext as the state for query execution, with SessionState used for query planning.

    the only way right now to identify task in TableProvider

    Correct, at the point in planning where TableProvider::scan is called, the task conceptually doesn't exist yet

    create new SessionContext for each query

    FWIW this is what IOx does

    Edit: I wrote some docs in #2655 last time I got thoroughly confused by this system that I think are still in date. I see we have grown a few more config structs since then though

    Edit Edit: It may also be the case that a single query is broken into multiple TaskContext - I think Ballista does this, but I'm not sure.

  9. kamilkonior commented on Dec 14, 2022

    @kamilkonior
    Contributor

    Thank you @tustvold for clarification, in such case I will follow SessionContext per query approach.

  10. alamb commented on Jul 25, 2023

    @alamb
    ContributorAuthor

    I think this is largely complete and we don't have any additional work planned here, so closing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions