Skip to content

Limited Inline Comment Support in AST #2065

Closed

Description

@RPG-Alex

Context

Currently, datafusion-sqlparser-rs strips out all inline comments (-- … and /* … */). There is existing support for COMMENT ON … statements, and it would be good to have support for inline comments as well, given they often used for documenting sql. However, these comments are currently lost while parsing.

There is an open issue on lossless AST, but it has been open for about 5 years. Moreover this topic is referenced in this PR. So there is a precedent to move forward with inline comment support.

Objective/Motivation

Capturing inline comments in the AST would be helpful for:

  1. Documentation Generation: This is the primary purpose of this issue, to extend the existing ability to use COMMENT ON support, for downstream documentation generation based on the parsed AST.
  2. Linting Coverage: Going along with 1, being able to track inline comments would be useful for checking for completeness of documentation, and especially helpful in places where documentation in the form of comments is necessary as inline comments and not COMMENT ON comments.

Example of proposed usage:

Inline Comment Associated with a table:
-- Table for storing values for users
CREATE TABLE IF NOT EXISTS users (
  id BIGINT PRIMARY KEY
  -- Stores the username as text, required value for creating a user account.
  name TEXT NOT NULL
);

Expectation: The AST node for the users table would include a comment: Option<String> that would then be populated with the comment located on the previous line: Table for storing values for users. Additionally the column name would have an associated comment: Option<String> that would contain Stores the username as text, required value for creating a user account.. Therefore all tables and columns would carry a comment: Option<String> derived from the trailing -- based comment preceding the field.

NOTE: For now the assumption is parsing the comments immediately preceding the line the field is on, though comments can be arranged as such as well:

CREATE TABLE IF NOT EXISTS users ( -- Table for storing values for users
  id BIGINT PRIMARY KEY
  name TEXT NOT NULL  -- Stores the username as text, required value for creating a user account.
);

It might prove fruitful to allow parsing both types of comments, and perhaps specifying when parsing which are relevant and which are not.

Inline Comments out of Scope
-- migration script v2025-01

CREATE TABLE IF NOT EXISTS users (
  id BIGINT PRIMARY KEY
  -- Stores the username as text, required value for creating a user account.
  name TEXT NOT NULL
);

In the above example there is a line separating the comment and the CREATE TABLE line, so this comment should be disregarded and considered out of scope for comment: Option<String> value.

Proposed Behavior

Add an optional parse mode: Parser::parse_sql_with_comments or Parser::parse_sql_with_inline_comments that retains the inline comments and associates them with nearby fields/schema objects, within the AST.

For CreateTable and ColumnDef add a comment: Option<String> that will parse and store the associated comment based on the -- (and potentially multi line comment /* */) that immediately precedes the comment, and/or possibly when the comment is on the same line as shown above.

Activity

  1. changed the title [-]Inline Comment Support for AST[/-] [+]Limited Inline Comment Support in AST[/+] on Oct 9, 2025
  2. RPG-Alex commented on Oct 14, 2025

    @RPG-Alex
    ContributorAuthor

    After looking around I have found this:

    pub enum Whitespace {
        Space,
        Newline,
        Tab,
        SingleLineComment { comment: String, prefix: String },
        MultiLineComment(String),
    }

    So not only does the tokenizer support comments it tracks the comment content!

    I have then gone to the parser mod, where the Tokenizer::new is used, and after adding a println! on a small sql string to parse, I can see:

    Tokens found: ... TokenWithSpan { token: Whitespace(SingleLineComment { comment: " this is a comment\n", prefix: "--" }), span: Span(Location(2,1)..Location(3,1)) }, ... }
    

    (ellipses for omitted output for brevity)

    For clarity the query I used:

    -- this is a comment
    SELECT * FROM foo; SELECT * FROM bar;

    So it seems that the single line comment is being successfully tokenized! I may try to implement support for this depending on the difficulty around integrating with the parser.

  3. RPG-Alex commented on Oct 17, 2025

    @RPG-Alex
    ContributorAuthor

    Connecting to PR #2069

  4. xitep commented on Nov 15, 2025

    @xitep
    Contributor

    maybe a simpler and more general purpose idea for the parser to deal with comments could be the following one (feedback is highly be appreciated):

    since comments can appear almost everywhere within a statement (or parsed script), either 1) the AST would have to provide a "before-/after-comment" for all tokens/nodes in the AST or 2) the parser would make up specific rules how to associated a single comment with a nearby token (this, i understood, is what is suggested as part of this issue.)

    the former approach would mean very convenient access to the comments right away from the tokens themselves, but it would grow the AST (assuming that most tokens would not be associated with comments, it could be considered wasteful.) on the other, the parser making up its own mind about how to interpret and associate comments, it could be restrictive for downstream clients.

    the other day, i wrote a project specific (java) code generator to process SQL like this ...

    select id         /* long */,
           name       /* String */,
           value      /* BigDecimal */,
           defined_at /* LocalDate */,
           ...

    ... essentially associating metadata with particular tokens in the SQL query in a format of my choosing (for later code generation.)

    it would be fully sufficient for my use-case(s) to have sqlparser-rs provide a Parser::parse_sql_with_comments method returning the AST as is, plus a separate, independent structure representing the extracted comments, e.g.

     Parser::parse_sql_with_comments(..) -> Result<(Vec<Statement>, Comments), ..>)`

    with Comments (and associated) having the following API (pseudo-code):

    ... Comments {
      /// Possibly locates a SQL comment directly preceding the specified location
      fn find_before(&self, location: &Location) -> Option<&CommentWithSpan>;
    
      /// Possibly locates a SQL comment directly following the specified location
      fn find_after(&self, location: &Location) -> Option<&CommentWithSpan>;
    
        /// Finds comments located / starting at or after `start` but before `end`.
      fn find_between(&self, start: &Location, end: &Location) -> impl Iterator<Item = &CommentWithSpan>;
    }
    
    impl From<Comments> for Vec<CommentWithSpan> {
      fn into(value: Comments) { ... }
    }
    
    struct CommentWithSpan {
      span: Span,
      text: Comment,
    }
    
    enum Comment {
        SingleLine { content: String, prefix: String },
        MultiLine(String),
    }

    I see the following advantages with this concept:

    1. It does not require to extend existing AST nodes; since most of them will not be associated with comments at all this would save space
    2. It would also mean that parsing with and without comments would not be a big difference; when parsing without comments we'd merely discard the comments (status-quo) instead of accumulating.
    3. Since parsing and encountering comments is done from begin to end of the parsed SQL script (and comments not being overlapping), accumulating the comments (as encountering them while parsing) in a plain Vec would in fact end-up with a vec that can efficiently be searched with binary search by a given span's start/end.

    What do you think? (@iffyio i would appreciate your thoughts on this 🙏)

    update: one difficulty with the proposal is, however, that some tokens are omitted from the AST entirely, namely punctuation, making it difficult for a client to tell a comment's semantic position. an example would be:

    select a /* one */ , /* two */ b from dual;

    :/

  5. iffyio commented on Nov 22, 2025

    @iffyio
    Contributor
     Parser::parse_sql_with_comments(..) -> Result<(Vec<Statement>, Comments), ..>)`

    This approach sounds reasonable to me!

    that some tokens are omitted from the AST entirely, namely punctuation, making it difficult for a client to tell a comment's semantic position

    @xitep do you mean that in the example the comments /* one */ and /* two */ aren't part of the tokenizer output to begin with? Offhand i would guess that doesn't have to be a special case in the tokenizer code

  6. xitep commented on Nov 22, 2025

    @xitep
    Contributor

    ah, i'm sorry for being misleading :-/ i'll try to explain in more details:

    in my proof-of-concept program i do have a fn find_between(&self, start: Location, end: Location) -> impl Iterator<Item = &CommentWithSpan> method (i just added it to the proposal above) allowing me to ask for comments between two locations. for example between the end location of one statement (ie. the statement's span's end) and the beginning of the next one (the statement's span's start.)

    -- doc for statement one
    select * from dual;
    
    -- doc for statement two
    select * from dual;
    

    this works nicely since those locations (statements' spans) are being part of the AST (or are going to be there in soon future.) but some tokens are simply not part of the AST (and will likely never be), in particular a comma (but also many others, eg. keywords.)

    so, in the select a /* one */ , /* two */ b from dual example, i would fail to associate /* one */ to a (and only to a) and/or /* two */ to b (and only to b) without knowing that there was a comma between the two comments. Unfortunately, the AST does not contain the location of the comma (and/or many more keywords.) Another example would be: SELECT ... GROUP /* here i am */ BY ...

    Unfortunately, i can't think of a good solution at the moment. (The litmus test being the reproduction of the original SQL's structure including comments.)

  7. iffyio commented on Nov 25, 2025

    @iffyio
    Contributor

    I see, thanks for clarifying! Yeah same, I can't think of a way for the parser to disambiguate such a scenario. But I think that could be fine that the decision be left up to the caller - in that since find_between('a', 'b') would be returning both /* one */ and /* two */, the caller might have a preference on the more sensible strategy to use to associate the comments across both tokens

  8. xitep commented on Nov 25, 2025

    @xitep
    Contributor

    ok, thanks for your feedback @iffyio. i'll prepare a PR for that. even with the shortcuts we discussed, having sqlparser-rs publish the comments still seems a better idea to me than leaving clients completely without or with writing their own parser (essentially processing the sql source once again.)

    btw: i'm more inclined to publish the comments' content as &str (as opposed to an owned String) with the lifetime bound to the parsed sql string. would you agree before i set off with the work?

  9. iffyio commented on Nov 25, 2025

    @iffyio
    Contributor

    re representing the comment as &str, hmm that would be ideal but I'm not sure that wouldn't be quite invasive as things stand - since the tokenizer already produces the comments as owned Strings - there's ongoing work here to do this end to end for the parser so maybe the comment handling can build on top of that work later on instead

  10. xitep commented on Nov 25, 2025

    @xitep
    Contributor

    alright, many thanks for the hint, @iffyio ! that sounds to me like at this point i should go with (owned) Strings. Eventually it'd be reworked to a more efficient presentation. I believe it would be simply more consistent with the current crate API.

    (btw: in my proof-of-concept, i keep comments internally as Comment<Range<u32>> with the range representing the start and end offsets in the parsed SQL. only when handing them out to the user i turn them into a Comment<&str> ... hence, the API proposal above.)

  11. xitep commented on Jan 13, 2026

    @xitep
    Contributor

    hello, i would like to share an experiment i've been working on (it's merely an educational effort for me to learn more about parsing and rust in general); i've tried coding the idea outlined by @RPG-Alex above with a "toy crate" and it seems to work. this example demonstrates the use-case.

    the crucial point is that orql exposes every token in its AST. but a fact is, that orql is merely a "toy." pushing it as far as sqlparser (limited to oracle sql), would be a tons of work that i don't think is feasible for me. the question that i face now is whether i should invest my time into orql or sqlparser. would the concept of exposing every token in the AST be OK with sqlparser? (i'd be very grateful for guidance / advice.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions