Repository navigation
Description
Context
Currently, datafusion-sqlparser-rs strips out all inline comments (-- … and /* … */). There is existing support for COMMENT ON … statements, and it would be good to have support for inline comments as well, given they often used for documenting sql. However, these comments are currently lost while parsing.
There is an open issue on lossless AST, but it has been open for about 5 years. Moreover this topic is referenced in this PR. So there is a precedent to move forward with inline comment support.
Objective/Motivation
Capturing inline comments in the AST would be helpful for:
- Documentation Generation: This is the primary purpose of this issue, to extend the existing ability to use
COMMENT ONsupport, for downstream documentation generation based on the parsed AST. - Linting Coverage: Going along with 1, being able to track inline comments would be useful for checking for completeness of documentation, and especially helpful in places where documentation in the form of comments is necessary as inline comments and not
COMMENT ONcomments.
Example of proposed usage:
Inline Comment Associated with a table:
-- Table for storing values for users
CREATE TABLE IF NOT EXISTS users (
id BIGINT PRIMARY KEY
-- Stores the username as text, required value for creating a user account.
name TEXT NOT NULL
);Expectation: The AST node for the users table would include a comment: Option<String> that would then be populated with the comment located on the previous line: Table for storing values for users. Additionally the column name would have an associated comment: Option<String> that would contain Stores the username as text, required value for creating a user account.. Therefore all tables and columns would carry a comment: Option<String> derived from the trailing -- based comment preceding the field.
NOTE: For now the assumption is parsing the comments immediately preceding the line the field is on, though comments can be arranged as such as well:
CREATE TABLE IF NOT EXISTS users ( -- Table for storing values for users
id BIGINT PRIMARY KEY
name TEXT NOT NULL -- Stores the username as text, required value for creating a user account.
);It might prove fruitful to allow parsing both types of comments, and perhaps specifying when parsing which are relevant and which are not.
Inline Comments out of Scope
-- migration script v2025-01
CREATE TABLE IF NOT EXISTS users (
id BIGINT PRIMARY KEY
-- Stores the username as text, required value for creating a user account.
name TEXT NOT NULL
);In the above example there is a line separating the comment and the CREATE TABLE line, so this comment should be disregarded and considered out of scope for comment: Option<String> value.
Proposed Behavior
Add an optional parse mode: Parser::parse_sql_with_comments or Parser::parse_sql_with_inline_comments that retains the inline comments and associates them with nearby fields/schema objects, within the AST.
For CreateTable and ColumnDef add a comment: Option<String> that will parse and store the associated comment based on the -- (and potentially multi line comment /* */) that immediately precedes the comment, and/or possibly when the comment is on the same line as shown above.
Activity
[-]Inline Comment Support for AST[/-][+]Limited Inline Comment Support in AST[/+]RPG-Alex commentedon Oct 14, 2025
After looking around I have found this:
So not only does the tokenizer support comments it tracks the comment content!
I have then gone to the parser mod, where the
Tokenizer::newis used, and after adding aprintln!on a small sql string to parse, I can see:(ellipses for omitted output for brevity)
For clarity the query I used:
So it seems that the single line comment is being successfully tokenized! I may try to implement support for this depending on the difficulty around integrating with the parser.
RPG-Alex commentedon Oct 17, 2025
Connecting to PR #2069
xitep commentedon Nov 15, 2025
maybe a simpler and more general purpose idea for the parser to deal with comments could be the following one (feedback is highly be appreciated):
since comments can appear almost everywhere within a statement (or parsed script), either 1) the AST would have to provide a "before-/after-comment" for all tokens/nodes in the AST or 2) the parser would make up specific rules how to associated a single comment with a nearby token (this, i understood, is what is suggested as part of this issue.)
the former approach would mean very convenient access to the comments right away from the tokens themselves, but it would grow the AST (assuming that most tokens would not be associated with comments, it could be considered wasteful.) on the other, the parser making up its own mind about how to interpret and associate comments, it could be restrictive for downstream clients.
the other day, i wrote a project specific (java) code generator to process SQL like this ...
... essentially associating metadata with particular tokens in the SQL query in a format of my choosing (for later code generation.)
it would be fully sufficient for my use-case(s) to have
sqlparser-rsprovide aParser::parse_sql_with_commentsmethod returning the AST as is, plus a separate, independent structure representing the extracted comments, e.g.with
Comments(and associated) having the following API (pseudo-code):I see the following advantages with this concept:
Vecwould in fact end-up with a vec that can efficiently be searched with binary search by a given span's start/end.What do you think? (@iffyio i would appreciate your thoughts on this 🙏)
update: one difficulty with the proposal is, however, that some tokens are omitted from the AST entirely, namely punctuation, making it difficult for a client to tell a comment's semantic position. an example would be:
:/
iffyio commentedon Nov 22, 2025
This approach sounds reasonable to me!
@xitep do you mean that in the example the comments
/* one */and/* two */aren't part of the tokenizer output to begin with? Offhand i would guess that doesn't have to be a special case in the tokenizer codexitep commentedon Nov 22, 2025
ah, i'm sorry for being misleading :-/ i'll try to explain in more details:
in my proof-of-concept program i do have a
fn find_between(&self, start: Location, end: Location) -> impl Iterator<Item = &CommentWithSpan>method (i just added it to the proposal above) allowing me to ask for comments between two locations. for example between the end location of one statement (ie. the statement's span's end) and the beginning of the next one (the statement's span's start.)this works nicely since those locations (statements' spans) are being part of the AST (or are going to be there in soon future.) but some tokens are simply not part of the AST (and will likely never be), in particular a comma (but also many others, eg. keywords.)
so, in the
select a /* one */ , /* two */ b from dualexample, i would fail to associate/* one */toa(and only toa) and/or/* two */tob(and only tob) without knowing that there was a comma between the two comments. Unfortunately, the AST does not contain the location of the comma (and/or many more keywords.) Another example would be:SELECT ... GROUP /* here i am */ BY ...Unfortunately, i can't think of a good solution at the moment. (The litmus test being the reproduction of the original SQL's structure including comments.)
iffyio commentedon Nov 25, 2025
I see, thanks for clarifying! Yeah same, I can't think of a way for the parser to disambiguate such a scenario. But I think that could be fine that the decision be left up to the caller - in that since
find_between('a', 'b')would be returning both/* one */and/* two */, the caller might have a preference on the more sensible strategy to use to associate the comments across both tokensxitep commentedon Nov 25, 2025
ok, thanks for your feedback @iffyio. i'll prepare a PR for that. even with the shortcuts we discussed, having
sqlparser-rspublish the comments still seems a better idea to me than leaving clients completely without or with writing their own parser (essentially processing the sql source once again.)btw: i'm more inclined to publish the comments' content as
&str(as opposed to an ownedString) with the lifetime bound to the parsed sql string. would you agree before i set off with the work?iffyio commentedon Nov 25, 2025
re representing the comment as
&str, hmm that would be ideal but I'm not sure that wouldn't be quite invasive as things stand - since the tokenizer already produces the comments as ownedStrings - there's ongoing work here to do this end to end for the parser so maybe the comment handling can build on top of that work later on insteadxitep commentedon Nov 25, 2025
alright, many thanks for the hint, @iffyio ! that sounds to me like at this point i should go with (owned)
Strings. Eventually it'd be reworked to a more efficient presentation. I believe it would be simply more consistent with the current crate API.(btw: in my proof-of-concept, i keep comments internally as
Comment<Range<u32>>with the range representing the start and end offsets in the parsed SQL. only when handing them out to the user i turn them into aComment<&str>... hence, the API proposal above.)xitep commentedon Jan 13, 2026
hello, i would like to share an experiment i've been working on (it's merely an educational effort for me to learn more about parsing and rust in general); i've tried coding the idea outlined by @RPG-Alex above with a "toy crate" and it seems to work. this example demonstrates the use-case.
the crucial point is that
orqlexposes every token in its AST. but a fact is, thatorqlis merely a "toy." pushing it as far assqlparser(limited to oracle sql), would be a tons of work that i don't think is feasible for me. the question that i face now is whether i should invest my time intoorqlorsqlparser. would the concept of exposing every token in the AST be OK withsqlparser? (i'd be very grateful for guidance / advice.)