Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,13 @@ snowflake().one_statement_parses_to(
```

## Analyzing Parsing Issues

For Snowflake `PUT`/`GET`, keep opaque location lexing confined to the two operands
at actual statement start. Preserve ordinary comments and identifiers/functions
named `PUT`/`GET`. Keep stage `ObjectName` quoting separate from the exact payload
prefix, avoid table-relation visitor annotations for named stages, and retain
bindings as normal expression placeholders. The parser must not resolve paths,
perform storage IO, evaluate options, or enforce an engine's transfer policy.
You can try to simplify the SQL statement to identify the root cause of the parsing issue. This may involve removing certain clauses or components of the SQL statement to see if it can be parsed successfully. Additionally, you can compare the problematic SQL statement with similar statements that are parsed correctly to identify any differences that may be causing the issue.

## Pre Commit Checks
Expand Down
37 changes: 36 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,42 @@ This crate avoids semantic analysis because it varies drastically
between dialects and implementations. If you want to do semantic
analysis, feel free to use this project as a base.

## Preserves Syntax Round Trip
## Snowflake transfer operands

`SnowflakeDialect` represents `PUT` and `GET` with dedicated statement nodes and
typed local URI, stage, and binding operands. A named stage retains its real
`ObjectName` identifiers and a separate exact payload prefix. User and table
stages remain distinct AST forms. Stage operands are not annotated as table
relations for generic relation discovery.

Only the first two operands after a statement-start transfer keyword use opaque
location tokenization. This keeps `file://`, `/*` globs, `--` filenames, `?`, and
`#` intact without changing ordinary SQL comments or identifiers/functions named
`PUT` and `GET`. Single-quoted transfer operands retain literal backslashes and
use doubled quotes for a literal quote; ordinary SQL string escape rules outside
these operands are unchanged. Whole-stage single quotes are a local grammar form,
not independent Snowflake reference evidence. Double-quoted local URIs and
`IDENTIFIER(...)` transfer operands are outside this grammar.

Options retain their original order, duplicates, and typed `Expr` values, including
signed numeric spelling. Optional numeric spelling metadata retains forms such as
`001` even when the `bigdecimal` feature normalizes the typed number; rendering
ignores stale spelling after a visitor changes that numeric value. Location
bindings use normal placeholder expressions so
expression visitors can apply the owning engine's existing binding accounting.
`dialect::snowflake::parse_file_transfer_stage_location` converts an exact bound
stage value into a typed target without manufacturing a SQL statement.

The parser performs no storage IO, option evaluation, range checking, async/batch
admission, or transfer execution. The execution layer must validate primitive
option shapes and supported stage kinds before planning/evaluating expressions
or submitting work. Accepting syntax does not claim a transfer transport or a
complete Snowflake workflow. In particular, path traversal/backslash admission,
`PARALLEL` range errors, duplicate/unknown options, and unsupported stage kinds
remain semantic responsibilities. These additions require owning qualification;
source tests alone do not establish runtime or reference parity.

## Preserves Syntax Round Trip

This crate allows users to recover the original SQL text (with comments removed,
normalized whitespace and keyword capitalization), which is useful for tools
Expand Down
229 changes: 227 additions & 2 deletions src/ast/helpers/stmt_data_loading.rs
Original file line number Diff line number Diff line change
Expand Up @@ -20,17 +20,242 @@
//! See [this page](https://docs.snowflake.com/en/sql-reference/commands-data-loading) for more details.

#[cfg(not(feature = "std"))]
use alloc::string::String;
use alloc::{format, string::String, vec::Vec};
use core::fmt;

#[cfg(feature = "serde")]
use serde::{Deserialize, Serialize};

use crate::ast::helpers::key_value_options::KeyValueOptions;
use crate::ast::{Ident, ObjectName, SelectItem};
use crate::ast::{Expr, Ident, ObjectName, SelectItem, UnaryOperator, Value};
#[cfg(feature = "visitor")]
use sqlparser_derive::{Visit, VisitMut};

/// Snowflake `PUT` upload syntax. Execution and transfer admission are separate concerns.
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Hash)]
#[cfg_attr(feature = "serde", derive(Serialize, Deserialize))]
#[cfg_attr(feature = "visitor", derive(Visit, VisitMut))]
pub struct Put {
/// Local source URI or a binding expression.
pub source: FileTransferLocation,
/// Stage destination or a binding expression.
pub target: FileTransferLocation,
/// Ordered options, including duplicates for owning semantic validation.
pub options: Vec<FileTransferOption>,
}

/// Snowflake `GET` download syntax.
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Hash)]
#[cfg_attr(feature = "serde", derive(Serialize, Deserialize))]
#[cfg_attr(feature = "visitor", derive(Visit, VisitMut))]
pub struct Get {
/// Stage source or a binding expression.
pub source: FileTransferLocation,
/// Local destination URI or a binding expression.
pub target: FileTransferLocation,
/// Ordered options, including duplicates for owning semantic validation.
pub options: Vec<FileTransferOption>,
}

/// A transfer operand, distinct from a table relation and from arbitrary SQL text.
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Hash)]
#[cfg_attr(feature = "serde", derive(Serialize, Deserialize))]
#[cfg_attr(feature = "visitor", derive(Visit, VisitMut))]
pub enum FileTransferLocation {
/// A local URI, preserving the `file://` prefix and path bytes.
Local {
/// Exact URI; the parser does not resolve or normalize a filesystem path.
uri: String,
/// Whether the complete operand was enclosed in single quotes.
quoted: bool,
},
/// A typed stage reference and separate optional payload prefix.
Stage(FileTransferStageLocation),
/// A native SQL placeholder. Visitors must consume it through normal expression binding.
Binding(Expr),
}

/// A stage reference used by transfer syntax, without table-relation visitor annotations.
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Hash)]
#[cfg_attr(feature = "serde", derive(Serialize, Deserialize))]
#[cfg_attr(feature = "visitor", derive(Visit, VisitMut))]
pub struct FileTransferStageLocation {
/// Named, user, or table stage syntax. Parsing does not imply execution support.
pub kind: FileTransferStage,
/// Exact bytes after the first slash outside quoted identifiers; `Some("")` retains `/`.
pub prefix: Option<String>,
/// Whether the complete operand was enclosed in single quotes.
pub quoted: bool,
}

/// Stage kinds must remain distinct so an execution layer can explicitly refuse unsupported ones.
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Hash)]
#[cfg_attr(feature = "serde", derive(Serialize, Deserialize))]
#[cfg_attr(feature = "visitor", derive(Visit, VisitMut))]
pub enum FileTransferStage {
/// A real qualified object name, retaining every identifier's quoting.
Named(ObjectName),
/// The current user's `@~` stage.
User,
/// A table's `@[namespace.]%table` stage, retaining its qualified name.
Table(ObjectName),
}

/// One transfer option. Values use normal expression/value nodes, not enum-like placeholders.
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Hash)]
#[cfg_attr(feature = "serde", derive(Serialize, Deserialize))]
#[cfg_attr(feature = "visitor", derive(Visit, VisitMut))]
pub struct FileTransferOption {
/// Original option name, including its identifier quoting.
pub name: Ident,
/// Typed value traversed by normal expression binding.
pub value: Expr,
/// Original primitive numeric spelling when the numeric AST feature normalizes it.
#[cfg_attr(
feature = "serde",
serde(default, skip_serializing_if = "Option::is_none")
)]
pub number_spelling: Option<String>,
}

pub(crate) fn file_transfer_number_matches(value: &Expr, raw: &str) -> bool {
let (value, digits) = match value {
Expr::UnaryOp {
op: UnaryOperator::Plus,
expr,
} => {
let Some(digits) = raw.strip_prefix('+') else {
return false;
};
(expr.as_ref(), digits)
}
Expr::UnaryOp {
op: UnaryOperator::Minus,
expr,
} => {
let Some(digits) = raw.strip_prefix('-') else {
return false;
};
(expr.as_ref(), digits)
}
_ if raw.starts_with(['+', '-']) => return false,
_ => (value, raw),
};
let Expr::Value(value) = value else {
return false;
};
let Ok(number) = digits.parse() else {
return false;
};
value.value == Value::Number(number, false)
}

impl fmt::Display for FileTransferStage {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
match self {
Self::Named(name) => write!(f, "{name}"),
Self::User => f.write_str("~"),
Self::Table(name) => {
for (index, part) in name.0.iter().enumerate() {
if index > 0 {
f.write_str(".")?;
}
if index + 1 == name.0.len() {
f.write_str("%")?;
}
write!(f, "{part}")?;
}
Ok(())
}
}
}
}

impl fmt::Display for FileTransferStageLocation {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
let mut value = format!("@{}", self.kind);
if let Some(prefix) = &self.prefix {
value.push('/');
value.push_str(prefix);
}
display_transfer_location(f, &value, self.quoted)
}
}

fn display_transfer_location(f: &mut fmt::Formatter, value: &str, quoted: bool) -> fmt::Result {
if quoted || file_transfer_location_needs_quotes(value) {
// Transfer paths keep backslashes literal; only the enclosing SQL quote is escaped.
write!(f, "'{}'", value.replace('\'', "''"))
} else {
f.write_str(value)
}
}

pub(crate) fn file_transfer_location_needs_quotes(value: &str) -> bool {
let mut stage_name = value.starts_with('@');
let mut identifier_quote = false;
let mut chars = value.chars().peekable();
while let Some(ch) = chars.next() {
if stage_name && ch == '"' {
if identifier_quote && chars.peek() == Some(&'"') {
chars.next();
} else {
identifier_quote = !identifier_quote;
}
} else if stage_name && ch == '/' && !identifier_quote {
stage_name = false;
} else if !identifier_quote && (ch.is_whitespace() || ch == ';') {
return true;
}
}
false
}

impl fmt::Display for FileTransferLocation {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
match self {
Self::Local { uri, quoted } => display_transfer_location(f, uri, *quoted),
Self::Stage(stage) => write!(f, "{stage}"),
Self::Binding(expr) => write!(f, "{expr}"),
}
}
}

impl fmt::Display for FileTransferOption {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
write!(f, "{}=", self.name)?;
if let Some(raw) = self
.number_spelling
.as_deref()
.filter(|raw| file_transfer_number_matches(&self.value, raw))
{
f.write_str(raw)
} else {
write!(f, "{}", self.value)
}
}
}

impl fmt::Display for Put {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
write!(f, "PUT {} {}", self.source, self.target)?;
for option in &self.options {
write!(f, " {option}")?;
}
Ok(())
}
}

impl fmt::Display for Get {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
write!(f, "GET {} {}", self.source, self.target)?;
for option in &self.options {
write!(f, " {option}")?;
}
Ok(())
}
}

#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Hash)]
#[cfg_attr(feature = "serde", derive(Serialize, Deserialize))]
#[cfg_attr(feature = "visitor", derive(Visit, VisitMut))]
Expand Down
9 changes: 8 additions & 1 deletion src/ast/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,8 @@ use alloc::{
use helpers::{
attached_token::AttachedToken,
stmt_data_loading::{
AlterFileFormatOperation, AlterStageOperation, FileStagingCommand, StageLoadSelectItemKind,
AlterFileFormatOperation, AlterStageOperation, FileStagingCommand, Get, Put,
StageLoadSelectItemKind,
},
};

Expand Down Expand Up @@ -5022,6 +5023,10 @@ pub enum Statement {
/// Snowflake `REMOVE`
/// See: <https://docs.snowflake.com/en/sql-reference/sql/remove>
Remove(FileStagingCommand),
/// Snowflake `PUT`: prepare an upload from a local URI to a stage.
Put(Put),
/// Snowflake `GET`: prepare a download from a stage to a local URI.
Get(Get),
/// RaiseError (MSSQL)
/// RAISERROR ( { msg_id | msg_str | @local_variable }
/// { , severity , state }
Expand Down Expand Up @@ -6723,6 +6728,8 @@ impl fmt::Display for Statement {
Statement::Return(r) => write!(f, "{r}"),
Statement::List(command) => write!(f, "LIST {command}"),
Statement::Remove(command) => write!(f, "REMOVE {command}"),
Statement::Put(command) => write!(f, "{command}"),
Statement::Get(command) => write!(f, "{command}"),
Statement::ExportData(e) => write!(f, "{e}"),
Statement::CreateUser(s) => write!(f, "{s}"),
Statement::AlterSchema(s) => write!(f, "{s}"),
Expand Down
5 changes: 4 additions & 1 deletion src/ast/spans.rs
Original file line number Diff line number Diff line change
Expand Up @@ -520,7 +520,10 @@ impl Spanned for Statement {
Statement::Print { .. } => Span::empty(),
Statement::WaitFor(_) => Span::empty(),
Statement::Return { .. } => Span::empty(),
Statement::List(..) | Statement::Remove(..) => Span::empty(),
Statement::List(..)
| Statement::Remove(..)
| Statement::Put(..)
| Statement::Get(..) => Span::empty(),
Statement::ExportData(ExportData {
options,
query,
Expand Down
10 changes: 9 additions & 1 deletion src/dialect/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -51,8 +51,8 @@ pub use self::mysql::MySqlDialect;
pub use self::oracle::OracleDialect;
pub use self::postgresql::PostgreSqlDialect;
pub use self::redshift::RedshiftSqlDialect;
pub use self::snowflake::parse_snowflake_stage_name;
pub use self::snowflake::SnowflakeDialect;
pub use self::snowflake::{parse_file_transfer_stage_location, parse_snowflake_stage_name};
pub use self::spark::SparkSqlDialect;
pub use self::sqlite::SQLiteDialect;
pub use self::teradata::TeradataDialect;
Expand Down Expand Up @@ -1273,6 +1273,14 @@ pub trait Dialect: Debug + Any {
false
}

/// Supports `PUT`/`GET` with two opaque transfer operands at statement start.
///
/// URI comment markers, globs, and stage prefixes are payload bytes only in those
/// operands. Normal SQL identifiers and comments retain their ordinary tokenization.
fn supports_file_transfer_commands(&self) -> bool {
false
}

/// Returns true if this dialect supports the `COMMENT` statement
fn supports_comment_on(&self) -> bool {
false
Expand Down
Loading
Loading