Repository navigation
Unpredictable escape sequences #5
Description
The line here in the CQLLexer allows certain characters from being escaped:
The CQL spec says:
Double quotes enclosing a sequence of any characters except double quote (unless preceded by backslash ()). Backslash escapes the character following it. The resultant value includes all backslash characters except those releasing a double quote (this allows other systems to interpret the backslash character). The surrounding double quotes are not included.
I'm curious why those specific characters are included and how those characters mean that they release a double quote.
It is actually making it impossible for me to represent a Windows-style file path, for example:
file = some\\path\\in\\windows\.exe
This gets turned into:
file = some\\path\\in\\windows.exe
because the . character is not on that list.
I happen to have a regular expression interpreter following the CQL parse that needs the value to be file = some\\path\\in\\windows\.exe. If I know that the CQL parser is consistently escaping the following character, I could just pass in: file=some\\\\path\\\\in\\\\windows\\.exe or if it consistently ignores it: file=some\\path\\in\\windows\.exe.
Activity
anacronw commentedon Aug 4, 2016
I found the rationale behind it, its the default masking behavior for relation modifiers:
http://www.loc.gov/standards/sru/cql/contextSets/theCqlContextSet.html#relmods
According to this, I should specify my query as:
Despite this, I don't see how it is useful to actually escape the masking characters within the parser. By consuming the symbols, the parser is basically saying that:
index = "my \* pony"is the same asindex = "my * pony"when it is semantically untrue. The interpretation of the original query should retain the fact that the former means the literal asterisk.I still believe that the line should be removed. I will submit a PR if requested.
adamdickmeiss commentedon Aug 9, 2016
Backslash has both meaning in CQL and regular expressions and that's part of the problem. Your initial CQL term is invalid "." has undefined semantics.. Instead use
anacronw commentedon Aug 9, 2016
I don't think the expression you provided means the same thing. I think need the expression to be in "unmasked" mode effectively and pass the string as-is.
However, what I don't understand is why this parser is interpreting the masking characters in the first place. Can you elaborate on that? I believe that should be the function of whatever is using the parser.
If the cql-java parser is interpreting the masking characters, then its effectively ignoring the difference between using a literal asterisk and the masked version of the asterisk (for example). After tokenization - any users of this library would receive the same output.
adamdickmeiss commentedon Aug 10, 2016
*?^\\. If followed by double-quotes it terminates. In all other cases it's eaten. Hence for all those characters that DO have meaning in CQL it is preserved and a user can look at the resulting term and judge to whether masked/anchored etc.. If backslash is followed by other character then (including the period.) it is removed. The spec even calls that INVALID. This is why you should use double-backslash plus dot for your regular expression!The CQL parser of YAZ is different in this respect - In that it always preserves them. I like that better, and you do too it seems. It certainly makes it possible to support the unmasked option (that is the lexer will NOT care about a mode or similar and things are preserved).,
indexdata/yaz@8521c0a
In the commit above you'll see that regression results 06/03.xcql and 06/06.xcql was changed .. Until that point they were identical.
anacronw commentedon Aug 10, 2016
Ah you're right, it does say its an error in my case.
I still disagree with one point:They have meanings in CQL in masked mode of course, I agree. However, why does that mean you must eat it? As I said before, as an example, by eating the backslash before the character, you lose the difference between a literal asterisk and the asterisk as a masked character.I would say, that by eating the backslash in that case, the user CANNOT judge whether its masked or not because whether I feed it to cql-java escaped or not, the lexer outputs the same thing.Edit: Actually I lost context of the current behavior and confused myself. I understand the point now - the fact that its eating all non-masked characters after the backslash is almost a preference since its undefined by the spec (or invalid rather). In the yaz project you mentioned, it does one behavior and here, it does another. Both are are technically valid, but I do prefer retaining the backslash in my case.
Thanks for working w/ me on this
adamdickmeiss commentedon Aug 11, 2016
First revisions of CQL only had the "masked" mode.. Now that we have unmasked / regular etc.. I think it's time to update the spec.. Of course for regular expressions we can't have anything being lost. (or any other masking rule for that sake).. So for that reason, IMHO, the lexer should never remove anything.
I tried to update-cql java a bit.. It's not a huge change.. a few tests needs update here and there.. And clearly, people could not have relied on the backslash-non-mask-char behavior because that was undefined or disallowed.
MikeTaylor commentedon Aug 17, 2016
Sorry, I am late to this party.
Let's be clear on what's what here. Your problem is one of lexing strings -- something that the CQL parser does. That is a completely separate issue from what the spec calls "masking" (i.e. pattern matching) which is done by the search engine. So using a relation modifier, as in
=/unmaskedmakes no difference at all to how the string is lexed: it's just an instruction to be passed to the search engine, so it knows how to interpret the query term.The only legitimate use for backslashes in a CQL string is to prevent a double-quote from terminating a string, as in
"I said \"Hello\".". In every other case, the backslash should be a literal, left for the search engine to interpret. This is what the specification is trying to say, albeit rather clumsily.The fact that CQL-Java tries to be "helpful" by special-casing the masking characters is very _un_helpful, and only causes confusion. (BTW., that was almost certainly my own code originally, so I can be as rude about it as I like :-) )
My conclusion: the fix for this bug is to remove the special-casing in the CQL-Java lexer.
anacronw commentedon Aug 17, 2016
I can attempt a pull request. I have it working locally, except I don't understand the rationale behind the regression tests (that ensured its behavior) so I hadn't submitted it yet.
psmagin commentedon Oct 8, 2025
Hi @adamdickmeiss, @MikeTaylor, I believe I’ve encountered the same issue. When parsing the query:
it gets converted to:
From this discussion, I understand this is more of a bug than a feature. Are there any plans to address this?
Retain escape sequences in lexer, fixes #5
Proper fixup for " when generating CQL query
Proper fixup for " when generating CQL query (#26)