Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Found problem with backslashes parsing #307

Closed
seregvan opened this issue Sep 18, 2019 · 26 comments
Closed

Found problem with backslashes parsing #307

seregvan opened this issue Sep 18, 2019 · 26 comments
Assignees
Labels

Comments

@seregvan
Copy link

seregvan commented Sep 18, 2019

You closed my previous issue as a question so I'm opening a new one.
I think I found the problem in your parsing. I can't see the reason to do this in string literal parsing except the case when I try to find " in the result.

if (next == '"')
{
     NextChar();
}

Because if I use query "contains(\"\\\")" it fails with unterminated literal, but if I change query to "contains(\"\\a\")" it works finding "\a" string. And also if I use "contains(\"\\\\\")" it works with string "\\" (string with two real slashes).

So I think the correct solution will be to remove this if statement and just replace \" to " in the result token if it was a string literal. In this case if user will want to put " in query string it should be escaped inside this string. At least it will not break any existing code.
The better approach will be (in my opinion) to escape all the characters in the string by user and to unescape by parser. That means \\\\ will evaluate to \ and not to \\.
I understand your workaround by using @0 but it's not the case when search string comes outside my app.

@alexweav
Copy link
Contributor

Hello @StefH, I just ran into this issue as well. The filter foo = "\"asdf\"" seems to match the literal including backslashes, and the filter foo = """asdf""", which is the literal format documented on the wiki, causes a parse failure.

Are there any workaround for parsing a dynamic linq lambda which contains a quote, other than substitutions? If not, would you please be able to point me towards this area in the code? I might be able to provide a PR, with some guidance.

@alexweav
Copy link
Contributor

alexweav commented Nov 19, 2019

@StefH I was able to work around this by running an ExpressionVisitor on the parsed lambda which simply calls Regex.Unescape() on all string constants. I wonder if something this simple would solve this issue? It ensures that all escape sequences inside string literals are parsed as escape sequences, not the literal characters. Do you think this is too drastic to apply everywhere?

The regex unescaping ensures that the expression Foo = "ab\"cd" returns true where Foo equals the string ab"cd. Without it, it will only return true where Foo equals ab\"cd with the backslash included.

@StefH
Copy link
Collaborator

StefH commented Nov 19, 2019

Please make a PullRequest with unit tests or a new sample project.

@alexweav
Copy link
Contributor

Test: #325

@StefH
Copy link
Collaborator

StefH commented Nov 25, 2019

@alexweav and @seregvan Can you please test if version System.Linq.Dynamic.Core.1.0.20-ci-12219.nupkg from MyGet solves your issue?

@StefH
Copy link
Collaborator

StefH commented Nov 25, 2019

@hypetsch and @SolidWhiteSven : you also where involved in string parsing issues, can you also verify if this fix is also in line with your usage?

@ghost
Copy link

ghost commented Nov 26, 2019

Hey, I've updated to 1.0.20-ci-12230 and all my tests were successful.
Thanks by the was for keeping this package nice and updated :-)

@StefH
Copy link
Collaborator

StefH commented Nov 26, 2019

@SolidWhiteSven The version you mention is a different one, you need System.Linq.Dynamic.Core.1.0.20-ci-12219.

@ghost
Copy link

ghost commented Nov 26, 2019

Ah, I see, thought the changes were also included there.
Retested with version 12219 and it all still works as expected.

@StefH StefH added the bug label Jan 10, 2020
@StefH StefH self-assigned this Jan 10, 2020
@StefH
Copy link
Collaborator

StefH commented Jan 10, 2020

Closing as solved in PR #326

@StefH StefH closed this as completed Jan 10, 2020
@StefH StefH reopened this Jan 10, 2020
@StefH
Copy link
Collaborator

StefH commented Jan 10, 2020

@alexweav --> Can you please test if version System.Linq.Dynamic.Core.1.0.20-ci-12219.nupkg from MyGet solves your issue?

@rppmartinsCollab
Copy link

rppmartinsCollab commented Jan 23, 2020

@alexweav --> Can you please test if version System.Linq.Dynamic.Core.1.0.20-ci-12219.nupkg from MyGet solves your issue?

Hi, testing this version, the problem persists.
When doing something like

query.Where("name.contains(" \ ")")

, I´ve got System.Linq.Dynamic.Core.Exceptions.ParseException: 'Syntax error ' \ ' '.

Can you help me with this?

@StefH
Copy link
Collaborator

StefH commented Feb 24, 2020

@rppmartinsCollab To be able to use Contains to check if a string contains a \, you need code like:

query.Where("name.contains(\"\\\")")
  1. Escape the " with \"
  2. Escape the \ with \\

Does this work for you?

@StefH
Copy link
Collaborator

StefH commented Feb 24, 2020

@alexweav, @seregvan and @hypetsch : If you have time, please double check if the updated NuGet (System.Linq.Dynamic.Core.1.0.20-ci-12219) works ok for you.

@rppmartinsCollab
Copy link

rppmartinsCollab commented Feb 24, 2020

@rppmartinsCollab To be able to use Contains to check if a string contains a \, you need code like:

query.Where("name.contains(\"\\\")")
  1. Escape the " with \"
  2. Escape the \ with \\

Does this work for you?

Hey, thanks for the response.

This way I get the error "Syntax error ' \\ ' " once TextParser does not recognize the character.

@StefH
Copy link
Collaborator

StefH commented Feb 24, 2020

@rppmartinsCollab Sorry for the confusion, you actually need to escape the \ twice. So \\\\.
For example see https://github.com/StefH/System.Linq.Dynamic.Core/pull/326/files#diff-d7a1783411cc2e7ca08b15984ed427c9R33

@rppmartinsCollab
Copy link

@rppmartinsCollab Sorry for the confusion, you actually need to escape the \ twice. So \\\\.
For example see https://github.com/StefH/System.Linq.Dynamic.Core/pull/326/files#diff-d7a1783411cc2e7ca08b15984ed427c9R33

The problem seems to be in scaping the quote character, I get error in \"

@silarmani
Copy link

silarmani commented Mar 18, 2020

Hello guys,

I've just upgraded it to 1.0.21 via nuget and I can confirm that the issue still exists.

It only breaks if the filter is a check for a single backslash

e.g. #1 if I search for \t ["Studyname.Contains(\"\\t\")"], EF produces the correct condition

exec sp_executesql N'SELECT [e.Studytype].[ParentID], [e.Studytype].[ChildID], [e.Studytype].[idx], [t].[SYSIDENT], [t].[tdid]
FROM [RV_riskstudytype_STUDIES] AS [e.Studytype]
INNER JOIN (
    SELECT [e0].[SYSIDENT], [e0].[tdid]
    FROM [RV_riskstudy] AS [e0]
    LEFT JOIN [RV_project_PROJECTSTUDIES] AS [jSTUDYPROJECT0] ON [e0].[tdid] = [jSTUDYPROJECT0].[ChildID]
    LEFT JOIN [RV_riskstudytype_STUDIES] AS [jSTUDYTYPE0] ON [e0].[tdid] = [jSTUDYTYPE0].[ChildID]
    WHERE ([e0].[rid] IN (''0'', @__registerUuid_0) AND [e0].[STUDY_STATUS] NOT IN (9, 3)) AND (CHARINDEX(N''\t'', [e0].[STUDYNAME]) > 0)
    ORDER BY [e0].[SYSIDENT], [e0].[tdid]
    OFFSET @__p_1 ROWS FETCH NEXT @__p_2 ROWS ONLY
) AS [t] ON [e.Studytype].[ChildID] = [t].[tdid]
ORDER BY [t].[SYSIDENT], [t].[tdid]',N'@__registerUuid_0 varchar(160),@__p_1 int,@__p_2 int',@__registerUuid_0='3.1.1.risk-register.da7ecd5e8ecc4c91ab3f5fa2a0686346',@__p_1=0,@__p_2=25

e.g. #2 if I search for \\ ["Studyname.Contains(\"\\\\\")"], EF produces the correct condition (i.e. text with single backslashes do not match)

exec sp_executesql N'SELECT [e.Studytype.RiskstudyType].[tdid], [e.Studytype.RiskstudyType].[CODE], [e.Studytype.RiskstudyType].[DESCRIPTION], [e.Studytype.RiskstudyType].[SYSIDENT], [e.Studytype.RiskstudyType].[rid], [e.Studytype.RiskstudyType].[STATUS], [e.Studytype.RiskstudyType].[STUDYTYPEAPPROACH], [e.Studytype.RiskstudyType].[STUDYTYPENAME], [e.Studytype.RiskstudyType].[STUDYTYPEOPTIONS], [e.Studytype.RiskstudyType].[STUDYTYPESTRUCTURE], [e.Studytype.RiskstudyType].[SYSCODE]
FROM (
    SELECT [e].*
    FROM [RV_riskstudy] AS [e]
    LEFT JOIN [RV_project_PROJECTSTUDIES] AS [jSTUDYPROJECT] ON [e].[tdid] = [jSTUDYPROJECT].[ChildID]
    LEFT JOIN [RV_riskstudytype_STUDIES] AS [jSTUDYTYPE] ON [e].[tdid] = [jSTUDYTYPE].[ChildID]
    WHERE ([e].[rid] IN (''0'', @__registerUuid_0) AND [e].[STUDY_STATUS] NOT IN (9, 3)) AND (CHARINDEX(N''\\'', [e].[STUDYNAME]) > 0)
    ORDER BY [e].[SYSIDENT]
    OFFSET @__p_1 ROWS FETCH NEXT @__p_2 ROWS ONLY
) AS [t]
INNER JOIN [RV_riskstudytype_STUDIES] AS [e.Studytype] ON [t].[tdid] = [e.Studytype].[ChildID]
INNER JOIN [RV_riskstudytype] AS [e.Studytype.RiskstudyType] ON [e.Studytype].[ParentID] = [e.Studytype.RiskstudyType].[tdid]
ORDER BY [e.Studytype].[idx]',N'@__registerUuid_0 varchar(160),@__p_1 int,@__p_2 int',@__registerUuid_0='3.1.1.risk-register.da7ecd5e8ecc4c91ab3f5fa2a0686346',@__p_1=0,@__p_2=25

e.g. #3 if I search for \ [""Studyname.Contains(\"\\\")""], dynamic core throws exception

System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.Linq.Dynamic.Core.Exceptions.ParseException: Unterminated string literal
at System.Linq.Dynamic.Core.Tokenizer.TextParser.NextToken() in C:\Users\azurestef\Documents\Github\System.Linq.Dynamic.Core\src\System.Linq.Dynamic.Core\Tokenizer\TextParser.cs:line 361
at System.Linq.Dynamic.Core.Parser.ExpressionParser.ParseArgumentList() in C:\Users\azurestef\Documents\Github\System.Linq.Dynamic.Core\src\System.Linq.Dynamic.Core\Parser\ExpressionParser.cs:line 1846
at System.Linq.Dynamic.Core.Parser.ExpressionParser.ParseMemberAccess(Type type, Expression instance) in C:\Users\azurestef\Documents\Github\System.Linq.Dynamic.Core\src\System.Linq.Dynamic.Core\Parser\ExpressionParser.cs:line 1633
at System.Linq.Dynamic.Core.Parser.ExpressionParser.ParsePrimary() in C:\Users\azurestef\Documents\Github\System.Linq.Dynamic.Core\src\System.Linq.Dynamic.Core\Parser\ExpressionParser.cs:line 744
at System.Linq.Dynamic.Core.Parser.ExpressionParser.ParseUnary() in C:\Users\azurestef\Documents\Github\System.Linq.Dynamic.Core\src\System.Linq.Dynamic.Core\Parser\ExpressionParser.cs:line 733
at System.Linq.Dynamic.Core.Parser.ExpressionParser.ParseMultiplicative() in C:\Users\azurestef\Documents\Github\System.Linq.Dynamic.Core\src\System.Linq.Dynamic.Core\Parser\ExpressionParser.cs:line 679
at System.Linq.Dynamic.Core.Parser.ExpressionParser.ParseAdditive() in C:\Users\azurestef\Documents\Github\System.Linq.Dynamic.Core\src\System.Linq.Dynamic.Core\Parser\ExpressionParser.cs:line 648
at System.Linq.Dynamic.Core.Parser.ExpressionParser.ParseShiftOperator() in C:\Users\azurestef\Documents\Github\System.Linq.Dynamic.Core\src\System.Linq.Dynamic.Core\Parser\ExpressionParser.cs:line 624
at System.Linq.Dynamic.Core.Parser.ExpressionParser.ParseComparisonOperator() in C:\Users\azurestef\Documents\Github\System.Linq.Dynamic.Core\src\System.Linq.Dynamic.Core\Parser\ExpressionParser.cs:line 433
at System.Linq.Dynamic.Core.Parser.ExpressionParser.ParseLogicalAndOrOperator() in C:\Users\azurestef\Documents\Github\System.Linq.Dynamic.Core\src\System.Linq.Dynamic.Core\Parser\ExpressionParser.cs:line 378
at System.Linq.Dynamic.Core.Parser.ExpressionParser.ParseIn() in C:\Users\azurestef\Documents\Github\System.Linq.Dynamic.Core\src\System.Linq.Dynamic.Core\Parser\ExpressionParser.cs:line 298
at System.Linq.Dynamic.Core.Parser.ExpressionParser.ParseAndOperator() in C:\Users\azurestef\Documents\Github\System.Linq.Dynamic.Core\src\System.Linq.Dynamic.Core\Parser\ExpressionParser.cs:line 281
at System.Linq.Dynamic.Core.Parser.ExpressionParser.ParseOrOperator() in C:\Users\azurestef\Documents\Github\System.Linq.Dynamic.Core\src\System.Linq.Dynamic.Core\Parser\ExpressionParser.cs:line 263
at System.Linq.Dynamic.Core.Parser.ExpressionParser.ParseLambdaOperator() in C:\Users\azurestef\Documents\Github\System.Linq.Dynamic.Core\src\System.Linq.Dynamic.Core\Parser\ExpressionParser.cs:line 243
at System.Linq.Dynamic.Core.Parser.ExpressionParser.ParseNullCoalescingOperator() in C:\Users\azurestef\Documents\Github\System.Linq.Dynamic.Core\src\System.Linq.Dynamic.Core\Parser\ExpressionParser.cs:line 230
at System.Linq.Dynamic.Core.Parser.ExpressionParser.ParseConditionalOperator() in C:\Users\azurestef\Documents\Github\System.Linq.Dynamic.Core\src\System.Linq.Dynamic.Core\Parser\ExpressionParser.cs:line 213
at System.Linq.Dynamic.Core.Parser.ExpressionParser.Parse(Type resultType, Boolean createParameterCtor) in C:\Users\azurestef\Documents\Github\System.Linq.Dynamic.Core\src\System.Linq.Dynamic.Core\Parser\ExpressionParser.cs:line 152
at System.Linq.Dynamic.Core.DynamicExpressionParser.ParseLambda(ParsingConfig parsingConfig, Boolean createParameterCtor, ParameterExpression[] parameters, Type resultType, String expression, Object[] values) in C:\Users\azurestef\Documents\Github\System.Linq.Dynamic.Core\src\System.Linq.Dynamic.Core\DynamicExpressionParser.cs:line 67
at System.Linq.Dynamic.Core.DynamicQueryableExtensions.Where(IQueryable source, ParsingConfig config, String predicate, Object[] args) in C:\Users\azurestef\Documents\Github\System.Linq.Dynamic.Core\src\System.Linq.Dynamic.Core\DynamicQueryableExtensions.cs:line 2276
at System.Linq.Dynamic.Core.DynamicQueryableExtensions.Where[TSource](IQueryable1 source, ParsingConfig config, String predicate, Object[] args) in C:\Users\azurestef\Documents\Github\System.Linq.Dynamic.Core\src\System.Linq.Dynamic.Core\DynamicQueryableExtensions.cs:line 2248 at System.Linq.Dynamic.Core.DynamicQueryableExtensions.Where[TSource](IQueryable1 source, String predicate, Object[] args) in C:\Users\azurestef\Documents\Github\System.Linq.Dynamic.Core\src\System.Linq.Dynamic.Core\DynamicQueryableExtensions.cs:line 2254

@StefH
Copy link
Collaborator

StefH commented Mar 18, 2020

@silarmani Did you use latest from normal NuGet ?
This will not yet work.

Can you please try version System.Linq.Dynamic.Core.1.0.20-ci-12980.nupkg from MyGet ?

@silarmani
Copy link

@StefH, yes, it does, as long as I escape all backlashes, which makes sense.

For clarity for everyone I am going to repeat all the scenarios I ran before but with this special version and then show the new queries in EF

e.g. #1 if I search for \t ["Studyname.Contains(\"\\\\t\")"], EF produces the correct condition

exec sp_executesql N'SELECT [e].[tdid]
FROM [RV_riskstudy] AS [e]
LEFT JOIN [RV_project_PROJECTSTUDIES] AS [jSTUDYPROJECT] ON [e].[tdid] = [jSTUDYPROJECT].[ChildID]
LEFT JOIN [RV_riskstudytype_STUDIES] AS [jSTUDYTYPE] ON [e].[tdid] = [jSTUDYTYPE].[ChildID]
WHERE ([e].[rid] IN (''0'', @__registerUuid_0) AND [e].[STUDY_STATUS] NOT IN (9, 3)) AND (CHARINDEX(N''\t'', [e].[STUDYNAME]) > 0)
ORDER BY [e].[SYSIDENT], [e].[tdid]
OFFSET @__p_1 ROWS FETCH NEXT @__p_2 ROWS ONLY',N'@__registerUuid_0 varchar(160),@__p_1 int,@__p_2 int',@__registerUuid_0='3.1.1.risk-register.da7ecd5e8ecc4c91ab3f5fa2a0686346',@__p_1=0,@__p_2=25

e.g. #2 if I search for \\ ["Studyname.Contains(\"\\\\\\\\\")"], EF produces the correct condition (i.e. text with single backslashes does not match)

exec sp_executesql N'SELECT [e].[tdid]
FROM [RV_riskstudy] AS [e]
LEFT JOIN [RV_project_PROJECTSTUDIES] AS [jSTUDYPROJECT] ON [e].[tdid] = [jSTUDYPROJECT].[ChildID]
LEFT JOIN [RV_riskstudytype_STUDIES] AS [jSTUDYTYPE] ON [e].[tdid] = [jSTUDYTYPE].[ChildID]
WHERE ([e].[rid] IN (''0'', @__registerUuid_0) AND [e].[STUDY_STATUS] NOT IN (9, 3)) AND (CHARINDEX(N''\\'', [e].[STUDYNAME]) > 0)
ORDER BY [e].[SYSIDENT], [e].[tdid]
OFFSET @__p_1 ROWS FETCH NEXT @__p_2 ROWS ONLY',N'@__registerUuid_0 varchar(160),@__p_1 int,@__p_2 int',@__registerUuid_0='3.1.1.risk-register.da7ecd5e8ecc4c91ab3f5fa2a0686346',@__p_1=0,@__p_2=25

e.g. #3 if I search for \ ["Studyname.Contains(\"\\\\\")"], EF produces the correct condition (i.e. text with either single or bouble backslashes matches)

exec sp_executesql N'SELECT [e].[tdid]
FROM [RV_riskstudy] AS [e]
LEFT JOIN [RV_project_PROJECTSTUDIES] AS [jSTUDYPROJECT] ON [e].[tdid] = [jSTUDYPROJECT].[ChildID]
LEFT JOIN [RV_riskstudytype_STUDIES] AS [jSTUDYTYPE] ON [e].[tdid] = [jSTUDYTYPE].[ChildID]
WHERE ([e].[rid] IN (''0'', @__registerUuid_0) AND [e].[STUDY_STATUS] NOT IN (9, 3)) AND (CHARINDEX(N''\'', [e].[STUDYNAME]) > 0)
ORDER BY [e].[SYSIDENT], [e].[tdid]
OFFSET @__p_1 ROWS FETCH NEXT @__p_2 ROWS ONLY',N'@__registerUuid_0 varchar(160),@__p_1 int,@__p_2 int',@__registerUuid_0='3.1.1.risk-register.da7ecd5e8ecc4c91ab3f5fa2a0686346',@__p_1=0,@__p_2=25

@silarmani
Copy link

silarmani commented Mar 18, 2020

I have also tested permutations using backslashes and double quotes and they all work flawlessly.

Is there a plan for making it available in the next release via NuGet?

@silarmani
Copy link

e.g. #4 if I search for \" ["Studyname.Contains(\"\\\\\\\"\")"], EF produces the correct condition

exec sp_executesql N'SELECT [e].[tdid]
FROM [RV_riskstudy] AS [e]
LEFT JOIN [RV_project_PROJECTSTUDIES] AS [jSTUDYPROJECT] ON [e].[tdid] = [jSTUDYPROJECT].[ChildID]
LEFT JOIN [RV_riskstudytype_STUDIES] AS [jSTUDYTYPE] ON [e].[tdid] = [jSTUDYTYPE].[ChildID]
WHERE ([e].[rid] IN (''0'', @__registerUuid_0) AND [e].[STUDY_STATUS] NOT IN (9, 3)) AND (CHARINDEX(N''\"'', [e].[STUDYNAME]) > 0)
ORDER BY [e].[SYSIDENT], [e].[tdid]
OFFSET @__p_1 ROWS FETCH NEXT @__p_2 ROWS ONLY',N'@__registerUuid_0 varchar(160),@__p_1 int,@__p_2 int',@__registerUuid_0='3.1.1.risk-register.da7ecd5e8ecc4c91ab3f5fa2a0686346',@__p_1=0,@__p_2=25

@StefH
Copy link
Collaborator

StefH commented Mar 18, 2020

@silarmani Thanks for extended testing!

I want to release this to normal NuGet, however I think I need to change the version from this library to 1.1 because it can be a breaking change.

Also I'm doubting if I should add a new dependency (<PackageReference Include="Sprache.Signed" Version="2.2.0" />) or that I should copy these source files into my library ?

@silarmani
Copy link

silarmani commented Mar 18, 2020

Yes, it is breaking change. I would changed it to 1.1.

e.g. the syntax for searching for \t changed from ["Studyname.Contains(\"\\t\")"] to ["Studyname.Contains(\"\\\\t\")"]

In regards to adding a new dependency, I don't see it as a problem.

I'll stick with the pre-release for now since it doesn't break anything so far.

Thank you!!!

@StefH
Copy link
Collaborator

StefH commented Apr 23, 2020

External dependency to Sprache which was introduced is reverted.
All the string parsing code is now in this project.

New version (1.1) will be released in some days.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Development

No branches or pull requests

5 participants