Feature: Add the "?." operator (null-conditional operator) to support navigation properties with null values

[jotab123]

Hi Stef,

I'm trying to project a sequence of items with a related property (object) which in some cases is null.

Having this classes:

  public class Item
  {
      public int Id { get; set; }
      public RelatedItem Related { get; set; }
  }

  public class RelatedItem
  {
      public string Value { get; set; }
  }

Consider this scenario:

var listItem = new List<Item>();

for (int i = 0; i < 100; i++)
{
    listItem.Add(new Item
    {
       Id = i,
       Related = i % 2 == 0 ? new RelatedItem{ Value = i % 4 == 0 ? "Related value " + i : null } : null
    });
}

var result1 = listItem.Select(i => i.Related?.Value);
var result2 = listItem.AsQueryable().Select("Related.Value");

Result1 works fine and has the expected result (a list of items with Related.Value value or null, in case of Item has Related object assigned or not) but Result2 generates an object reference exception. Is there any way to achieve this?

Thanks!

System.Linq.Dynamic.Core version 1.0.7.6

[StefH]

You could use the isnull(a,b) operator or the ?? operator.

[jotab123]

It seems not to work with navigation properties. Ex:

var result2 = listItem.AsQueryable().Select("Related.Value ?? null");
//or
var result2 = listItem.AsQueryable().Select("isnull(Related.Value,null)");

Throws the exception.

Finally, I could solve it with:
var result2 = listItem.AsQueryable().Select("Related == null ? null : Related.Value");

It would be great for future versions if you consider supporting the ? operator to improve this kind of queries.

Thanks again!

[StefH]

Actually you want the ?. operator ?

[jotab123]

Yes. I'm having problems because of the implicit conversion of types. In the example before, if Value of RelatedItem is a non-nullable type (double, int...) I cannot use a expression like "Related == null ? null : Related.Value", it throws an exception.

[StefH]

https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/operators/null-conditional-operators

[StefH]

@jotab123
I found out that it's not supported for Lambda expressions : An expression tree lambda may not contain a null propagating operator

See also here:
https://stackoverflow.com/questions/44681362/an-expression-tree-lambda-may-not-contain-a-null-propagating-operator

So I can try to add this functionality, but probably it wont work...

[peter-base]

Hi,
I am in the same scenario trying to sort a List of objects by their navigation properties.
How did you ended up sorting by the navigation property (RelatedItem.Value)?
I ended up implementing something like the following:

            string propertyName = "RelatedItem.Value";
            var properties = propertyName.Split('.');
            var prop1 = properties[0]; //RelatedItem
            var prop2 = properties[1]; //Value
           
            bool isValueType = people.AsQueryable().ElementType.GetProperty(prop1).PropertyType.GetTypeInfo().IsValueType;

            if (!isValueType)
            {
                propertyName = $"{prop1} == null ? null : {propertyName}";
            }
           
            var orderedItems = listItems.AsQueryable().OrderBy(propertyName);
            foreach (var item in orderedItems)
            {
                Console.WriteLine($"{item.Id}");
            }

Thank you

[jotab123]

Hi @peter-base ,
You can use something like:

var relatedItemType = people.AsQueryable().ElementType.GetProperty(prop1).PropertyType;
var valueTypeName = relatedItemType.GetProperty(prop2).PropertyType.Name;

var query = $"{prop1} == null ? null : {valueTypeName}?({propertyName})"
var orderedItems = listItems.AsQueryable().OrderBy(query);

Hope it helps!

[jotab123]

Hi Stef,

I've been researching about null-conditional operator and I found some interesting things.

• First: I created this program to analyse the code:

 class Program
    {
        static void Main(string[] args)
        {
            var item = new Item();

            var result = item?.Relation1?.Relation2?.Id;
        }

        class Item
        {
            public int Id { get; set; }
            public Relation1 Relation1 { get; set; }
        }

        class Relation1
        {
            public int Id { get; set; }
            public Relation2 Relation2 { get; set; }
        }

        class Relation2
        {
            public int Id { get; set; }
        }
    }

• Second: Decompile it with ILSpy and look how the Main method is generated by compiler:

// C# Code
// ConsoleApp1.Program
private static void Main(string[] args)
{
	Program.Item item = new Program.Item();
	if (item == null)
	{
		int? arg_48_0 = null;
	}
	else
	{
		Program.Relation1 expr_1B = item.Relation1;
		if (expr_1B == null)
		{
			int? arg_48_0 = null;
		}
		else
		{
			Program.Relation2 expr_2F = expr_1B.Relation2;
			if (expr_2F == null)
			{
				int? arg_48_0 = null;
			}
			else
			{
				new int?(expr_2F.Id);
			}
		}
	}
}

//IL Code
.method private hidebysig static void Main (string[] args) cil managed 
{
	// Method begins at RVA 0x2050
	// Code size 74 (0x4a)
	.maxstack 2
	.entrypoint
	.locals init (
		[0] class ConsoleApp1.Program/Item item,
		[1] valuetype [mscorlib]System.Nullable`1<int32> result,
		[2] valuetype [mscorlib]System.Nullable`1<int32>
	)

	IL_0000: nop
	IL_0001: newobj instance void ConsoleApp1.Program/Item::.ctor()
	IL_0006: stloc.0
	IL_0007: ldloc.0
	IL_0008: brtrue.s IL_0015

	IL_000a: ldloca.s 2
	IL_000c: initobj valuetype [mscorlib]System.Nullable`1<int32>
	IL_0012: ldloc.2
	IL_0013: br.s IL_0048

	IL_0015: ldloc.0
	IL_0016: call instance class ConsoleApp1.Program/Relation1 ConsoleApp7.Program/Item::get_Relation1()
	IL_001b: dup
	IL_001c: brtrue.s IL_002a

	IL_001e: pop
	IL_001f: ldloca.s 2
	IL_0021: initobj valuetype [mscorlib]System.Nullable`1<int32>
	IL_0027: ldloc.2
	IL_0028: br.s IL_0048

	IL_002a: call instance class ConsoleApp1.Program/Relation2 ConsoleApp7.Program/Relation1::get_Relation2()
	IL_002f: dup
	IL_0030: brtrue.s IL_003e

	IL_0032: pop
	IL_0033: ldloca.s 2
	IL_0035: initobj valuetype [mscorlib]System.Nullable`1<int32>
	IL_003b: ldloc.2
	IL_003c: br.s IL_0048

	IL_003e: call instance int32 ConsoleApp1.Program/Relation2::get_Id()
	IL_0043: newobj instance void valuetype [mscorlib]System.Nullable`1<int32>::.ctor(!0)

	IL_0048: stloc.1
	IL_0049: ret
} // end of method Program::Main

• Third: As you can see, the null-conditional operator only generates code that check if non value types are null and creates new nullable type for value types.
  I've been able to reproduce it in your library doing the next changes in ExpressionParser.cs:

//ExpressionParser.cs

Expression ParsePrimary()
        {
            Expression expr = ParsePrimaryStart();
            while (true)
            {
                if (_textParser.CurrentToken.Id == TokenId.Dot)
                {
                    _textParser.NextToken();
                    expr = ParseMemberAccess(null, expr);
                }
                else if (_textParser.CurrentToken.Id == TokenId.NullPropagation)
                {
                    _textParser.NextToken();
                    expr = ParseAsNullableMemberAccess(expr);
                }
                else if (_textParser.CurrentToken.Id == TokenId.OpenBracket)
                {
                    expr = ParseElementAccess(expr);
                }
                else
                {
                    break;
                }
            }
            return expr;
        }

  Expression ParseAsNullableMemberAccess(Expression expr)
        {
            expr = ParseMemberAccess(null, expr);

            if (expr.Type.IsValueType && Nullable.GetUnderlyingType(expr.Type) == null)
            {
                var nullableType = typeof(Nullable<>).MakeGenericType(expr.Type);
                expr = Expression.Convert(expr, nullableType);
            }

            return expr;
        }

And now, I can use this kind of query:

var result1 = itemsQueryable.Select("new(it.Id as Id, it.Relation1?.Relation2?.Id as RelationId)")

//or

var result2 = itemsQueryable.Where("it.Relation1?.Relation2?.Id != null")

Could you consider and test this code to include it in your library?

Regards!

[StefH]

I have to take a look at your code and update some unit tests.

[StefH]

Another option would be to pre-process the string, so that

"it.Relation1?.Relation2?.Id != null"

becomes

"(it.Relation1 != null ? (it.Relation1.Relation2 != null ? it.Relation1.Relation2.Id : null) : null) != null"

[AmbroiseCouissin]

@StefH Hi,

It's what I did on my side. This is how I do it:

private static string IgnoreIfNull(string fieldNames)
{
    string[] splits = fieldNames.Split('?.');
    for (int i = 0; i < splits.Count() - 1; i ++)
    {
        string toBeReplaced = string.Join('.', splits.Take(i + 1));
        fieldNames = fieldNames.Replace(toBeReplaced, $"{toBeReplaced} == null ? null : ({toBeReplaced}") + ")";
    }

    return fieldNames;
}

It will transform:

"it?.Relation1?.Relation2?.Id"

into

"(it == null ? null : (it.Relation1 == null ? null : (it.Relation1.Relation2 == null ? null : it.Relation1.Relation2.Id)))"

[StefH]

@AmbroiseCouissin This looks like a possible solution, however where to execute this code?

[AmbroiseCouissin]

@StefH In my case, I use this way:

list = list.AsQueryable().OrderBy($"{IgnoreIfNull("it?.Relation1?.Relation2?.Id")} desc");

[mtozlu]

Any progress on this?

By the way @AmbroiseCouissin I wrote something like your code. This is more failproof and also null-checks even if no null coalescing operator is defined:

public static class DynamicQueryExtensions
{
    public static string ConvertToNullableNested(string expression, string result = "", int index = 0)
    {
        //Transforms => "a.b.c" to "(a != null ? (a.b != null ? a.b.c : null) : null)"
        if (string.IsNullOrEmpty(expression))
            return null;
        if (string.IsNullOrEmpty(result))
            result = expression;
        var properties = expression.Split(".");
        if (properties.Length == 0 || properties.Length - 1 == index)
            return result;
        var property = string.Join(".", properties.Take(index + 1));
        if (string.IsNullOrEmpty(property))
            return result;
        result = result.Replace(expression, $"({property} != null ? {expression} : 0)");
        return ConvertToNullableNested(expression, result, index + 1);
    }
}

I use it the same way you specified:

query.OrderBy($"{DynamicQueryExtensions.ConvertToNullableNested("Entity.Relation1.Relation2.Id")} desc");

[StefH]

@1dot44mb

Thanks. I did fix a small issue and added generic support.

public static string ConvertToNullableNested<T>(string expression, object defaultValue = null)
{
	return ConvertToNullableNested<T>(expression, string.Empty, 0, defaultValue);
}

public static string ConvertToNullableNested<T>(string expression, string result, int index, object defaultValue)
{
	if (string.IsNullOrEmpty(expression))
	{
		return null;
	}

	if (string.IsNullOrEmpty(result))
	{
		result = expression;
	}

	string[] properties = expression.Split('.');
	if (properties.Length == 0 || properties.Length - 1 == index)
	{
		if (typeof(T).IsValueType)
		{
			return result.Replace(expression, $"{typeof(T).Name}?({expression})");
		}

		return result;
	}

	string property = string.Join(".", properties.Take(index + 1));
	if (string.IsNullOrEmpty(property))
	{
		return result;
	}

	string defaultReplacement = defaultValue != null ? defaultValue.ToString() : "null";
	result = result.Replace(expression, $"({property} != null ? {expression} : {defaultReplacement})");
	return ConvertToNullableNested<T>(expression, result, index + 1, defaultValue);
}

Can be used like:

string finalT = ConvertToNullableNested<int>("it.Relation1.Relation2.Id");
// (it != null ? (it.Relation1 != null ? (it.Relation1.Relation2 != null ? Int32?(it.Relation1.Relation2.Id) : null) : null) : null)

string finalTDef = ConvertToNullableNested<int>("it.Relation1.Relation2.Id", default(int));
// (it != null ? (it.Relation1 != null ? (it.Relation1.Relation2 != null ? Int32?(it.Relation1.Relation2.Id) : 0) : 0) : 0)

[StefH]

@1dot44mb
I did some more research and I've update the code in this PR #222 to support automatic casting to nullable valuetypes.

So this code will work now:

var r1 = q.Select("it != null && it.NestedDto2 != null ? it.NestedDto2.Id : null");

Previously you need to use this code:

var r1 = q.Select("it != null && it.NestedDto2 != null ? int?(it.NestedDto2.Id) : null");

This new logic can be used in combination with ConvertToNullableNested helper method.

A simpler approach for that helper method can also be:

string expression = "it.Relation1.Relation2.Id";
string[] properties = expression.Split('.');
	
var list = new List<string>();
for (int idx = 0; idx < properties.Length - 1; idx++)
{
    string property = string.Join(".", properties.Take(idx + 1));
    list.Add($"{property} != null");
}

string str = $"({string.Join(" && ", list)} ? {expression} : null)";

This code generates a string like:

(it != null && it.Relation1 != null && it.Relation1.Relation2 != null ? it.Relation1.Relation2.Id : null)

[mtozlu]

Thanks.
PR #222 has been a great addition for dynamic Select and OrderBy parameters. In my application, OrderBy parameter is sent from the client table so it can be any column header (including nested properties). Without #222 ; ConvertToNullableNested did not work for me because i couldn't know the type of nested property before sending it to OrderBy extension method.

[StefH]

If you want to this, you can already use this NuGet:
https://www.myget.org/feed/system-linq-dynamic-core/package/nuget/System.Linq.Dynamic.Core/1.0.9.1-ci-1527

[StefH]

@1dot44mb

I've added more logic in PR #223 [not yet merged...]
Can be used like:

var q1 = q.Select("np(it.NestedDto2.NestedDto3.Id, 0)"); // returns int's

// or

var q1 = q.Select("np(it.NestedDto2.NestedDto3.Id)"); // returns nullable int's

NuGet can be found at:
https://www.myget.org/feed/system-linq-dynamic-core/package/nuget/System.Linq.Dynamic.Core/1.0.9.1-ci-1530

[StefH]

@jotab123
@peter-base
@1dot44mb
@AmbroiseCouissin

If you have time, can you do some testing with this NuGet:
https://www.myget.org/feed/system-linq-dynamic-core/package/nuget/System.Linq.Dynamic.Core/1.0.9.1-ci-1530

Null Propagating has been implemented like:

var q1 = q.Select("np(it.NestedDto2.NestedDto3.Id, 0)"); // returns int's

// or

var q1 = q.Select("np(it.NestedDto2.NestedDto3.Id)"); // returns nullable int's

[mtozlu]

Dear @StefH I really appreciate the hard work. Until new year, i am a bit busy with some deadlines and projects. After that, i will happily test the nuget. Sorry for not being able to help sooner.
Best regards.

[StefH]

@1dot44mb Thanks for your help.

[StefH]

@1dot44mb Did you have time to check this new code?

[StefH]

Hello @jotab123, @peter-base, @1dot44mb, @AmbroiseCouissin : can you please test this functionality?

[StefH]

closed via PR

[MaklaCof]

@StefH
Hi, should this work:

public class DbUser
{
    public int? PartnerId {get;set;}
    public DbPartner Partner {get;set;}
}

public class DbPartner
{
    public int Id {get;set;}
    public string Title {get;set;}
}

? Because I get exception (version 1.0.10):

An expression tree lambda may not contain a null propagating operator.

for

.Select("new(Partner?.Title)")

Without ?. I get

Nullable object must have a value

[StefH]

@MaklaCof
you have to use:

var q1 = q.Select("np(it.NestedDto2.NestedDto3.Id, 0)"); // returns int's

// or

var q1 = q.Select("np(it.NestedDto2.NestedDto3.Id)"); // returns nullable int's

[StefH]

I probably should have added this to the An expression tree lambda may not contain a null propagating operator. exception message...

[ani2479]

Hi,

I'm trying to use the np function on a Where:

e.g User.Contact.FirstName.ToLower().Contains("or") where the User can be null, so I was trying to get the User?.Contact?.FirstName.

Maybe I don't really know how to use your library, but can you help me?

Entity.Where ("np(User.Contact.FirstName.ToLower().Contains("tor")") and I get The 'np' (null-propagation) function requires the first argument to be a MemberExpression

Maybe you see something I don't. Thanks.

[StefH]

If the User can be null, just use this code

Entity.Where ("User != null && User.Contact.FirstName.ToLower().Contains(\"tor\")");

[ani2479]

Thanks. is there no way to use the np with the Where?

Thanks again

[StefH]

Sure you can use the np in Where.

Orders.AsQueryable().Where("np(Customer.City, \"\").Contains(\"e\")").Dump();

results in:

-- Region Parameters
DECLARE @p0 NVarChar(1000) = ''
DECLARE @p1 NVarChar(1000) = '%e%'
-- EndRegion
SELECT [t0].[OrderID], [t0].[CustomerID], [t0].[EmployeeID], [t0].[OrderDate], [t0].[RequiredDate], [t0].[ShippedDate], [t0].[ShipVia], [t0].[Freight], [t0].[ShipName], [t0].[ShipAddress], [t0].[ShipCity], [t0].[ShipRegion], [t0].[ShipPostalCode], [t0].[ShipCountry]
FROM [Orders] AS [t0]
LEFT OUTER JOIN [Customers] AS [t1] ON [t1].[CustomerID] = [t0].[CustomerID]
WHERE (
    (CASE 
        WHEN [t0].[CustomerID] IS NOT NULL THEN [t1].[City]
        ELSE CONVERT(NVarChar(15),@p0)
     END)) LIKE @p1

[ani2479]

Thanks,

Everything seems to be working, except for dates. If I do .OrderBy("np(Customer.EndDate)"), we are getting a System.DateTime cannot be null value.

I will try a couple more things.

[StefH]

You mean this error?

ArgumentException: GenericArguments[0], 'System.Nullable1[System.DateTime]', on 'System.Nullable1[T]' violates the constraint of type 'T'.

[ani2479]

Hi, yes.

It is fine if we use any other null columns

[StefH]

I've created a new Issue

[frankiDotNet]

Hi,
is there also a way to generate this for lists?

I have an expression like this where e.g. myList is a srting lis and a member of dataSource;

var expressionText = "np(myList).FirstOrDefault"

LambdaExpression e = DynamicExpressionParser.ParseLambda(ParsingConfig.Default, false, dataSource.GetType(),
               typeof(object), expressionText, null);
            var result = (e.Compile()).DynamicInvoke(dataSource);

does not work as expression, the LambdaExpression looks as following:

{Param_0 => Convert(IIF((Param_0 != null), Param_0.myList, null).FirstOrDefault())}

The C# way would look like this:

myList?.FirstOrDefault()

what would work..

[StefH]

What happens if you use

var expressionText = "np(myList.FirstOrDefault)"

[frankiDotNet]

np(myList.FirstOrDefault) throws an exception with:

{"No property or field 'FirstOrDefault' exists in type 'List1'"}`

[StefH]

and
var expressionText = "np(myList.FirstOrDefault())"

[frankiDotNet]

Exception = {"The 'np' (null-propagation) function requires the first argument to be a MemberExpression"}

But dataSource has a member that is called myList

[StefH]

This should work. Do you have the latest version?

See
https://github.com/StefH/System.Linq.Dynamic.Core/blob/ea887692d003139961c313a1b723a6799a5e8979/test/System.Linq.Dynamic.Core.Tests/ExpressionTests.cs#L1395

[frankiDotNet]

I have updated to the latest nuget version.. should I use a myget version?

[frankiDotNet]

Maybe it is because I am using the DynamicParser?

[StefH]

DynamicParsers uses same code.

In my test, I use .Name
https://github.com/StefH/System.Linq.Dynamic.Core/blob/ea887692d003139961c313a1b723a6799a5e8979/test/System.Linq.Dynamic.Core.Tests/ExpressionTests.cs#L1401

Maybe this only works like this.

Else, can you open a new issue with an complete C# example?

[frankiDotNet]

What is striking, is that I am not using code inside a select or a where clause. I am only using an expression based on an object :
Here my test code. If you want I will create a new issue, maybe I am missing something and we will see it here:

public class TestClass{
    public List<string> MyList {get; set;}
}

var dataSource = new TestClass();
// Let MyList null...

var expressionText = "np(myList.FirstOrDefault())"
LambdaExpression e = DynamicExpressionParser.ParseLambda(ParsingConfig.Default, dataSource.GetType(),
               typeof(object), expressionText, null);
            var result = (e.Compile()).DynamicInvoke(dataSource);

[frankiDotNet]

Opened a new issue based on the information, I think it is because of the primitive list type. #366