JSON Module rewrite with custom JSONPath implementation #974
Conversation
| </ItemGroup> | ||
|
|
||
| <ItemGroup> | ||
| <ProjectReference Include="..\..\libs\server\Garnet.server.csproj" /> | ||
| <InternalsVisibleTo Include="Garnet.test" Key="0024000004800000940000000602000000240000525341310004000001000100011b1661238d3d3c76232193c8aa2de8c05b8930d6dfe8cd88797a8f5624fdf14a1643141f31da05c0f67961b0e3a64c7120001d2f8579f01ac788b0ff545790d44854abe02f42bfe36a056166a75c6a694db8c5b6609cff8a2dbb429855a1d9f79d4d8ec3e145c74bfdd903274b7344beea93eab86b422652f8dd8eecf530d2" /> | ||
| </ItemGroup> |
There was a problem hiding this comment.
Instead of playground, can we use a top level folder called modules and put this in there?
|
Edit: Ran |
|
@darrenge Add expected values in BDN_Benchmark_Config.json |
There was a problem hiding this comment.
What an extensive effort! Nicely done @Vijay-Nirmal. I've added some comments. I haven't looked deeply into the unmitigated issues yet (custom writer & null sets), I'll look at those next and give my opinion...
|
Thoughts about INDENT / NEWLINE / SPACE...
|
|
@TalZaccai I have fixed the review comments
I don't think rewriting a
I didn't get this. Can you elaborate a bit more? I didn't get what you mean by "JSON string"
We can support these params in future, I have some ideas but need some complex work. Will do it as part of a separate PR. I don't recommend returning an error because the current implementation uses a default format to format the output JSON when any of these parameters is passed which has helped me in readying the output. Without any format, the output is very hard to read. Let's get this initial rewrite of the JSON module PR go in then we can do these as part of enhancements.
Basically, write a semi-custom JSON writer using Utf8jsonwriter. |
| /// <param name="offset">The current offset in the input arguments.</param> | ||
| /// <param name="value">The parsed expire option if successful, otherwise <see cref="ExistOptions.None"/>.</param> | ||
| /// <returns>True if the expire option was successfully parsed, otherwise false.</returns> | ||
| public static bool TryGetExpireOption(this ref ObjectInput input, scoped ref int offset, out ExistOptions value) |
There was a problem hiding this comment.
naming: Should be TryGetExistOption (also in the comment)
|
|
||
| // Find parent node using parent path | ||
| var parentNode = rootNode.SelectNodes(GetParentPath(pathStr, out var pathParentOffset)).FirstOrDefault(); | ||
| if (result is null) |
There was a problem hiding this comment.
This is always false, were you trying to do something different here?
There was a problem hiding this comment.
It was supposed to be parentNode is null to break early. Corrected it
| { | ||
| if (rootNode is null) | ||
| { | ||
| errorMessage = default; |
|
@Vijay-Nirmal - just wanted to check - where are we on getting this PR to a point where it is complete and ready as a module to merge? |
|
@badrishc I will fix the review comments soon, I didn't get the review comment notification till yesterday for this |
|
Fixed all review remaining comments @badrishc @TalZaccai |
As part of this PR, the JSON module implementation has been rewritten using a custom implementation of JsonPath to achieve better performance and compatibility with Redis. As part of this rewrite other parts of the Json module have also been rewritten
Main Changes:
JsonPath.Netpackage referenceMiscellaneous:
CmdStringsclass public inCmdStrings.csto allow external access.ExistOptionsenum to public inRespEnums.csfor broader accessibility.Known Issues:
nullusingJSON.SETcommand then the update will fail, it can't find the parent object to update the value. This is a limitation of the custom JSONPath implementation because of the way the Null is represented in JsonNode. Will find a way to support this in future without affecting the performance much. This issue shouldn't block the PR from being merged as this scenario was failing in the current implementation as well.Utf8JsonWriterdoesn't allow enough customizability to supportINDENT,NEWLINEandSPACEoptions in JSON.GET command. We can't inherit the Utf8JsonWriter class as well as it's a sealed class to write our own logic to customize it. This feature will not be supported in STJ in future as well [API Proposal]: Allow inheritance for Utf8JsonWriter class dotnet/runtime#111899 (comment). We have only two options, either Garnet can never support the customizability provided by Redis for certain JSON commands, or Garnet should use Newtonsoft.Json (which supports customisation). Both of these options are not ideal.TODO
Custom JSONPath Benchmark
Note and a disclaimer: JsonCraft.JsonPath is a package I published. If you need Garent's implementation as a separate package you can use JsonCraft.JsonPath package. In the benchmark, it looks slightly slower because it's benchmarked against the
JsonElementimplementation instead ofJsonNode. You can find the exact same implementations as Garent in Experimental folderGarnet BDN Benchmark
In the main branch as of 762a9d7
In this PR as of 7650f5f: (More improvements to come)
Out of scope of this PR
Some of these items I will be raising future PRs to address it
/