Fix filter scalar output and quoted properties#67
Open
Conversation
There was a problem hiding this comment.
Pull request overview
Adds and documents enhancements to the native filter command so it better supports jq-like JSON shaping in CosmosDBShell pipelines, including quoted property access and consistent JSON scalar output for downstream commands.
Changes:
- Extend the parser/AST/evaluator to support filter paths (including quoted properties),
|piping, and builtin calls (length,keys,type,contains,map,select,sort_by). - Normalize filter scalar/sequence outputs to structured JSON in
CommandState.Resultand add LSP/highlighter support for new AST nodes. - Add docs and tests (unit + offline integration) covering quoted projections and scalar JSON output.
Reviewed changes
Copilot reviewed 23 out of 23 changed files in this pull request and generated 10 comments.
Show a summary per file
| File | Description |
|---|---|
| docs/navigation.md | Lists filter among JSON pipeline-aware commands. |
| docs/filter-v1-spec.md | New v1 filter language grammar/semantics spec. |
| docs/commands.md | Adds filter command help section + examples. |
| CosmosDBShell/lang/en.ftl | Adds localized help + error strings for filter. |
| CosmosDBShell/Azure.Data.Cosmos.Shell.Parser/ShellObject/ShellSequence.cs | Introduces a sequence shell object for multi-result filter evaluation. |
| CosmosDBShell/Azure.Data.Cosmos.Shell.Parser/Lexer.cs | Adds ? token support for optional access in filter paths. |
| CosmosDBShell/Azure.Data.Cosmos.Shell.Parser/IAstVisitor.cs | Extends AST visitor surface for new filter expression nodes. |
| CosmosDBShell/Azure.Data.Cosmos.Shell.Parser/ExpressionParser.cs | Parses filter pipes, filter paths (incl. quoted), and filter builtin calls. |
| CosmosDBShell/Azure.Data.Cosmos.Shell.Parser/Expression/FilterPropertySegment.cs | New filter path segment for property access. |
| CosmosDBShell/Azure.Data.Cosmos.Shell.Parser/Expression/FilterPipeExpression.cs | New AST node + evaluator for `a |
| CosmosDBShell/Azure.Data.Cosmos.Shell.Parser/Expression/FilterPathSegment.cs | New base type for filter path segments. |
| CosmosDBShell/Azure.Data.Cosmos.Shell.Parser/Expression/FilterPathExpression.cs | New AST node + evaluator for .-rooted filter paths. |
| CosmosDBShell/Azure.Data.Cosmos.Shell.Parser/Expression/FilterIterateSegment.cs | New filter path segment for [] iteration. |
| CosmosDBShell/Azure.Data.Cosmos.Shell.Parser/Expression/FilterIndexSegment.cs | New filter path segment for [n] indexing. |
| CosmosDBShell/Azure.Data.Cosmos.Shell.Parser/Expression/FilterExpressionUtilities.cs | Shared JSON conversion/equality/compare utilities for filter evaluation. |
| CosmosDBShell/Azure.Data.Cosmos.Shell.Parser/Expression/FilterCallExpression.cs | New AST node + evaluator for filter builtins like map(...). |
| CosmosDBShell/Azure.Data.Cosmos.Shell.Lsp/CosmosShellSemanticTokensHandler.cs | Adds semantic token support for new filter AST nodes. |
| CosmosDBShell/Azure.Data.Cosmos.Shell.Core/ShellInterpreter.Highlighter.cs | Adds syntax-highlighting traversal for new filter AST nodes. |
| CosmosDBShell/Azure.Data.Cosmos.Shell.Commands/FilterCommand.cs | New filter command implementation + output normalization to JSON. |
| CosmosDBShell.Tests/Parser/ExpressionTests.cs | Adds parsing + evaluation unit tests for filter expressions. |
| CosmosDBShell.Tests/Integration/FilterIntegrationTests.cs | Adds offline integration tests for pipeline printing/JSON output. |
| CosmosDBShell.Tests/CommandTests/FilterCommandTests.cs | Adds command-level tests for normalization + error behavior. |
| .github/workflows/validate-and-package.yml | Adjusts CI behavior (skip version/pack steps on PRs) and packaging validation/upload steps. |
- Fix ParseExpression consuming filter pipes greedily (broke "$x = 42 | echo $x"). Introduce ParseFilterExpression for filter-pipe-aware parsing; update FilterCommand and Expression.Parse callers. - ExpressionParser: change TryConsumeQuestion to return the consumed '?' token; use questionToken.End for FilterPath span lengths so '[]?', '[n]?', '.name?', and '."name"?' don't extend into the next token. - ExpressionParser: replace hard-coded English errors with MessageService keys (expected_open_paren, expected_close_paren, expected_close_bracket, expected_array_index). - FilterPathExpression: only set sequence=true when iteration over an array actually occurs (so '.foo[]?' returns scalar null per spec). - FilterExpressionUtilities: cache a static null JsonElement; drop unnecessary .Clone() calls in ToJsonArray and JsonEquals. - docs/commands.md: filter example uses .items (matches query output shape). - validate-and-package.yml: replace 'Remove pointer package' (which deleted the file the upload step required) with a validation step.
ParsePrimary previously treated any identifier starting with '.' as a FilterPathExpression unconditionally. That broke command arguments like 'jq .msg', where '.msg' is a literal jq program string passed to the external command -- the shell evaluated it against the piped input first. Gate filter-path/filter-call/zero-arg-builtin handling behind an inFilterMode flag set by ParseFilterExpression. The filter command (and Expression.Parse used by tests) enable filter mode; ordinary command arguments keep their previous shell-word semantics.
The Compute version properties step is gated by 'github.event_name != pull_request', so on PR runs steps.version.outputs.* are empty. Previously the dotnet build step still passed /p:Version=, /p:FileVersion=, /p:InformationalVersion= with empty values. Wrap each /p:* arg in a guard that omits the property when the corresponding step output is empty.
Adds a 'Zip signed RID publish folders' step to .pipelines/CosmosDB-Shell-Official.yml that runs immediately after the signed publish output is produced. For each RID directory under \\out\, it creates an archive named cosmosdbshell_<arch>_<version>.zip (e.g. cosmosdbshell_linux-arm64_1.0.273-preview.zip) under out\zip\, so the resulting archives are picked up by the OneBranch output artifact upload and can be downloaded and re-uploaded as-is.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
.["Volcano Name"]and."Volcano Name".filter 'type',contains("prod"), and boolean results print and pipeline correctly.ls | filter '.items | length'and quoted property projections.Validation:
dotnet test .\CosmosDBShell.Tests\CosmosDBShell.Tests.csproj --filter "FullyQualifiedName~ExpressionTests|FullyQualifiedName~FilterCommandTests|FullyQualifiedName~FilterIntegrationTests" -p:BaseOutputPath="$env:TEMP\CosmosDBShell-agent-filter-itest\bin\"