fix[physical-expr-adapter]: support casting structs nested inside complex types#20907
fix[physical-expr-adapter]: support casting structs nested inside complex types#20907kosiew merged 1 commit intoapache:mainfrom
Conversation
github-actions
bot
added
logical-expr
| /// # Arguments | ||
| /// * `source_col` - The source array to cast | ||
| /// * `target_field` - The target field definition (including type and metadata) | ||
| /// * `target_type` - The target data type to cast to |
There was a problem hiding this comment.
This refactor made sense. target_field was not used at all, only its inner DataType. There were several callsites that constructed "mock" fields just to be able to call this function with a datatype.
asubiotto
changed the title
kosiew
left a comment
kosiew
left a comment
There was a problem hiding this comment.
@asubiotto
Thanks for working on this.
| assert!(b_col.is_null(0)); | ||
| assert!(b_col.is_null(1)); | ||
| } | ||
|
|
There was a problem hiding this comment.
The new functionality adds fairly intricate ListView and dictionary rebuild logic, but the new tests exercise only the List<Struct> execution path and dictionary compatibility validation.
Do you think It would be worth adding tests for Dictionary<_, Struct> and for ListView<Struct>?
There was a problem hiding this comment.
I initially had parameterized test cases for all types but it was a fair amount of code. I'm not sure, but I'm happy to add them back and we can decide.
There was a problem hiding this comment.
Added concrete tests instead. Let me know what you think
|
Thanks for the review! I will address the comments and have a new version by the middle of next week. |
…types Previously, DefaultPhysicalExprAdapterRewriter.create_cast_column_expr would only allow struct evolution for top-level structs. However, it is valid to have structs that are nested inside other complex types (e.g. Dictionary<_, Struct>, List<Struct>). These cases would previously return an error although they were valid. This commit handles Structs that are arbitrarily nested within certain complex types (Lists, ListViews, and Dicts). Other complex types can be supported in the future, but I think this is good enough for now. Signed-off-by: Alfonso Subiotto Marques <alfonso.subiotto@polarsignals.com>
asubiotto
force-pushed
the
asubiotto/cast
branch
from
1253bae to
6a2a0c7
Compare
|
Addressed the comments. I'm realizing now that this struct casting is not supported if you're also doing an outer cast (list->listview). That seems like a more involved change though, so will open an issue for a future PR. |
|
Is this ready to merge? |
…plex types (apache#20907) Previously, DefaultPhysicalExprAdapterRewriter.create_cast_column_expr would only allow struct evolution for top-level structs. However, it is valid to have structs that are nested inside other complex types (e.g. Dictionary<_, Struct>, List<Struct>). These cases would previously return an error although they were valid. This commit handles Structs that are arbitrarily nested within certain complex types (Lists, ListViews, and Dicts). Other complex types can be supported in the future, but I think this is good enough for now. ## Which issue does this PR close? - Closes apache#20906 ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> Struct evolution is supported, this PR fixes an edge case where struct evolution is not supported when structs are nested inside other complex types. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? Yes, I initially wrote parametrized tests, but I think that's probably overkill. ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> Queries that would previously error, now complete successfully. Signed-off-by: Alfonso Subiotto Marques <alfonso.subiotto@polarsignals.com>
3 participants
Previously, DefaultPhysicalExprAdapterRewriter.create_cast_column_expr would only allow struct evolution for top-level structs. However, it is valid to have structs that are nested inside other complex types (e.g. Dictionary<_, Struct>, List). These cases would previously return an error although they were valid.
This commit handles Structs that are arbitrarily nested within certain complex types (Lists, ListViews, and Dicts). Other complex types can be supported in the future, but I think this is good enough for now.
Which issue does this PR close?
Rationale for this change
Struct evolution is supported, this PR fixes an edge case where struct evolution is not supported when structs are nested inside other complex types.
What changes are included in this PR?
Are these changes tested?
Yes, I initially wrote parametrized tests, but I think that's probably overkill.
Are there any user-facing changes?
Queries that would previously error, now complete successfully.