Skip to content

fix: align BroadcastPartition routing in LinkConfig with broadcast semantics#5032

Merged
aglinxinyuan merged 3 commits into
apache:mainfrom
Ma77Ball:fix/zip
May 13, 2026
Merged

fix: align BroadcastPartition routing in LinkConfig with broadcast semantics#5032
aglinxinyuan merged 3 commits into
apache:mainfrom
Ma77Ball:fix/zip

Conversation

@Ma77Ball
Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

LinkConfig.toPartitioning's BroadcastPartition() arm was building channels with fromWorkerIds.zip(toWorkerIds), producing a 1:1 pairing rather than the full sender-by-receiver cross product that broadcast semantics imply.
The sibling ChannelConfig.generateChannelConfigs in the same package already emits a cross product for the broadcast arm, so the two helpers disagreed on routing for the same inputs. As with #4799 for OneToOnePartition,
List.zip also silently truncated the surplus side on unequal-length inputs. This PR replaces the zip with the same flatMap / map cross product already used by the HashPartition, RangePartition, and
UnknownPartition arms in this file, which fixes the semantics and removes the silent truncation hazard.

Any related issues, documentation, or discussions?

Closes: #4802

How was this PR tested?

The two LinkConfigSpec cases that pinned the previous behavior (with comments explicitly inviting this realignment) were flipped to assert the cross product: 3 senders by 3 receivers now produces 9 channels, and the
Asymmetric 3 by 2 case now produces 6 channels rather than silently truncating to 2. sbt scalafmtAll was run; CI should pass the existing LinkConfigSpec, ChannelConfigSpec, and downstream scheduling specs.

Was this PR authored or co-authored using generative AI tooling?

Co-authored with Claude Opus 4.7 in compliance with ASF

@Ma77Ball
Copy link
Copy Markdown
Contributor Author

/request-review @aglinxinyuan

@github-actions github-actions Bot requested a review from aglinxinyuan May 12, 2026 19:50
@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented May 12, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 42.85%. Comparing base (1497806) to head (c53431f).

Additional details and impacted files
@@             Coverage Diff              @@
##               main    #5032      +/-   ##
============================================
+ Coverage     42.77%   42.85%   +0.08%     
- Complexity     2193     2199       +6     
============================================
  Files          1045     1031      -14     
  Lines         39985    38156    -1829     
  Branches       4217     4007     -210     
============================================
- Hits          17102    16352     -750     
+ Misses        21824    20787    -1037     
+ Partials       1059     1017      -42     
Flag Coverage Δ *Carryforward flag
access-control-service 39.53% <ø> (ø) Carriedforward from deb0e2c
agent-service 33.72% <ø> (ø) Carriedforward from deb0e2c
amber 43.54% <100.00%> (+0.06%) ⬆️
computing-unit-managing-service 0.00% <ø> (ø) Carriedforward from deb0e2c
config-service 0.00% <ø> (ø) Carriedforward from deb0e2c
file-service 32.18% <ø> (ø) Carriedforward from deb0e2c
frontend 33.08% <ø> (-0.96%) ⬇️ Carriedforward from deb0e2c
python 88.89% <ø> (+0.01%) ⬆️ Carriedforward from deb0e2c
workflow-compiling-service 47.72% <ø> (ø) Carriedforward from deb0e2c

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes LinkConfig.toPartitioning so that BroadcastPartition() produces the full sender×receiver channel cross product (broadcast semantics) instead of incorrectly using zip (1:1 pairing with silent truncation on length mismatch). This aligns LinkConfig routing behavior with the existing ChannelConfig.generateChannelConfigs behavior in the same package and resolves issue #4802.

Changes:

  • Update LinkConfig.toPartitioning’s BroadcastPartition() arm to generate channels via cross product (flatMap/map) rather than zip.
  • Update LinkConfigSpec broadcast test cases to assert the cross-product routing for symmetric and asymmetric worker list sizes.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
amber/src/main/scala/org/apache/texera/amber/engine/architecture/scheduling/config/LinkConfig.scala Fix broadcast routing to emit sender×receiver channel cross product instead of zip pairing.
amber/src/test/scala/org/apache/texera/amber/engine/architecture/scheduling/config/LinkConfigSpec.scala Update broadcast partition specs to match corrected broadcast semantics (including unequal list sizes).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copy link
Copy Markdown
Contributor

@aglinxinyuan aglinxinyuan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@aglinxinyuan aglinxinyuan merged commit 5bff1b4 into apache:main May 13, 2026
13 checks passed
@Ma77Ball Ma77Ball deleted the fix/zip branch May 13, 2026 02:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

LinkConfig.toPartitioning emits broadcast channels via zip instead of cross product

4 participants