feat(hydro_test): add example of AWS + SQS support by MingweiSamuel · Pull Request #2746 · hydro-project/hydro

MingweiSamuel · 2026-04-15T23:41:01Z

Not sure if this should land in hydro_std or a new e.g. hydro_aws (or even finer grain hydro_sqs) crate

cloudflare-workers-and-pages · 2026-04-15T23:41:26Z

Deploying hydro with Cloudflare Pages

Latest commit:	`2df53f5`
Status:	✅ Deploy successful!
Preview URL:	https://2ebb4fa9.hydroflow.pages.dev
Branch Preview URL:	https://mingwei-sqs-rebase.hydroflow.pages.dev

View logs

Copilot

Pull request overview

Adds an AWS + SQS integration example to hydro_test, gated behind new Cargo features, to demonstrate sourcing from and writing to SQS within Hydro flows.

Changes:

Introduce hydro_test::aws module with helpers to load AWS SDK config and build an SQS client.
Add SQS source/sink utilities and a runnable aws_sqs example.
Add feature flags (aws, aws_sqs) and wire the example to required-features.

Reviewed changes

Copilot reviewed 6 out of 7 changed files in this pull request and generated 13 comments.

Show a summary per file

File	Description
hydro_test/src/lib.rs	Gate new `aws` module behind `feature = "aws"`.
hydro_test/src/aws/mod.rs	Add AWS SDK config source helper and stageleft rewrite mappings.
hydro_test/src/aws/sqs.rs	Add SQS client/source/dest utilities.
hydro_test/examples/aws_sqs.rs	Add deployment-backed example using the new SQS helpers.
hydro_test/Cargo.toml	Add `aws`/`aws_sqs` features, optional AWS deps, and example feature gating.
Cargo.lock	Pull in AWS SDK dependency graph.
.vscode/settings.json	Enable `aws_sqs` feature for rust-analyzer by default.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Changes made: - Fix typo: `chaning` → `changing` in sqs.rs comment - Replace concrete AWS account ID/queue URL with placeholders in example - Weaken `source_sqs_fifo` guarantees from `TotalOrder`/`ExactlyOnce` to `NoOrder`/`AtLeastOnce` since SQS FIFO only guarantees ordering within a single message group; removed `NonDet` parameter; added detailed doc comment - Add doc comment to `dest_sqs` noting it's for standard queues only (no `message_group_id` set) - Add `features = ["alloc"]` to `futures-util` dependency for stream module - Remove `aws_sqs` from `.vscode/settings.json` cargo features - Improve `nondet!` justification comments in example - Remove unused `TotalOrder` import from sqs.rs Comments dismissed as incorrect/not actionable: - EC2 hardcoded region/AMI: acceptable for an example - Delete-before-return: already has TODO, AtLeastOnce marker is accurate - `?` terminates stream on empty poll: intentional design (same as resolved) - flat_map yields Vec not Message: wrong (Option iterates over inner Vec) Co-authored-by: Infinity 🤖 <infinity@hydro.run>

shadaj · 2026-04-22T00:42:56Z

    "rust-analyzer.rustfmt.extraArgs": [
        "+nightly"
    ],
+    "rust-analyzer.cargo.features": [],


Can be removed?

shadaj · 2026-04-22T00:45:07Z

+        .map(q!(|(message, client)| self::sqs_send(
+            &client, queue_url, message
+        )))
+        .resolve_futures_blocking();


We should probably split out APIs for side effecting resolve vs non side effecting.

Changes made: - Fix typo: `chaning` → `changing` in sqs.rs comment - Replace concrete AWS account ID/queue URL with placeholders in example - Weaken `source_sqs_fifo` guarantees from `TotalOrder`/`ExactlyOnce` to `NoOrder`/`AtLeastOnce` since SQS FIFO only guarantees ordering within a single message group; removed `NonDet` parameter; added detailed doc comment - Add doc comment to `dest_sqs` noting it's for standard queues only (no `message_group_id` set) - Add `features = ["alloc"]` to `futures-util` dependency for stream module - Remove `aws_sqs` from `.vscode/settings.json` cargo features - Improve `nondet!` justification comments in example - Remove unused `TotalOrder` import from sqs.rs Comments dismissed as incorrect/not actionable: - EC2 hardcoded region/AMI: acceptable for an example - Delete-before-return: already has TODO, AtLeastOnce marker is accurate - `?` terminates stream on empty poll: intentional design (same as resolved) - flat_map yields Vec not Message: wrong (Option iterates over inner Vec) Co-authored-by: Infinity 🤖 <infinity@hydro.run>

- Add hydro_test/src/kafka/mod.rs with kafka_producer, kafka_consumer, dest_kafka, and setup_topic helpers following the SQS PR hydro-project#2746 pattern - Complete hydro_test/examples/kafka.rs: leader produces 1M financial transactions, consumer cluster computes per-account balances - Add 'kafka' feature flag gating rdkafka and futures-util as optional deps - Add llvm-tools component to rust-toolchain.toml for rust-lld

PR: #2746

- hydro_test/src/aws/sqs.rs: Replace redundant closures `|vec| futures_util::stream::iter(vec)` with direct function reference `futures_util::stream::iter` (lines 45 and 70) - hydro_test/examples/aws_sqs.rs: Replace explicit auto-deref `&*args.queue_url` with `&args.queue_url` (lines 63 and 80) Co-authored-by: Infinity 🤖 <infinity@hydro.run> PR: #2746

PR: #2746

Changes made: - Fix typo: `chaning` → `changing` in sqs.rs comment - Replace concrete AWS account ID/queue URL with placeholders in example - Weaken `source_sqs_fifo` guarantees from `TotalOrder`/`ExactlyOnce` to `NoOrder`/`AtLeastOnce` since SQS FIFO only guarantees ordering within a single message group; removed `NonDet` parameter; added detailed doc comment - Add doc comment to `dest_sqs` noting it's for standard queues only (no `message_group_id` set) - Add `features = ["alloc"]` to `futures-util` dependency for stream module - Remove `aws_sqs` from `.vscode/settings.json` cargo features - Improve `nondet!` justification comments in example - Remove unused `TotalOrder` import from sqs.rs Comments dismissed as incorrect/not actionable: - EC2 hardcoded region/AMI: acceptable for an example - Delete-before-return: already has TODO, AtLeastOnce marker is accurate - `?` terminates stream on empty poll: intentional design (same as resolved) - flat_map yields Vec not Message: wrong (Option iterates over inner Vec) Co-authored-by: Infinity 🤖 <infinity@hydro.run> PR: #2746

- Add hydro_test/src/kafka/mod.rs with kafka_producer, kafka_consumer, dest_kafka, and setup_topic helpers following the SQS PR hydro-project#2746 pattern - Complete hydro_test/examples/kafka.rs: leader produces 1M financial transactions, consumer cluster computes per-account balances - Add 'kafka' feature flag gating rdkafka and futures-util as optional deps - Add llvm-tools component to rust-toolchain.toml for rust-lld

PR: #2746

- hydro_test/src/aws/sqs.rs: Replace redundant closures `|vec| futures_util::stream::iter(vec)` with direct function reference `futures_util::stream::iter` (lines 45 and 70) - hydro_test/examples/aws_sqs.rs: Replace explicit auto-deref `&*args.queue_url` with `&args.queue_url` (lines 63 and 80) Co-authored-by: Infinity 🤖 <infinity@hydro.run> PR: #2746

PR: #2746

Changes made: - Fix typo: `chaning` → `changing` in sqs.rs comment - Replace concrete AWS account ID/queue URL with placeholders in example - Weaken `source_sqs_fifo` guarantees from `TotalOrder`/`ExactlyOnce` to `NoOrder`/`AtLeastOnce` since SQS FIFO only guarantees ordering within a single message group; removed `NonDet` parameter; added detailed doc comment - Add doc comment to `dest_sqs` noting it's for standard queues only (no `message_group_id` set) - Add `features = ["alloc"]` to `futures-util` dependency for stream module - Remove `aws_sqs` from `.vscode/settings.json` cargo features - Improve `nondet!` justification comments in example - Remove unused `TotalOrder` import from sqs.rs Comments dismissed as incorrect/not actionable: - EC2 hardcoded region/AMI: acceptable for an example - Delete-before-return: already has TODO, AtLeastOnce marker is accurate - `?` terminates stream on empty poll: intentional design (same as resolved) - flat_map yields Vec not Message: wrong (Option iterates over inner Vec) Co-authored-by: Infinity 🤖 <infinity@hydro.run> PR: #2746

- Add hydro_test/src/kafka/mod.rs with kafka_producer, kafka_consumer, dest_kafka, and setup_topic helpers following the SQS PR hydro-project#2746 pattern - Complete hydro_test/examples/kafka.rs: leader produces 1M financial transactions, consumer cluster computes per-account balances - Add 'kafka' feature flag gating rdkafka and futures-util as optional deps - Add llvm-tools component to rust-toolchain.toml for rust-lld

MingweiSamuel force-pushed the mingwei/sqs-rebase branch from 74306c4 to 2fc30e0 Compare April 15, 2026 23:41

MingweiSamuel force-pushed the mingwei/sqs-rebase branch 3 times, most recently from 0f2fc28 to 780922c Compare April 20, 2026 18:39

MingweiSamuel changed the title ~~feat(dfir_lang): sqs rebase wip~~ feat(hydro_test): add example of AQS + SQS support Apr 20, 2026

MingweiSamuel force-pushed the mingwei/sqs-rebase branch from ad30558 to 2ada126 Compare April 20, 2026 19:42

MingweiSamuel changed the title ~~feat(hydro_test): add example of AQS + SQS support~~ feat(hydro_test): add example of AWS + SQS support Apr 20, 2026

MingweiSamuel force-pushed the mingwei/sqs-rebase branch from 2ada126 to 67c7bb3 Compare April 20, 2026 19:43

MingweiSamuel marked this pull request as ready for review April 20, 2026 19:43

MingweiSamuel requested review from a team and Copilot April 20, 2026 19:43

Copilot started reviewing on behalf of MingweiSamuel April 20, 2026 19:43 View session

Copilot AI reviewed Apr 20, 2026

View reviewed changes

shadaj approved these changes Apr 22, 2026

View reviewed changes

MingweiSamuel force-pushed the mingwei/sqs-rebase branch from c1c9479 to 636a8b4 Compare April 22, 2026 23:04

MingweiSamuel added a commit that referenced this pull request Apr 28, 2026

add aws, sqs hydro_test

f7dd09c

PR: #2746

MingweiSamuel added a commit that referenced this pull request Apr 28, 2026

SQS working!

d7f5a17

PR: #2746

MingweiSamuel added a commit that referenced this pull request Apr 28, 2026

cleanup sqs one-write code

50143cf

PR: #2746

MingweiSamuel added a commit that referenced this pull request Apr 28, 2026

enable credentials-login feature for getting login from aws cli

016e66e

PR: #2746

MingweiSamuel added a commit that referenced this pull request Apr 28, 2026

enable credentials-login feature for getting login from aws cli

571ab07

PR: #2746

MingweiSamuel force-pushed the mingwei/sqs-rebase branch from 636a8b4 to 2befe7c Compare April 28, 2026 17:09

MingweiSamuel added a commit that referenced this pull request Apr 28, 2026

re-enable aws feature gating

da5208c

PR: #2746

MingweiSamuel and others added 8 commits April 30, 2026 23:14

add aws, sqs hydro_test

6db3591

PR: #2746

SQS working!

6ced45f

PR: #2746

cleanup sqs one-write code

2a8fdfd

PR: #2746

enable credentials-login feature for getting login from aws cli

433e90e

PR: #2746

enable credentials-login feature for getting login from aws cli

82c176a

PR: #2746

re-enable aws feature gating

ee16f97

PR: #2746

MingweiSamuel force-pushed the mingwei/sqs-rebase branch from 2befe7c to 2df53f5 Compare April 30, 2026 23:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(hydro_test): add example of AWS + SQS support#2746

feat(hydro_test): add example of AWS + SQS support#2746
MingweiSamuel wants to merge 8 commits intomainfrom
mingwei/sqs-rebase

MingweiSamuel commented Apr 15, 2026 •

edited

Loading

Uh oh!

cloudflare-workers-and-pages Bot commented Apr 15, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shadaj Apr 22, 2026

Uh oh!

shadaj Apr 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

MingweiSamuel commented Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cloudflare-workers-and-pages Bot commented Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying hydro with Cloudflare Pages

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shadaj Apr 22, 2026

Choose a reason for hiding this comment

Uh oh!

shadaj Apr 22, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

MingweiSamuel commented Apr 15, 2026 •

edited

Loading

cloudflare-workers-and-pages Bot commented Apr 15, 2026 •

edited

Loading