feat: Example of produce and consume from Kafka#2805
Open
akainth015 wants to merge 1 commit intohydro-project:mainfrom
Open
feat: Example of produce and consume from Kafka#2805akainth015 wants to merge 1 commit intohydro-project:mainfrom
akainth015 wants to merge 1 commit intohydro-project:mainfrom
Conversation
akainth015
force-pushed
the
feat/kafka-example
branch
6 times, most recently
from
0873fbe to
ff3c338
Compare
akainth015
force-pushed
the
feat/kafka-example
branch
7 times, most recently
from
82d91db to
9427980
Compare
- Add hydro_test/src/kafka/mod.rs with kafka_producer, kafka_consumer, dest_kafka, and setup_topic helpers following the SQS PR hydro-project#2746 pattern - Complete hydro_test/examples/kafka.rs: leader produces 1M financial transactions, consumer cluster computes per-account balances - Add 'kafka' feature flag gating rdkafka and futures-util as optional deps - Add llvm-tools component to rust-toolchain.toml for rust-lld
akainth015
force-pushed
the
feat/kafka-example
branch
from
9427980 to
d05a2d3
Compare
akainth015
changed the title
shadaj
reviewed
May 1, 2026
| # https://github.com/GitoxideLabs/cargo-smart-release/issues/36 | ||
| example_test = { path = "../example_test", version = "^0.0.0", optional = true } | ||
| hydro_build_utils = { path = "../hydro_build_utils", version = "^0.0.1", optional = true } | ||
| rdkafka = { version = "0.39.0", optional = true, features = ["cmake-build", "ssl-vendored"] } |
Member
There was a problem hiding this comment.
Is there an option to use aws-lc-sys here? Then wouldn't need to install libcurl?
Author
There was a problem hiding this comment.
Shadaj found this open PR to fix the issue, but it hasn't gotten traction. That's the reason we're seeing the issue with curl.h even though we're not using curl
| "clippy", | ||
| # https://github.com/dtolnay/trybuild?tab=readme-ov-file#troubleshooting | ||
| "rust-src", | ||
| "llvm-tools", |
Author
There was a problem hiding this comment.
Linker tools for the statically-linked libraries that get pulled in by my changes
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This creates a Kafka example that can do produce and/or consume and print basic throughput information. The performance is not perfect yet, but we'll make changes to the Hydro API that let us do even better.