Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make setting of EventHub Partition Key configurable #148

Closed
colmsnowplow opened this issue Jun 22, 2022 · 0 comments
Closed

Make setting of EventHub Partition Key configurable #148

colmsnowplow opened this issue Jun 22, 2022 · 0 comments
Labels
bug Something isn't working enhancement New feature or request

Comments

@colmsnowplow
Copy link
Collaborator

Current behaviour of eventHub target is to set the Partition Key to whatever the message partition key is.

In instrumenting unit testing for this package, I discovered that this actually impedes the EventHub client's batching behaviour, which appears to be as follows:

  • If there is an explicitly set partition key in the eventHub event, it will only batch events who share a partition key.
  • If there's no explicitly set partition key, it will batch data (not sure how it handles batching logic here)

So, our current instrumentation creates a scenario where default behaviour is that we only have single event batches - because we always explicitly set the partition key.

If the user is setting the partition key via transformation, we will batch appropriately.

I propose we introduce an option to not set the PK in the EH event.

That leaves the question of default behaviour. The options are:

  1. Set the partition key by default, and document that it must be switched off when batching for the EventHubs target
  2. Don't set the partition key by default, and document that it must be switched on when setting partition key at transformation.

Since option 1 keeps the nuance contained within the EventHubs target's configuration docs, I prefer that. Open to alternative viewpoints.

@colmsnowplow colmsnowplow added bug Something isn't working enhancement New feature or request labels Jun 24, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant