Veritech testing ground #5566
Draft
Veritech testing ground #5566
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
/try |
|
Okay, starting a try! I'll update this comment once it's running...\n |
Dependency Review✅ No vulnerabilities or OpenSSF Scorecard issues found.OpenSSF Scorecard
Scanned Files |
github-actions
bot
added
A-si-settings
nickgerace
force-pushed
the
nick/349374b
branch
from
9e461c0 to
d81056a
Compare
Signed-off-by: Nick Gerace <[email protected]>
|
/try |
|
Okay, starting a try! I'll update this comment once it's running...\n |
Signed-off-by: Nick Gerace <[email protected]>
Signed-off-by: Nick Gerace <[email protected]>
|
/try |
|
Okay, starting a try! I'll update this comment once it's running...\n |
Signed-off-by: Nick Gerace <[email protected]>
|
/try |
|
Okay, starting a try! I'll update this comment once it's running...\n |
|
/try |
|
Okay, starting a try! I'll update this comment once it's running...\n |
Signed-off-by: Nick Gerace <[email protected]>
Signed-off-by: Nick Gerace <[email protected]>
|
/try |
|
Okay, starting a try! I'll update this comment once it's running...\n |
nickgerace
added a commit
that referenced
this pull request
Mar 11, 2025
This commit adds a heartbeat app with force reconnection abilities to veritech, adds naxum metrics and tunes metrics and telemetry everywhere. Many of the changes in this commit come from the testing grounds PR (#5566), but not all changes were included. The "pause-resume" stream wrapper and client hot-swapper changes are either not ready for production use or are not tactics we should employ for one reason or another. For veritech, the new heartbeat app is disabled in tests (by default) and enabled in all other scenarios (by default). The heartbeat app publishes messages via core NATS on a cadence and works with subject prefixes. It uses the instance ID to determine the destination subject. Metrics have been added to track pool exhaustion, For naxum, metrics have been added for the core service and in middleware. Tracing has been tuned to be as noiseless as possible while making use of metrics where possible. These changes include tracking failures for "double ack failure" and standardizing maintain progress task logging. The "ciruitbreaker" with 4 max messages has been removed as a result of these changes. Not only is it an anti-pattern for handling NACK'd messages, but it also masks over potential poison pill messages for consumers who allow for more than 4 re-deliveries. Finally, the graceful shutdown time is now tunable for veritech and its default has moved from 6 hours to 20 minutes. Signed-off-by: Nick Gerace <[email protected]>
nickgerace
added a commit
that referenced
this pull request
Mar 11, 2025
This commit adds a heartbeat app with force reconnection abilities to veritech, adds naxum metrics and tunes metrics and telemetry everywhere. Many of the changes in this commit come from the testing grounds PR (#5566), but not all changes were included. The "pause-resume" stream wrapper and client hot-swapper changes are either not ready for production use or are not tactics we should employ for one reason or another. For veritech, the new heartbeat app is disabled in tests (by default) and enabled in all other scenarios (by default). The heartbeat app publishes messages via core NATS on a cadence and works with subject prefixes. It uses the instance ID to determine the destination subject. Metrics have been added to track pool exhaustion, publisher stats, client stats, handlers doing work and more. For naxum, metrics have been added for the core service and in middleware. Tracing has been tuned to be as noiseless as possible while making use of metrics where possible. These changes include tracking failures for "double ack failure" and standardizing maintain progress task logging. The "ciruitbreaker" with 4 max messages has been removed as a result of these changes. Not only is it an anti-pattern for handling NACK'd messages, but it also masks over potential poison pill messages for consumers who allow for more than 4 re-deliveries. Finally, the graceful shutdown time is now tunable for veritech and its default has moved from 6 hours to 20 minutes. Signed-off-by: Nick Gerace <[email protected]>
nickgerace
added a commit
that referenced
this pull request
Mar 11, 2025
This commit adds a heartbeat app with force reconnection abilities to veritech, adds naxum metrics and tunes metrics and telemetry everywhere. Many of the changes in this commit come from the testing grounds PR (#5566), but not all changes were included. The "pause-resume" stream wrapper and client hot-swapper changes are either not ready for production use or are not tactics we should employ for one reason or another. For veritech, the new heartbeat app is disabled in tests (by default) and enabled in all other scenarios (by default). The heartbeat app publishes messages via core NATS on a cadence and works with subject prefixes. It uses the instance ID to determine the destination subject. Metrics have been added to track pool exhaustion, publisher stats, client stats, handlers doing work and more. For naxum, metrics have been added for the core service and in middleware. Tracing has been tuned to be as noiseless as possible while making use of metrics where possible. These changes include tracking failures for "double ack failure" and standardizing maintain progress task logging. The "ciruitbreaker" with 4 max messages has been removed as a result of these changes. Not only is it an anti-pattern for handling NACK'd messages, but it also masks over potential poison pill messages for consumers who allow for more than 4 re-deliveries. Finally, the graceful shutdown time is now tunable for veritech and its default has moved from 6 hours to 20 minutes. Signed-off-by: Nick Gerace <[email protected]>
nickgerace
added a commit
that referenced
this pull request
Mar 11, 2025
This commit adds a heartbeat app with force reconnection abilities to veritech, adds naxum metrics and tunes metrics and telemetry everywhere. Many of the changes in this commit come from the testing grounds PR (#5566), but not all changes were included. The "pause-resume" stream wrapper and client hot-swapper changes are either not ready for production use or are not tactics we should employ for one reason or another. For veritech, the new heartbeat app is disabled in tests (by default) and enabled in all other scenarios (by default). The heartbeat app publishes messages via core NATS on a cadence and works with subject prefixes. It uses the instance ID to determine the destination subject. Metrics have been added to track pool exhaustion, publisher stats, client stats, handlers doing work and more. For naxum, metrics have been added for the core service and in middleware. Tracing has been tuned to be as noiseless as possible while making use of metrics where possible. These changes include tracking failures for "double ack failure" and standardizing maintain progress task logging. The "ciruitbreaker" with 4 max messages has been removed as a result of these changes. Not only is it an anti-pattern for handling NACK'd messages, but it also masks over potential poison pill messages for consumers who allow for more than 4 re-deliveries. Finally, the graceful shutdown time is now tunable for veritech and its default has moved from 6 hours to 20 minutes. Signed-off-by: Nick Gerace <[email protected]>
nickgerace
added a commit
that referenced
this pull request
Mar 11, 2025
This commit adds a heartbeat app with force reconnection abilities to veritech, adds naxum metrics and tunes metrics and telemetry everywhere. Many of the changes in this commit come from the testing grounds PR (#5566), but not all changes were included. The "pause-resume" stream wrapper and client hot-swapper changes are either not ready for production use or are not tactics we should employ for one reason or another. For veritech, the new heartbeat app is disabled in tests (by default) and enabled in all other scenarios (by default). The heartbeat app publishes messages via core NATS on a cadence and works with subject prefixes. It uses the instance ID to determine the destination subject. Metrics have been added to track pool exhaustion, publisher stats, client stats, handlers doing work and more. For naxum, metrics have been added for the core service and in middleware. Tracing has been tuned to be as noiseless as possible while making use of metrics where possible. These changes include tracking failures for "double ack failure" and standardizing maintain progress task logging. The "ciruitbreaker" with 4 max messages has been removed as a result of these changes. Not only is it an anti-pattern for handling NACK'd messages, but it also masks over potential poison pill messages for consumers who allow for more than 4 re-deliveries. Finally, the graceful shutdown time is now tunable for veritech and its default has moved from 6 hours to 20 minutes. Signed-off-by: Nick Gerace <[email protected]>
nickgerace
added a commit
that referenced
this pull request
Mar 11, 2025
This commit adds a heartbeat app with force reconnection abilities to veritech, adds naxum metrics and tunes metrics and telemetry everywhere. Many of the changes in this commit come from the testing grounds PR (#5566), but not all changes were included. The "pause-resume" stream wrapper and client hot-swapper changes are either not ready for production use or are not tactics we should employ for one reason or another. For veritech, the new heartbeat app is disabled in tests (by default) and enabled in all other scenarios (by default). The heartbeat app publishes messages via core NATS on a cadence and works with subject prefixes. It uses the instance ID to determine the destination subject. Metrics have been added to track pool exhaustion, publisher stats, client stats, handlers doing work and more. For naxum, metrics have been added for the core service and in middleware. Tracing has been tuned to be as noiseless as possible while making use of metrics where possible. These changes include tracking failures for "double ack failure" and standardizing maintain progress task logging. The "ciruitbreaker" with 4 max messages has been removed as a result of these changes. Not only is it an anti-pattern for handling NACK'd messages, but it also masks over potential poison pill messages for consumers who allow for more than 4 re-deliveries. For NATS event callbacks, we will now add the "sid" in the label for slow consumers and have removed the trace log. Finally, the graceful shutdown time is now tunable for veritech and its default has moved from 6 hours to 20 minutes. Signed-off-by: Nick Gerace <[email protected]>
nickgerace
added a commit
that referenced
this pull request
Mar 11, 2025
This commit adds a heartbeat app with force reconnection abilities to veritech, adds naxum metrics and tunes metrics and telemetry everywhere. Many of the changes in this commit come from the testing grounds PR (#5566), but not all changes were included. The "pause-resume" stream wrapper and client hot-swapper changes are either not ready for production use or are not tactics we should employ for one reason or another. For veritech, the new heartbeat app is disabled in tests (by default) and enabled in all other scenarios (by default). The heartbeat app publishes messages via core NATS on a cadence and works with subject prefixes. It uses the instance ID to determine the destination subject. Metrics have been added to track pool exhaustion, publisher stats, client stats, handlers doing work and more. For naxum, metrics have been added for the core service and in middleware. Tracing has been tuned to be as noiseless as possible while making use of metrics where possible. These changes include tracking failures for "double ack failure" and standardizing maintain progress task logging. The "ciruitbreaker" with 4 max messages has been removed as a result of these changes. Not only is it an anti-pattern for handling NACK'd messages, but it also masks over potential poison pill messages for consumers who allow for more than 4 re-deliveries. For NATS event callbacks, all trace logging is gone and now everything flows through metrics. Finally, the graceful shutdown time is now tunable for veritech and its default has moved from 6 hours to 20 minutes. Signed-off-by: Nick Gerace <[email protected]>
nickgerace
added a commit
that referenced
this pull request
Mar 11, 2025
This commit adds a heartbeat app with force reconnection abilities to veritech, adds naxum metrics and tunes metrics and telemetry everywhere. Many of the changes in this commit come from the testing grounds PR (#5566), but not all changes were included. The "pause-resume" stream wrapper and client hot-swapper changes are either not ready for production use or are not tactics we should employ for one reason or another. For veritech, the new heartbeat app is disabled in tests (by default) and enabled in all other scenarios (by default). The heartbeat app publishes messages via core NATS on a cadence and works with subject prefixes. It uses the instance ID to determine the destination subject. Metrics have been added to track pool exhaustion, publisher stats, client stats, handlers doing work and more. For naxum, metrics have been added for the core service and in middleware. Tracing has been tuned to be as noiseless as possible while making use of metrics where possible. These changes include tracking failures for "double ack failure" and standardizing maintain progress task logging. The "ciruitbreaker" with 4 max messages has been removed as a result of these changes. Not only is it an anti-pattern for handling NACK'd messages, but it also masks over potential poison pill messages for consumers who allow for more than 4 re-deliveries. For NATS event callbacks, all trace logging is gone and now everything flows through metrics. Finally, the graceful shutdown time is now tunable for veritech and its default has moved from 6 hours to 20 minutes. Signed-off-by: Nick Gerace <[email protected]>
nickgerace
added a commit
that referenced
this pull request
Mar 11, 2025
This commit adds a heartbeat app with force reconnection abilities to veritech, adds naxum metrics and tunes metrics and telemetry everywhere. Many of the changes in this commit come from the testing grounds PR (#5566), but not all changes were included. The "pause-resume" stream wrapper and client hot-swapper changes are either not ready for production use or are not tactics we should employ for one reason or another. For veritech, the new heartbeat app is disabled in tests (by default) and enabled in all other scenarios (by default). The heartbeat app publishes messages via core NATS on a cadence and works with subject prefixes. It uses the instance ID to determine the destination subject. Metrics have been added to track pool exhaustion, publisher stats, client stats, handlers doing work and more. Upon shutdown, we will now explicitly cancel the shutdown token if the stream has closed for the core app so that the kill app also closes. For naxum, metrics have been added for the core service and in middleware. Tracing has been tuned to be as noiseless as possible while making use of metrics where possible. These changes include tracking failures for "double ack failure" and standardizing maintain progress task logging. The "ciruitbreaker" with 4 max messages has been removed as a result of these changes. Not only is it an anti-pattern for handling NACK'd messages, but it also masks over potential poison pill messages for consumers who allow for more than 4 re-deliveries. For NATS event callbacks, all trace logging is gone and now everything flows through metrics. Finally, the graceful shutdown time is now tunable for veritech and its default has moved from 6 hours to 20 minutes. Signed-off-by: Nick Gerace <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR has become a testing ground for preventing veritech failures to process existing work and take on new work.