You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When deleting a namespace with an injected mesh, the system continues to process resources even though Kubernetes services and other objects may already be deleted. This results in unnecessary errors and processing. In one case, deleting a namespace with 800 services (2 replicas each) generated 125,000 log lines.
Two types of repeated errors occur:
Pod-to-dataplane conversion errors due to missing services:
ERROR discovery.k8s.pod-to-dataplane-converter could not get K8S Service for service tag {"error": "failed to get Service \"kuma-test/fake-service-799\": Service \"fake-service-799\" not found"}
DataplaneInsight errors due to namespace termination:
ERROR xds.sink failed to flush DataplaneInsight {"dataplaneid": {"Mesh":"default","Name":"fake-service-736-6b479fb987-zjsxr.kuma-test"}, "error": "failed to create k8s resource: dataplaneinsights.kuma.io \"fake-service-736-6b479fb987-zjsxr\" is forbidden: unable to create new content in namespace kuma-test because it is being terminated"}
To Reproduce
Create a namespace with an injected mesh.
Deploy ~1000 services with 2 replicas each.
Delete the namespace.
Observe logs attempting to convert pods to dataplanes and flush DataplaneInsights while the namespace is terminating.
Expected behavior
The system should detect namespace termination and avoid processing resources that may already be deleted.
Additional context (optional)
The system should recognize when a namespace is terminating and avoid unnecessary operations like pod-to-dataplane conversion and DataplaneInsight updates.
The text was updated successfully, but these errors were encountered:
bartsmykla
changed the title
Avoid pod-to-dataplane conversion when namespace is terminating
Avoid unnecessary processing when namespace is terminating
Feb 17, 2025
Kuma Version
0.0.0-preview.v7bf53c40b
Describe the bug
When deleting a namespace with an injected mesh, the system continues to process resources even though Kubernetes services and other objects may already be deleted. This results in unnecessary errors and processing. In one case, deleting a namespace with 800 services (2 replicas each) generated 125,000 log lines.
Two types of repeated errors occur:
Pod-to-dataplane conversion errors due to missing services:
DataplaneInsight errors due to namespace termination:
To Reproduce
Expected behavior
The system should detect namespace termination and avoid processing resources that may already be deleted.
Additional context (optional)
The system should recognize when a namespace is terminating and avoid unnecessary operations like pod-to-dataplane conversion and
DataplaneInsight
updates.The text was updated successfully, but these errors were encountered: