Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid unnecessary processing when namespace is terminating #12858

Open
bartsmykla opened this issue Feb 14, 2025 · 1 comment
Open

Avoid unnecessary processing when namespace is terminating #12858

bartsmykla opened this issue Feb 14, 2025 · 1 comment
Labels
kind/bug A bug triage/accepted The issue was reviewed and is complete enough to start working on it

Comments

@bartsmykla
Copy link
Contributor

bartsmykla commented Feb 14, 2025

Kuma Version

0.0.0-preview.v7bf53c40b

Describe the bug

When deleting a namespace with an injected mesh, the system continues to process resources even though Kubernetes services and other objects may already be deleted. This results in unnecessary errors and processing. In one case, deleting a namespace with 800 services (2 replicas each) generated 125,000 log lines.

Two types of repeated errors occur:

  1. Pod-to-dataplane conversion errors due to missing services:

    ERROR discovery.k8s.pod-to-dataplane-converter could not get K8S Service for service tag {"error": "failed to get Service \"kuma-test/fake-service-799\": Service \"fake-service-799\" not found"}
    
  2. DataplaneInsight errors due to namespace termination:

    ERROR xds.sink failed to flush DataplaneInsight {"dataplaneid": {"Mesh":"default","Name":"fake-service-736-6b479fb987-zjsxr.kuma-test"}, "error": "failed to create k8s resource: dataplaneinsights.kuma.io \"fake-service-736-6b479fb987-zjsxr\" is forbidden: unable to create new content in namespace kuma-test because it is being terminated"}
    

To Reproduce

  1. Create a namespace with an injected mesh.
  2. Deploy ~1000 services with 2 replicas each.
  3. Delete the namespace.
  4. Observe logs attempting to convert pods to dataplanes and flush DataplaneInsights while the namespace is terminating.

Expected behavior

The system should detect namespace termination and avoid processing resources that may already be deleted.

Additional context (optional)

The system should recognize when a namespace is terminating and avoid unnecessary operations like pod-to-dataplane conversion and DataplaneInsight updates.

@bartsmykla bartsmykla added kind/bug A bug triage/pending This issue will be looked at on the next triage meeting labels Feb 14, 2025
@bartsmykla bartsmykla changed the title Avoid pod-to-dataplane conversion when namespace is terminating Avoid unnecessary processing when namespace is terminating Feb 17, 2025
@lukidzi
Copy link
Contributor

lukidzi commented Feb 17, 2025

Triage: we should improve this by checking if namespace is terminating

@lukidzi lukidzi added triage/accepted The issue was reviewed and is complete enough to start working on it and removed triage/pending This issue will be looked at on the next triage meeting labels Feb 17, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug A bug triage/accepted The issue was reviewed and is complete enough to start working on it
Projects
None yet
Development

No branches or pull requests

2 participants