burst deadline exceeded error in v1.38.0?

[ukai]

What version of gRPC are you using?

v1.38.0

What version of Go are you using (go version)?

go version go1.16.4 linux/amd64

What operating system (Linux, Windows, …) and version?

Linux

What did you do?

If possible, provide a recipe for reproducing the error.

I've rolled up grpc from v1.37.1 to v1.38.0 in canary recently.

What did you expect to see?

no significant changes between v1.37.1 and v1.38.0

What did you see instead?

We observed busty deadline exceeded error in canary (for ~30 minutes).
error rate in production is usually < 0.0004, but error rate in canary at that time is ~0.004. (10x than prod).
most error seems to be deadline exceeded error.

also see error like

"[transport]transport: loopyWriter.run returning. Err: write tcp 10.49.154.2:5050->10.49.120.4:36094: use of closed network connection"

[zasweq]

The team needs a bit more information. Please clarify with details about your environment. Please try 1.37.1 in Canary and see if the delta is as pronounced.

[ukai]

We're using grpc client in GKE pod to communicate with remotebuildexecution.googleapis.com.

We didn't see such deadline exceeded error spikes with v1.37.1.
(v1.37.1 was used in prod, and we didn't see any difference if we revert canary back to v1.37.1)

[github-actions]

This issue is labeled as requiring an update from the reporter, and no update has been received after 6 days. If no update is provided in the next 7 days, this issue will be automatically closed.

[dfawley]

(Sorry, we didn't remove the label.)

There's so little to go on here, we will definitely need more information to be able to make any progress.

You say this only happened for 30 minutes -- is this kind of burstiness periodically coming & going? Any other information you can provide about your environment (proxies or no proxies, etc), configuration of client & server, etc, would help. And if it's possible to distill it down to a minimal reproduction example you can share, that would be the best.

[dfawley]

The bot is supposed to send a friendly ping 7 days before closing. Reopening for another 7d.

[github-actions]

This issue is labeled as requiring an update from the reporter, and no update has been received after 6 days. If no update is provided in the next 7 days, this issue will be automatically closed.

[ukai]

I tried v1.38.0 in canary again, but couldn't see the similar symptoms for 2 days.
instead, some high latency in client side observed in prod (grpc v1.37.1), so it might not be a regression in v1.38.0
(but some potential issues before v1.38.0?)