[Dest-S3DataLake]: Bugfix: New interface setup does not always await … #56347
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
johnny-schmidt
force-pushed
the
jschmidt/dest-s3-datalake/bugfix-setup-doesn-not-await-start
branch
from
4763f00 to
92acee8
Compare
There was a problem hiding this comment.
🤷 I guess we got lucky and always got the "correct" behavior out of the race condition?
(it sounds like this would have caused loud failure, rather than silent error?)
Yeah I assumed. I guess I could try injecting an artificial delay at the start of setup and see what happens. Actually I will do that right now.
I inserted a 10 second delay and verified that setup didn't even run until after everything else was done and ... nothing broke and the tests passed 🫨 |
What
The load pipeline new interface waits to start a stream (StreamLoader::start) until the first time it sees data for it. However, it does not wait on setup (DestitionWriter::setup) to complete running start. This caused issues in the new BulkLoader. It really should cause issues in S3DataLake, which relies on setup, but I can't find any sign that it does. Tests all pass before and after this. However I thought I'd go ahead and break this out as a bugfix for S3DataLake specifically.