Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

weave auto-patch breaks OpenAIGenerators streaming #9014

Open
1 task done
tstadel opened this issue Mar 11, 2025 · 0 comments
Open
1 task done

weave auto-patch breaks OpenAIGenerators streaming #9014

tstadel opened this issue Mar 11, 2025 · 0 comments
Labels
P3 Low priority, leave it in the backlog

Comments

@tstadel
Copy link
Member

tstadel commented Mar 11, 2025

Describe the bug
weave wraps OpenAI's Stream in a way that breaks class inheritance which is why our check for isinstance(response, Stream) fails which leads to an error for anyone who wants to use streaming in OpenAIChatGenerator or OpenAIGenerator together with weave. We ran into this in deepset Studio.

Although it's in general bad practice by the tracing lib providers to wrap well-know API types such as OpenAI's Stream, we've learned that more than one of these libs make use of this pattern anyways. Therefore I'd suggest to part from the type checks in question in Haystack alltogether.

Responsible Code:

is_streaming = isinstance(chat_completion, Stream)
assert is_streaming or streaming_callback is None

if isinstance(completion, Stream):

Error message

Exception has occurred: AssertionError       (note: full exception trace is shown but execution is paused at: <module>)
exception: no description
  File "/home/haystackd/.local/lib/python3.12/site-packages/haystack/components/generators/chat/openai.py", line 251, in run
    assert is_streaming or streaming_callback is None
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/haystackd/.local/lib/python3.12/site-packages/haystack/core/pipeline/pipeline.py", line 80, in _run_component
    component_output = instance.run(**component_inputs)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/haystackd/.local/lib/python3.12/site-packages/haystack/core/pipeline/pipeline.py", line 248, in run
    component_outputs = self._run_component(component, inputs, component_visits, parent_span=span)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/haystackd/debug/debug_pipeline.py", line 17, in <module> (Current frame)
    result = pipeline.run(data={"query": query, "text": query, "question": query, "streaming_callback": lambda x: print(x.content, end="")})
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/runpy.py", line 88, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.12/runpy.py", line 198, in _run_module_as_main
    return _run_code(code, main_globals, None,
AssertionError: 

(Stacktrace produced with Haystack 2.10.3, but current main has the same issue)

Expected behavior
Streaming of OpenAIGenerators works with tracing-libs such as weave.

Additional context
We've seen issues with type checks on tracing-lib-wrapped responses before (See deepset-ai/haystack-core-integrations#1454)

To Reproduce

  1. Run weave.init(project_name) using default settings
  2. Create a pipeline using OpenAIChatGenerator with streaming_callback set.
  3. Run pipeline

FAQ Check

System:

  • OS:
  • GPU/CPU:
  • Haystack version (commit or version number): 2.10.3 (current main has the same issue)
  • DocumentStore:
  • Reader:
  • Retriever:
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
P3 Low priority, leave it in the backlog
Projects
None yet
Development

No branches or pull requests

3 participants