Significant rendering lag when using Vulkan

[rastertail]

Version/Branch of Dear ImGui:

Version: 1.73, 1.75 WIP
Branch: master
Back-end/Renderer/Compiler/OS

Back-ends: Yatekii/imgui-wgpu-rs, imgui_impl_opengl3.cpp, imgui_impl_vulkan.cpp
Operating System: Arch Linux, awesomewm

My Issue/Question:

In my own application using imgui-wgpu-rs, I've noticed that there is a significant input lag for me on Linux, and wgpu uses Vulkan, so I tested further and I've been able to reproduce the issue with the OpenGL and Vulkan examples that are here. Inputs, particularly things like moving windows, are noticeably delayed when using Vulkan compared to with OpenGL.

Screenshots/Video

(left is Vulkan, right is OpenGL)

Standalone, minimal, complete and verifiable example:

examples/example_sdl_opengl3

examples/example_sdl_vulkan

[ocornut]

What does looking at the example
vulkan renderer tells you? As you can imagine, if you were to draw a simple triangle following the mouse you would get the same effect, so this is not really an issue affecting core imgui.

However as many people are relying on the example app/backends we should aim at reducing those issues in them. Would need to investigate the Vulkan example, any help appreciated there!

[rastertail]

I get that some lag is inevitable, and I can see that when using OpenGL, but its much more significant when using Vulkan which surprises me. Given that this issue exists in both the example backend and imgui-wgpu-rs, I agree that it's probably still an issue outside the scope of imgui itself, but I haven't been able to find anything regarding high input latency particularly with Vulkan. Maybe I haven't been searching the right thing, though.

[ocornut]

If you find a solution it would be great if you post about it, but I'm afraid we can't really help here.

Note that if you vsync you'll always get some form of lag between a hardware mouse cursor and normal GPU render path. Try to enable io.MouseDrawCursor = true your cursor will be drawing using regular GPU path and you might be able to compare GL and Vulkan better, but the mouse cursor will feel slower to the user. A possible solution to investigate would be to switch a normal GPU render path cursor when dragging and revert to a HW cursor when not.

[mocabe]

Not sure it's related to your problem, but I remember some of official Vulkan backends using FIFO presentation mode, which introduces much more input lag compared to Mailbox mode. I noticed FIFO introduces too much input lag so switched to Mailbox with FPS limit while implementing my own Vulkan backend.

[rastertail]

Yeah, that's pretty much the solution I ended up on too. I'll leave this issue open for now if someone comes up with any other solution though.

[ocornut]

The comments above suggest changing the main.cpp of our example from:

    VkPresentModeKHR present_modes[] = { VK_PRESENT_MODE_FIFO_KHR };

Into

    VkPresentModeKHR present_modes[] = { VK_PRESENT_MODE_MAILBOX_KHR, VK_PRESENT_MODE_FIFO_KHR };

However I cannot get Mailbox to behave as expected here :(
Keeping up with Vulkan oddities has been quite tedious.

On my laptop with the Intel graphics card, only VK_PRESENT_MODE_FIFO_KHR is supported and even though the documentation says it should be waiting for vertical blank, I'm getting 250+ fps on a 60 Hz screen.

On same laptop with Nvidia with VK_PRESENT_MODE_FIFO_KHR I get a stable 60 FPS, and with VK_PRESENT_MODE_MAILBOX_KHR I get 2000+ FPS. VK_PRESENT_MODE_MAILBOX_KHR is ALSO described as waiting for vsync.

[mocabe]

@ocornut As vulkan spec says, VK_PRESENT_MODE_MAILBOX_KHR does wait vertical sync to present image on screen, but all images to present are internally queued, which means it does not block renderer to send image to the queue. In other words, it works similarly to Nvidia's FastSync. If you use VK_PRESENT_MODE_MAILBOX_KHR you need to implement frame rate limiter.

[ocornut]

Would you be able to sugget minimum code in the Vulkan examples to use VK_PRESENT_MODE_MAILBOX_KHR with a frame rate limiter?

[mocabe]

Maybe the simplest solution is just wait small amount of time after sending commands like this (from my own backend implementation):

    auto endTime         = std::chrono::high_resolution_clock::now();
    auto frameTime       = (endTime - m_pimpl->lastTime);
    auto frameTimeWindow = std::chrono::nanoseconds(1000000000) / m_pimpl->fps;
    auto sleepTime       = frameTimeWindow - frameTime;

    if (sleepTime.count() > 0) {
      m_pimpl->lastTime = endTime + sleepTime;
      std::this_thread::sleep_for(sleepTime);
    } else {
      m_pimpl->lastTime = endTime;
    }

I'm not Vulkan expert so someone can propose better solution though.

[melroy89]

Maybe the simplest solution is just wait small amount of time after sending commands like this (from my own backend implementation):

No this is never the right solution. Never introduce sleeps/waits in your production code people. I think Imgui and GLFW/Vulkan really have some synchronisation issues here. I already tried to disable double buffering (glfwWindowHint(GLFW_DOUBLEBUFFER, GLFW_FALSE)) without any luck. And using glfwWaitEvents() instead of polling, without luck.

Ps. I'm using Imgui for a GUI application, not a game. This issue is definitely related to: #1805

[rokups]

I thought this is an inherent flaw of Linux graphics stack. Windows and MacOS do not exhibit this behavior, and it is somewhat amortized on Linux if 144hz monitor is used. Experience is absolutely terrible with 60hz monitor however.

[melroy89]

Well that could be the case, but imgui is the only interface under Linux that lagg so much. Applications like Firefox, who are also using hardware acceleration doesn't suffer from all this under Linux...

[rokups]

I em experiencing this in all applications. When refresh rate is set to 60hz, even dragging a desktop window produces a visible lag between position cursor grabbed on the window and actual mouse cursor position. Same is true for UI in other 3D applications. Likewise it gets much better with higher refresh rate.

[rastertail]

Totally forgot about this issue... I'm glad to say that I have found a real solution! The problem comes from blocking on acquiring the next swapchain image, losing window events during that period. In my case using wgpu-rs, I moved swapchain image acquisition to another thread, notifying the main window event loop when the next image is ready such that window events are processed as soon as possible, and the input latency is identical to when using OpenGL. I'm not sure why this issue only occurs under Linux (and maybe even only X11), but I'm glad I was able to find a solution.

[melroy89]

@Rytone that is great to hear! Could you maybe add this fix to the Imgui glfw +opengl and/or glfw+vulkan example? That would be great for everybody!

[pattop]

This issue will be fixable by using the VK_EXT_present_timing extension when it becomes available.

As I understand it VK_EXT_present_timing allows for two things: scheduling a presentation time for an image, and understanding the actual time an image was presented.

By knowing the pesentation time it is possible to build a rendering loop which optimally reduces input lag by dynamically adjusting the render loop timing to sample inputs and render the next frame just before it is presented.

[mika314]

Here is my workaround.

modified   main.cpp
@@ -18,8 +18,9 @@
 #include <SDL_vulkan.h>
 #include <vulkan/vulkan.h>
 #include <log/log.hpp>
+#include <chrono>
 
-//#define IMGUI_UNLIMITED_FRAME_RATE
+#define IMGUI_UNLIMITED_FRAME_RATE
 #ifdef _DEBUG
 #define IMGUI_VULKAN_DEBUG_REPORT
 #endif
@@ -496,6 +497,9 @@ int main(int, char**)
 
     // Main loop
     bool done = false;
+
+    const auto TargetFps = 120;
+    auto t0 = std::chrono::high_resolution_clock::now();
     while (!done)
     {
         // Poll and handle events (inputs, window resize, etc.)
@@ -503,14 +507,37 @@ int main(int, char**)
         // - When io.WantCaptureMouse is true, do not dispatch mouse input data to your main application, or clear/overwrite your copy of the mouse data.
         // - When io.WantCaptureKeyboard is true, do not dispatch keyboard input data to your main application, or clear/overwrite your copy of the keyboard data.
         // Generally you may always pass all inputs to dear imgui, and hide them from your application based on those two flags.
-        SDL_Event event;
-        while (SDL_PollEvent(&event))
+        t0 += std::chrono::microseconds(1'000'000 / TargetFps);
+        while (std::chrono::high_resolution_clock::now() < t0)
         {
-            ImGui_ImplSDL2_ProcessEvent(&event);
-            if (event.type == SDL_QUIT)
+          const auto delay = std::chrono::duration_cast<std::chrono::milliseconds>(t0 - std::chrono::high_resolution_clock::now()).count();
+          if (delay < -200)
+            t0 = std::chrono::high_resolution_clock::now();
+          if (delay > 0)
+          {
+            SDL_Event event;
+            auto res = SDL_WaitEventTimeout(&event, delay);
+            if (res)
+            {
+              ImGui_ImplSDL2_ProcessEvent(&event);
+              if (event.type == SDL_QUIT)
                 done = true;
-            if (event.type == SDL_WINDOWEVENT && event.window.event == SDL_WINDOWEVENT_CLOSE && event.window.windowID == SDL_GetWindowID(window))
+              if (event.type == SDL_WINDOWEVENT && event.window.event == SDL_WINDOWEVENT_CLOSE && event.window.windowID == SDL_GetWindowID(window))
                 done = true;
+            }
+          }
+          else
+          {
+            SDL_Event event;
+            while (SDL_PollEvent(&event))
+            {
+              ImGui_ImplSDL2_ProcessEvent(&event);
+              if (event.type == SDL_QUIT)
+                done = true;
+              if (event.type == SDL_WINDOWEVENT && event.window.event == SDL_WINDOWEVENT_CLOSE && event.window.windowID == SDL_GetWindowID(window))
+                done = true;
+            }
+          }
         }

[ShenCiao]

Sadly it's still a problem on May, 2022. There are still perceivable mouse lag after enabling io.MouseDrawCursor on my laptop with a 165Hz monitor. Meanwhile the Opengl or Directx12 backend doesn't have the issue.

After disabling v-sync, framerate of the opengl backend can reach around 1000FPS and Direct12 2000FPS. Vulkan doesn't suffer from the lag issue but it's framerate can only reach 500FPS. Perhaps there are some errors inside the vulkan implementation.

Here is my laptop setup:
OS: Windows 11
CPU: AMD Ryzen 7 4800H with Radeon Graphics
GPU: AMD Radeon(TM) Graphics, NVIDIA GeForce RTX 3060 Laptop GPU

[MrROBUST]

I'm trying to make a simple GUI application (not a game) based on Vulkan, GLFW and Dear ImGui for Windows 10. Faced the same issue, the interface can't keep up with the system cursor with severe lag.
The most annoying thing is that Firefox can run ImStudio without any lag at a steady 60fps in the same environment. What black magic do they do?

@ocornut is it possible to make imgui automatically switch the cursor to software mode when dragging UI elements?

Also I don't think VK_EXT_present_timing can help. It will only give you a hint when the next presentation will happen, so you can adjust the time delta for the physics calculation (to avoid motion jitter at unstable framerates). But you won't know where the user will move the cursor at that point in time.

[flyingpie]

@MrROBUST No satisfying solution really, but you could toggle the cursor mode like this, which does make the cursor stick to any dragged item, though the lag that is then occurred on the cursor is noticable, however less jarring.

Somewhere early in the draw loop:

io.MouseDrawCursor = ImGui::IsMouseDown(ImGuiMouseButton_::ImGuiMouseButton_Left);

[codecnotsupported]

You can use github.com/ishitatsuyuki/LatencyFleX to minimize latency if PresentModeKHR is IMMEDIATE (vsync-off).

[michaeleggers]

Just found this issue (again) after having to deal with this super laggy behaviour on Linux (only). I used @rastertail 's solution and it works. Thanks. Imgui is hardly to blame here. It is a Vulkan oddity as @ocornut put it correctly. It is quite laughable from Vulkan's side in my opinion. The most basic things are quite hard to do and library maintainers (I have to maintain a Vulkan app myself for work) have to deal with these things all the time. I'd not even blame Omar for letting the Vulkan backends to die. I certainly would if I could.

[ocornut]

I'm not letting Vulkan backends to die but all those problems are generally more multi-faceted and difficult to find one right solution for than most people want to see or care about. So those issues generally gets stuck for a while until someone puts on the deep work.

[michaeleggers]

Didn't mean to put words in your mouth. If it made the impression, I'm sorry! It just reflects my frustration with the API. It went in the right direction but fails on so many things. I just tried an OpenGL 4.6 app (with imgui) that I had only running on Windows so far on Linux. It took me 5min to adjust come linker flags and it runs just as well as on Windows. Vulkan just fails at something like that from my experience.

[ocornut]

There are variety of issues with OpenGL drivers, it's equally a mess unfortunately :( Vulkan is just so much complicated, it's a shame IMHO there isn't a modern simple API .

[ocornut]

This still seems to be an issue and I agree that I'm sensing latency when using the Vulkan example, which I don't as much with e.g. DX11 example.

Here are discussions which MIGHT be related. #2938 #3171 #3669 #7825 #7834
I would love if someone could dig into it and provide a PR to fix our examples.