io->WantCaptureKeyboard not catching SDL touch finger events the first time

[ingvart]

Dear ImGui 1.89.2 (18920)
--------------------------------
sizeof(size_t): 8, sizeof(ImDrawIdx): 2, sizeof(ImDrawVert): 20
define: __cplusplus=202002
define: _WIN32
define: _WIN64
define: __MINGW32__
define: __MINGW64__
define: __GNUC__=12
--------------------------------
io.BackendPlatformName: imgui_impl_sdl
io.BackendRendererName: imgui_impl_opengl3
io.ConfigFlags: 0x00000000
io.ConfigInputTextCursorBlink
io.ConfigWindowsResizeFromEdges
io.ConfigMemoryCompactTimer = 60.0
io.BackendFlags: 0x0000000E
 HasMouseCursors
 HasSetMousePos
 RendererHasVtxOffset
--------------------------------

Version/Branch of Dear ImGui:

Version: 1.89.2
Branch: master

Back-end/Renderer/Compiler/OS

Back-ends:
imgui_impl_sdl
imgui_impl_opengl3

My Issue/Question:

Example code:

while (SDL_PollEvent(&e))   {
  ImGui_ImplSDL2_ProcessEvent(&e);
  printf(Mouse: %c  Keyboard: %c\n", (io->WantCaptureMouse ? 'Y' : 'N'), (io->WantCaptureKeyboard ? 'Y' : 'N'));
  ...
}

Issue description:

My laptop has a touchscreen.
My application draws OpenGL on the screen and uses ImGui.
If I click the OpenGL area with the mouse, "io->WantCaptureMouse" will return false as expected.
If I move the mouse to the ImGui area and click, it will return true, as expected.
If I then move the mouse back to the OpenGL area, it will again return false, as expected.
If I then touch the screen with my finger on the ImGui area, it will return false the first time, and then return true consecutively.
If I then (after touching ImGui components) touch the OpenGL area again, it will return true the first time, and then return false consecutively.
So the first touch with a finger is not "updated" in ImGui, and WantCaptureMouse returns the wrong value.

I have not found a way to work around this yet.

I am suspecting that the issue is due to ImGui not receiving any mouse move events and depends on these to determine that the mouse has entered it's area.
Between two clicks with the mouse, a series of mouse move events will be generated by SDL.
Between a click with the mouse or finger and another click with a finger, no mouse move events are generated.

Perhaps I can create a "fake" mouse move event before sending the mouse click event from a finger to "update" ImGui with the current mouse position, but it does seem a little unnecessary. But until I find a better solution, I will attempt this.

Or am I doing something wrong?

Thank you!

[GamingMinds-DanielC]

WantCaptureMouse and WantCaptureKeyboard get updated during ImGui::NewFrame(). So for an actual mouse it works since moving the cursor over ImGui items and clicking are distributed over lots of frames. With touch events, the emulated mouse moves to the touch position when it happens, so no frames in between. Your workaround would need to distribute the additional move and original touch down events across frames, introducing a latency of one frame.

An alternative would be to just give those events to ImGui, but buffer them for a frame for your own viewport. Then you could use WantCaptureMouse to determine if the events from the previous frame should be handled by your OpenGL viewport, resulting in the added latency for that part only. If you want to get rid of that latency as well, you need to decide based on position if you want to handle that event. But this gets complicated fast since even positions within the bounds of your viewport can be obstructed by ImGui windows, so you would need to iterate through all windows visible during the last frame to decide.

[smilediver]

It was a long time I've wrote this code, but I think this is exactly to solve this problem:

    auto position = convertTouchToImGuiSpace(touch);
    auto& io = ImGui::GetIO();
    auto oldPosition = io.MousePos;
    io.MousePos = position;

    ImGui::UpdateHoveredWindowAndCaptureFlags();

    if (ImGui::IsWindowHovered(ImGuiHoveredFlags_AnyWindow) or io.WantCaptureMouse) {
        event->stopPropagation();
        _capturedTouches.insert(touch->getID());
    }

    io.MousePos = oldPosition;

[ingvart]

Thank you.

This worked. My solution is now:

#include "imgui_internal.h"

...

bool Main::handleEvents()
{
  SDL_Event e;
  while (SDL_PollEvent(&e))
  {
    // Touch finger events are not superceeded by mouse movement events, and we therefore
    // update ImGui with the mouse position and give it a chance to process it before
    // checking if ImGui is going to use the mouse or not.
    if (e.type ==SDL_MOUSEBUTTONDOWN || e.type ==SDL_MOUSEBUTTONUP) {
      io->MousePos = ImVec2(e.button.x, e.button.y);
      ImGui::UpdateHoveredWindowAndCaptureFlags();
    }

    ImGui_ImplSDL2_ProcessEvent(&e);
    
    if (e.type ==SDL_MOUSEBUTTONDOWN || e.type ==SDL_MOUSEBUTTONUP || e.type == SDL_MOUSEMOTION) {
      // Check if ImGui needs the mouse click or move
      if (ImGui::IsWindowHovered(ImGuiHoveredFlags_AnyWindow) || io->WantCaptureMouse) {
        return true;
      }      
    }

    ...
  }
  ...
}

Note:

I had to include "imgui_internal.h" to use UpdateHoveredWindowAndCaptureFlags()

I also found that setting up a fake "mouse move"-SDL_Event with a new mouse position and calling ImGui_ImplSDL2_ProcessEvent before calling io->WantCaptureMouse did not work.
It seems like ImGui_ImplSDL2_ProcessEvent does not update io->MousePos immediately, but queues this event (probably for the next "frame", as explained above).

Many thanks!

[smilediver]

Btw, I'm not 100% sure, but I think ImGui::IsWindowHovered(ImGuiHoveredFlags_AnyWindow) part is not necessary and a check on io->WantCaptureMouse is enough.

[ingvart]

Changing from
if (ImGui::IsWindowHovered(ImGuiHoveredFlags_AnyWindow) or io.WantCaptureMouse) {
to
if (io.WantCaptureMouse) {
works fine.

[ocornut]

Issue #8431 is discussing a "mouse" version of this issue, which is trickier to solve.

But this particular thread for touch had a solution pushed in 1.89.5 (April 2023. Your issue was posted November 2023 but you were using 1.89.2):

"IO: Added io.AddMouseSourceEvent() and ImGuiMouseSource enum. This is to allow backend to specify actual event source between Mouse/TouchScreen/Pen. (#2702, #2334, #2372, #3453, #5693)"

In the SDL backend it manifest into calls like:

io.AddMouseSourceEvent(event->wheel.which == SDL_TOUCH_MOUSEID ? ImGuiMouseSource_TouchScreen : ImGuiMouseSource_Mouse);

When in ImGuiMouseSource_TouchScreen mode, successive teleport + down events are trickled into two frames, which was necessarily for many other features than just setting the io.WantCaptureXXXX flags.

I believe this should solve your problem. If since then you have updated dear imgui and your backend, it should automatically be solved and I'd suggest trying to remove your hack above to verify that it was.

I'll close this because AFAIK the change in 1.89.5 was exactly for this, and I apologize for not answering/spotting this earlier.
If you feel there is any doubt let me know and i will reopen.

[codelimit]

As I understood 1.89.5 release fixes the issue for the case where imgui is the only input event consumer.
Consider you have custom backend with its own UI system and imgui is used as a developer/debug overlay.
So during touch input event handling you need to decide whether to block event (hits any imgui window), or pass down to the next input listener (backend UI system).
How this specific case should be handled?