mouse support on ttyd ( xterm.js )

[gwpl]

I've tried examples/app/mouse-drawing and it works locally, however does not work on xterm.js.

Please note that e.g. mc works on xterm.js via ttyd , so mouse works there.

Problem:

e.g. in ratatui repository
https://github.com/ratatui/ratatui
example using crossterm:

running cd examples/apps/mouse-drawing && ttyd -p 2050 cargo run checking http://127.0.0.1:2050 and mouse-drawing does not work.

Expected behaviour:

when running ttyd -p 4321 mc I can click in browser http://127.0.0.1:4321 and clicking with mouse works.

Links

• https://xtermjs.org/ - https://github.com/xtermjs/xterm.js
• https://tsl0922.github.io/ttyd/ - https://github.com/tsl0922/ttyd

Cross linking issue

Cross linking from here , as it' s not clear actually how to triage this, i.e. which repo is "responsible" / can have fix:

• mouse for ratatui work in local terminal but not xterm.js ttyd xtermjs/xterm.js#5311
• mouse for ratatui work in local terminal but not xterm.js ttyd tsl0922/ttyd#1423
• Potentially related crossterm issue: Always read user input from /dev/tty #396
• I learned HERE that it maybe due to crossterm : mouse support on xterm.js ratatui/ratatui#1674 (comment)

Temporary instance to test

I've started instance for few day with ratatui demos for convenience of trying that by keyboard works and mouse not,
for security reasons I will not start instance with mc ;).

mouse-drawing
http://188.166.37.88:2050/ (ensure "http" , not "https" in address bar, as some browsers may try to force https)

And select over examples directory:

using default webgl2:
~/ratatui# ttyd -t disableLeaveAlert=true -t disableResizeOverlay=true -t enableSixel=true -W -p 2048 ~/cargo-example-select --loop
http://188.166.37.88:2048/

using canvas:
~/ratatui# ttyd -t rendererType=canvas -t disableLeaveAlert=true -t disableResizeOverlay=true -t enableSixel=true -W -p 2047 ~/cargo-example-select --loop
http://188.166.37.88:2047/

[joshka]

It might be worth fixing up the wording here to be more crossterm specific - right now the problem description is still mentioning Ratatui examples as it was copied over from the Ratatui issue. Copying my comment over here from Ratatui:

---

Snip

Ratatui doesn't handle input events, we only handle the rendering part of UIs. Our examples all use crossterm for handling this (except for the few that target termion and termwiz backend). I'm closing this not because it's not a problem, but because the issue belongs mainly in terminal libs, not in the rendering lib.

I'd suggest creating a few small test apps using crossterm, termion, and termwiz to explore how each of these handle mouse reporting. I'm guessing that there's likely some OS specific stuff that will impact how this works also. The relevant crossterm code to look at is the EnableMouseCapture type, which has some comments should help you understand how this works. On OSes that use ANSI escape codes, you can take the escape sequences and you may be able to slap them in a shell script for the same effect as what you're seeing on the crossterm side.

All that said, I suspect that this might possibly be something to do with #396

---

Grab the ansi escape sequences from the mentioned code, and work out which ones work for xterm.js, or do a comparison against what xterm needs and what crossterm provides to understand where the gap is.

[gwpl]

Big apology and AGAIN BIG THANK YOU for all contributing to issue, it motivated me to deep dive into terminal mouse protocols and learn a lot!

After I did some reading and "hello world" apps, and restarted everything from scratch, I found that mouse-drawing example from ratatui actually works if everything is compiled and run correctly.

Sharing with community, minimal mouse example:

• "crossterm hello world" app listening to mouse and keyboard events : https://github.com/edu-samples/term_mouse_rust/blob/master/examples/term_mouse_crossterm.rs

some resources on Linux mouse protocols, how to enable them and what resources are available:

• https://www.perplexity.ai/search/what-are-mouse-protocols-for-l-EINcxe5SR9.kxvrBfsUs5Q
• https://www.perplexity.ai/search/what-are-available-terminal-mo-K2rCCN0xS6S0V0FW3d5GQA
• https://www.perplexity.ai/search/what-do-we-know-about-those-te-GEe5Z8GORmSG.pWUFqOUXg

LAST NOT LEAST:

Standard mouse protocols seem to read position in text column/line, which is great for text purposes.

Still would be great to find a way how to read in pixel precision for drawing in teminal via Sixel protocol, but that's topic for other tickets ( e.g. saitoha/libsixel#186 )