mouse for ratatui work in local terminal but not xterm.js ttyd

[gwpl]

FYI I've made issue

• mouse support on xterm.js ratatui/ratatui#1674
  but not sure which repo actually should be "responsible" / "interested" so linking from here.

• mouse support on ttyd ( xterm.js ) crossterm-rs/crossterm#971

• Always read user input from /dev/tty crossterm-rs/crossterm#396

[jerch]

@gwpl This is most likely an upstream issue with the mouse protocols/encodings the lib can handle. We only support X10, SGR and SGR-PIXEL as value encoding. Also see https://github.com/xtermjs/xterm.js/blob/master/src/common/services/CoreMouseService.ts.

[gwpl]

Let me follow up in ratatui repository : ratatui/ratatui#1674

[joshka]

Copying my comment from Ratatui here (I've closed the issue there as we only handle the rendering side of a UI).

---

Ratatui doesn't handle input events, we only handle the rendering part of UIs. Our examples all use crossterm for handling this (except for the few that target termion and termwiz backend). I'm closing this not because it's not a problem, but because the issue belongs mainly in terminal libs, not in the rendering lib. I'd suggest creating a few small test apps using crossterm, termion, and termwiz to explore how each of these handle mouse reporting. I'm guessing that there's likely some OS specific stuff that will impact how this works also. The relevant crossterm code to look at is the EnableMouseCapture type, which has some comments should help you understand how this works. On OSes that use ANSI escape codes, you can take the escape sequences and you may be able to slap them in a shell script for the same effect as what you're seeing on the crossterm side.

All that said, I suspect that this might possibly be something to do with crossterm-rs/crossterm#396

[jerch]

@gwpl Im not sure, if crossterm-rs/crossterm#396 is the right direction here - that issue is more about enabling terminal interactivity for piped commands, which is a quite old debate on its own - should a terminal command still behave with interactivity in mind, if its is piped?

Answer for most older cmdline apps - nope. If they get their input from a pipe, they would typically switch off any interactivity settings and behave like a piping filter command. Thats esp. true for all POSIX / shell scripting helpers out there.
Some pager apps decided to deviate from that behavior or have at last a cmdline switch to explicitly allow interactivity via a 3rd filedescriptor (usually from /dev/tty, which always points to the controlling terminal attached to the process).

All in all - thats a very different discussion. For sgr-pixel to work in crossterm, it would have to support that protocol/encoding in the first place as the driving terminal lib underneath and expose some kind of API to hook into your business logic for that.

[gwpl]

@jerch , thanks for input! Although in case of "mouse-drawing" example from ratatui repo, run via ttyd , there is no piping involved IIUC (if I understand correctly), and ttyd is giving terminal to program (which works e.g. in KDE Konsole terminal).

[jerch]

@gwpl Exactly - for your case there is no command piping involved at all. ttyd acts a terminal emulator (TE) here, thats like running your KDE terminal or xterm or any other TE. For commands run in the terminal provided by the terminal emulator it does not matter, how complicated the TE setup itself is (ttyd has to do a few more things than a typical classical desktop TE) - the commands would see the same PTY interface provided by the OS.

What does matter to commands though - the emulated terminal and hence the supported terminal sequences. If you say, that the DEC locator example works in KDE terminal, it basically means, that they implemented the DEC locator protocol and handle it accordingly. Imho xterm also does and a few older TEs as well. xterm.js does not, thus ttyd cannot handle it (as it provides the emulation part of ttyd). Others like gnome terminal also dont understand that protocol. sgr-pixel is more wide-spread among TEs and much easier to implement, thus prolly the better bet here.

Idk what role crossterm plays in the setup here - is that some "terminal driver lib" in a sense that it provides easy access to terminal sequences for cli programmers (aka curses-like)?

[joshka]

The crossterm code is at https://docs.rs/crossterm/latest/src/crossterm/event.rs.html#316 / https://github.com/crossterm-rs/crossterm/blob/12f36ec31699b3099766fa7b68b01bc6c091bf9e/src/event.rs#L322

#[cfg(feature = "events")]
impl Command for EnableMouseCapture {
    fn write_ansi(&self, f: &mut impl fmt::Write) -> fmt::Result {
        f.write_str(concat!(
            // Normal tracking: Send mouse X & Y on button press and release
            csi!("?1000h"),
            // Button-event tracking: Report button motion events (dragging)
            csi!("?1002h"),
            // Any-event tracking: Report all motion events
            csi!("?1003h"),
            // RXVT mouse mode: Allows mouse coordinates of >223
            csi!("?1015h"),
            // SGR mouse mode: Allows mouse coordinates of >223, preferred over RXVT mode
            csi!("?1006h"),
        ))
    }

    #[cfg(windows)]
    fn execute_winapi(&self) -> std::io::Result<()> {
        sys::windows::enable_mouse_capture()
    }

    #[cfg(windows)]
    fn is_ansi_code_supported(&self) -> bool {
        false
    }
}

So depending if you're running this on Windows (@gwpl you haven't mentioned AFACT), your app will call Windows system calls to enable this (there's a wrapper API for crossterm that sits around the windows apis). Otherwise the Ansi sequences in use are right there in front of you for diagnosis. I don't know what xterm.js / ttyd are expecting, but that's what you're getting right now.

[gwpl]

Big apology and AGAIN BIG THANK YOU for all contributing to issue, it motivated me to deep dive into terminal mouse protocols and learn a lot!

(regarding System: I am using Linux)

After I did some reading and "hello world" apps, and restarted everything from scratch, I found that mouse-drawing example from ratatui actually works if everything is compiled and run correctly.

Sharing with community, minimal mouse example:

• "crossterm hello world" app listening to mouse and keyboard events : https://github.com/edu-samples/term_mouse_rust/blob/master/examples/term_mouse_crossterm.rs

some resources on Linux mouse protocols, how to enable them and what resources are available:

• https://www.perplexity.ai/search/what-are-mouse-protocols-for-l-EINcxe5SR9.kxvrBfsUs5Q
• https://www.perplexity.ai/search/what-are-available-terminal-mo-K2rCCN0xS6S0V0FW3d5GQA
• https://www.perplexity.ai/search/what-do-we-know-about-those-te-GEe5Z8GORmSG.pWUFqOUXg

LAST NOT LEAST:

Standard mouse protocols seem to read position in text column/line, which is great for text purposes.

Still would be great to find a way how to read in pixel precision for drawing in teminal via Sixel protocol, but that's topic for other tickets ( e.g. saitoha/libsixel#186 )

[gwpl]

(you can now see for some time I will host demo: http://188.166.37.88:2050/ , select "21" mouse drawing for mouse)

[jerch]

LAST NOT LEAST:

Standard mouse protocols seem to read position in text column/line, which is great for text purposes.

Still would be great to find a way how to read in pixel precision for drawing in teminal via Sixel protocol, but that's topic for other tickets ( e.g. saitoha/libsixel#186 )

For that there is SGR-PIXEL - it basically replaces the row;col values with x;y pixel values.

@joshka It seems, that 1016 for SGR-PIXEL is not listed in your code snippet?