Renderdoc doesn't work on Linux

[u3shit]

Malt version

Release & Development

Blender version

v3.6.2-69-g71b55b491e9

OS

Gentoo Linux amd64

Hardware info

AMD Ryzen 9 5950X | 128GB RAM | AMD RX 580, RX 6700 XT

Issue description and reproduction steps

I've set up the path to renderdoccmd in the addon settings, but when I switch to the Malt renderer I get an error in the console:

Injecting into PID 30606                         
Failed to inject: RenderDoc injection failed: Injecting into already running processes is not supported on non-Windows systems

And of course it doesn't work. I think you'd need to use renderdoccmd capture instead of inject, but doing that with python's multiprocessing lib will probably not be simple.
As a workaround, you can start blender with renderdoccmd capture --opt-hook-children ./blender-launcher to run everything under renderdoc or set LD_PRELOAD correctly before self.process.start() in Client_API.py, but then I had a different problem: it doesn't capture anything at all. It works with other OpenGL programs, I can even capture blender's gui, but not with malt. The problem seems to be that PyOpenGL directly dlopen's libOpenGL and gets the symbols from it, bypassing the LD_PRELOAD hack. Editing Malt/.Dependencies-310/OpenGL/platform/glx.py to load renderdoc instead of OpenGL solves the problem, but this is certainly not a solution that should go into release... A probably better solution is to fix renderdoc to also handle libOpenGL, not just the (apparently) deprecated libGL: https://github.com/baldurk/renderdoc/blob/30a013907dc1b308a33c0c5c89be2261e51f00a9/renderdoc/driver/gl/glx_hooks.cpp#L814

Attachments

Blend: just select Malt as renderer and try to render the default cube.

[pragma37]

Thank you for the detailed report! And sorry for the late reply.

I wonder if multiprocessing.set_executable could be used to workaround the issue.

[u3shit]

No problem.
I was thinking about that, but unfortunately you can only specify a single executable, no command line arguments. Could be worked around by a little shell script, but then you have the problem of figuring out where the python executable is (you want to run it using blender's python, not the system python).

[pragma37]

The blender's Python path can be retrieved with sys.executable.
So, if the shell script workaround works, it should be possible to generate it at load time.

[u3shit]

Oh, so you want to generate the shell script runtime? Should be possible, just make sure to properly escape the string (shlex.quote should handle it). And of course, this is only for unices, on Windows the current solution needs to stay.