Phantom Bugs in Places You Won't Expect
Recently, I've been dealing with problems when running the latest Mesa from git (the open Linux userspace graphics driver stack).
I was getting random crashes on startup in some games unless I enabled logging in DXVK. I spent hours trying to find the problem in Proton, bisecting it to recent X11 changes in Proton – but none of them made any sense! After conversing with the authors of the commits, we both had no idea still, and they couldn't reproduce it, so I decided to move on for now and work on something else.
Fast forward a bit where I was wanting to implement accelerated NV12 video compositing into Gamescope (a Wayland compositor designed for gaming in both nested and embedded cases) but it was hanging on startup... Weird. That doesn't happen with the normal mesa package.
I managed to bisect that down to the new virtio driver that was added in and enabled by default in the AUR mesa-git package recently. I am not using the driver at all, nor does it even initialize or expose any devices how bizzare!
Digging deeper, I managed to find that in it's initialization code it was zero-initializing some structure called virtgpu, trying to find+initialize a virtual gpu, failed, then destroys it. When destroying it, it tries to close the file descriptor (fd) of the GPU it opened, but invalid FDs are -1, and it was zero initialized so it was closing fd 0, which is initially reserved for stdout.
The reason Gamescope and Proton were hanging/crashing, was because when fd 0 was closed, something else would swoop in it's place. In Gamescope's case, some Xorg socket thing. The reason enabling DXVK logging worked around it, was because DXVK opened a file to log to, which took the place of fd 0 and all future stdout ended up getting spewed there.
Anyway, it was a nice way to kill two birds with one stone!
https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10814
Additional stuff along the way
I also ended up having to fix some a driver bug (in the one I was actually using lol) to implement the video compositing too: https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10816
