I avoided transcoding by making sure to only host videos compatible with all clients. Also, I will spend the time to reencode codes that are too large for my taste. Which happens after a rip of uhd disks.
H265 with slow preset really works wonders for non-grainy film. AV1 is what you have to use when there is grain.
I would vote for running jellyfish off the NAS with powerful hardware Intel 8th Gen and up you should be fine. I personally don’t transcode video on my server only audio.
If I’m not sure about a video format working, I’ll manually re-encode the video with handbrake to h265 or av1. I also have used handbrake to create lower resolution copies of the same movies to reduce bandwidth needs when roaming on LTE.
Bottles was having the same issue. I was attempting to install SimTower off the CD copy I have and they all were having issues running. I believe it’s a permissions issue that I could figure out, but not sure. After spending hours i went down the distro route and it worked perfectly the first time. Was able to run the installation for the game and launch it from wine.