I guess your OPNSense rule from Edit3 is not working because the source is not your mailu instance, because connections are initiated from the outside and mailu only answers (TCP ACK). So you have asynchornous routing.
You may get this working if you set the “reply-to” option to the wg gateway on the firewall rule that allows VPS -> wg -> mailu traffic.
However there is a much cleaner solution using the PROXY protocol, which mailu seems to support: https://mailu.io/master/reverse.html
They are using traefik, but nginx also supports the PROXY protocol.
It matters only if “the docker hosts external IP” your dns resolves is a public IP. In that case packets travel to the router which needs to map/send them back to the docker hosts LAN IP (NAT-Reflection). With cgnat this would need to be enabled on the carrier side, where you set up the port forwarding. If that’s not possible, split-DNS may be an alternative.
If “the docker hosts external IP” is actually your docker hosts LAN IP, all of that is irrelevant. Split-DNS would accomplish that.
Do you want to supplement your usenet sources? If so, what type of content/language are you looking for? Do you have any proof like screenshots of the old times left?
Things that should be answered before anyone would consider giving away an invite, because that is also a liability.