My setup is smaller, but when my venerable old router died about a year ago, I acquired an Asus TUF-AX3000_V2 where I installed FreshTomato. One can login via SSH and dump all settings for backup. Likewise, individual or all settings can be done on the command line instead of the GUI. I have a script on my computer that reads CSV files with MAC addresses and more to apply changes in an automated way.

Yes, XMPP with proper TLS on the server side and Conversations or one of its forks (preferably fetched from F-Droid) using OMEMO encryption should be good enough. If you are brave or paranoid, give Tox a try: https://tox.chat/

Maybe the first question is what your budget is, both regarding money and time. For example, you could buy a pre-configured NAS from Synology or QNAP, which requires less technical skills but more money, or a home-made solution reusing used components (but fresh disks for reliability). Depending on your electricity costs, you may want to choose a low-power solution or something which you power off when not used. For storage, maybe a three-disk RAID5 is a good compromise. For backups, plain S3 cloud storage encrypted via restic is a good idea.

#Peertube got already mentioned, but just serving video files may already suffice. Modern webbrowsers are capable of playing videos. Some tweaking of parameters may be necessary when encoding them. Also, no frills such as dynamic adoption of bitrate/quality or high-level stuff like commenting, likes, or subtitles.