Jan.ai might be a good starting point or ollama? There’s https://tales.fromprod.com/2024/111/using-your-own-hardware-for-llms.html which has some guidance for using jan.ai for both server and client