Yes, but it’s a bit more than that. All models are produced using a process known as neural network quantization, which optimizes them to be able to run on a CPU. This, plus appropriate backend code written in C, means GPT4All is quite efficient and needs only 4-8GB of RAM (depending on the model) and a CPU with AVX/AVX2 support.
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !technology@beehaw.org
A nice place to discuss rumors, happenings, innovations, and challenges in the technology sphere. We also welcome discussions on the intersections of technology and society. If it’s technological news or discussion of technology, it probably belongs here.
Remember the overriding ethos on Beehaw: Be(e) Nice. Each user you encounter here is a person, and should be treated with kindness (even if they’re wrong, or use a Linux distro you don’t like). Personal attacks will not be tolerated.
Yeah, those concerns are valid. Not running on your machine and not FOSS.
Are there any viable alternatives?
Check out Meta’s LLaMa 2. Not FOSS, but source available and self hostable.
GPT4all, it’s open source and you can run it on your own machine.
How is this able to run without a gpu? Is it that the models are small enough so that only a cpu is needed?
Yes, but it’s a bit more than that. All models are produced using a process known as neural network quantization, which optimizes them to be able to run on a CPU. This, plus appropriate backend code written in C, means GPT4All is quite efficient and needs only 4-8GB of RAM (depending on the model) and a CPU with AVX/AVX2 support.