I’m looking for a duplicate/similarity checker against a custom set of documents. This is possibly like a plagiarism checker, but with a custom reference (instead of everything that exists).
But I could not find a solution that can be selfhosted, and have some simple UI and capabilities like Turnitin. Any suggestions?
Maybe look into self hosted llm. I’ve used two recently to help analyze a large volume of books, by ingesting them into the data set, then chat with the bot for specifics. It worked pretty well but there are some limitations, such as token length and general hallucinations. But they both use citations of the data they used, so it helps to check their work.
Both have simple selfhosted webui or actual applications. So, in theory you should be able to ingest data and then see if it then matches any submissions you submit later. But I have not really tried it for this though, so it might not work.
Thanks for the suggestions! I’m actually also looking into llamaindex for more conceptual comparison, though didn’t get to building an app yet.
Any general suggestions for locally hosted LLM with llamaindex by the way? I’m also running into some issues with hallucination. I’m using Ollama with llama2-13b and bge-large-en-v1.5 embedding model.
Anyway, aside from conceptual comparison, I’m also looking for more literal comparison, AFAIK, the choice of embedding model will affect how the similarity will be defined. Most of the current LLM embedding models are usually abstract and the similarity will be conceptual, like “I have 3 large dogs” and “There are three canine that I own” will probably be very similar. Do you know which choice of embedding model I should choose to have it more literal comparison?
That aside, like you indicated, there are some issues. One of it involves length. I hope to find something that can build up to find similar paragraphs iteratively from similar sentences. I can take a stab at coding it up but was just wondering if there are some similar frameworks out there already that I can model after.
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !selfhosted@lemmy.world
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don’t control.
Rules:
Be civil: we’re here to support and learn from one another. Insults won’t be tolerated. Flame wars are frowned upon.
No spam posting.
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it’s not obvious why your post topic revolves around selfhosting, please include details to make it clear.
Don’t duplicate the full text of your blog or github here. Just post the link for folks to click.
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
No trolling.
Resources:
selfh.st Newsletter and index of selfhosted software and apps
This is very specific and niche without a good reason to exist, so I doubt this exists.
Maybe look into self hosted llm. I’ve used two recently to help analyze a large volume of books, by ingesting them into the data set, then chat with the bot for specifics. It worked pretty well but there are some limitations, such as token length and general hallucinations. But they both use citations of the data they used, so it helps to check their work.
PrivateGPT - https://github.com/imartinez/privateGPT
Llamaindex - https://github.com/run-llama/llama_index
Both have simple selfhosted webui or actual applications. So, in theory you should be able to ingest data and then see if it then matches any submissions you submit later. But I have not really tried it for this though, so it might not work.
Thanks for the suggestions! I’m actually also looking into llamaindex for more conceptual comparison, though didn’t get to building an app yet.
Any general suggestions for locally hosted LLM with llamaindex by the way? I’m also running into some issues with hallucination. I’m using Ollama with llama2-13b and bge-large-en-v1.5 embedding model.
Anyway, aside from conceptual comparison, I’m also looking for more literal comparison, AFAIK, the choice of embedding model will affect how the similarity will be defined. Most of the current LLM embedding models are usually abstract and the similarity will be conceptual, like “I have 3 large dogs” and “There are three canine that I own” will probably be very similar. Do you know which choice of embedding model I should choose to have it more literal comparison?
That aside, like you indicated, there are some issues. One of it involves length. I hope to find something that can build up to find similar paragraphs iteratively from similar sentences. I can take a stab at coding it up but was just wondering if there are some similar frameworks out there already that I can model after.
Grep. If you have the sources digitally and local, you can just run a search for a string.