So, I’m selfhosting immich, the issue is we tend to take a lot of pictures of the same scene/thing to later pick the best, and well, we can have 5~10 photos which are basically duplicates but not quite.
Some duplicate finding programs put those images at 95% or more similarity.
I’m wondering if there’s any way, probably at file system level, for the same images to be compressed together.
Maybe deduplication?
Have any of you guys handled a similar situation?
The problem is that OP is asking for something to automatically make decisions for him. Computers don’t make decisions, they follow instructions.
If you have 10 similar images and want a script to delete 9 you don’t want, then how would it know what to delete and keep?
If it doesn’t matter, or if you’ve already chosen the one out of the set you want, just go delete the rest. Easy.
As far as identifying similar images, this is high school level programming at best with a CV model. You just run a pass through something with Yolo or whatever and have it output similarities in confidence of a set of images. The problem is you need a source image to compare it to. If you’re running through thousands of files comprising dozens or hundreds of sets of similar images, you need a source for comparison.
computers make decisions all the time. For example, how to route my packets from my instance to your instance. Classification functions are well understood in computer science in general, and, while stochastic, can be constructed to be arbitrarily precise.
Human facial detection has been at 99% accuracy since the 90s and OPs task I’d likely a lot easier since we can exploit time and location proximity data and know in advance that 10 pictures taken of Alice or Bob at one single party are probably a lot less variant than 10 pictures taken in different contexts over many years.
What OP is asking to do isn’t at all impossible-- I’m just not sure you’ll save any money on power and GPU time compared to buying another HDD.
Everything you just described is instruction.
Everything from an input path and desired result can be tracked and followed to a conclusory instruction. That is not decision making.
Agree to disagree. Something makes a decision about how to classify the images and it’s certainly not the person writing 10 lines of code. I’d be interested in having a good faith discussion, but repeating a personal opinion isn’t really that. I suspect this is more of a metaphysics argument than anything and I don’t really care to spend more time on it.
I hope you have a wonderful day, even if we disagree.
You seem very upset, so I hate to inform you that neither one of those are peer reviewed sources and that they are simplifying things.
“Learning” is definitely something a machine can do and then they can use that experience to coordinate actions based on data that is inaccesible to the programmer. If that’s not “making a decision”, then we aren’t speaking the same language. Call it what you want and argue with the entire published field or AI, I guess. That’s certainly an option, but generally I find it useful for words to mean things without getting too pedantic.
“Pedantic Asshole tries the whole ‘You seem upset’ but on the Internet and proceeds to try and explain their way out of being embarrassed about being wrong, so throws some idiotic semantics into a further argument while wrong.”
Great headline.
Computers also don’t learn, or change state. Apparently you didn’t read the CS101 link after all.
Also, another newsflash is coming in here, one sec:
“Textbooks and course plans written by educators and professors in the fields they are experts in are not ‘peer reviewed’ and worded for your amusement, dipshit.”
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !selfhosted@lemmy.world
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don’t control.
Rules:
Be civil: we’re here to support and learn from one another. Insults won’t be tolerated. Flame wars are frowned upon.
No spam posting.
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it’s not obvious why your post topic revolves around selfhosting, please include details to make it clear.
Don’t duplicate the full text of your blog or github here. Just post the link for folks to click.
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
The problem is that OP is asking for something to automatically make decisions for him. Computers don’t make decisions, they follow instructions.
If you have 10 similar images and want a script to delete 9 you don’t want, then how would it know what to delete and keep?
If it doesn’t matter, or if you’ve already chosen the one out of the set you want, just go delete the rest. Easy.
As far as identifying similar images, this is high school level programming at best with a CV model. You just run a pass through something with Yolo or whatever and have it output similarities in confidence of a set of images. The problem is you need a source image to compare it to. If you’re running through thousands of files comprising dozens or hundreds of sets of similar images, you need a source for comparison.
OP didn’t want to delete anything, but to compress them all, exploiting the fact they’re similar to gain efficiency.
Using that as an example. Same premise.
No, not really.
The computer is not asked to make decisions like “pick the best image”. The computer is asked to optimize, like with lossless compression.
That’s not what he’s asking at all
yes, they are. reread the post, I just did so and I’m still confident
computers make decisions all the time. For example, how to route my packets from my instance to your instance. Classification functions are well understood in computer science in general, and, while stochastic, can be constructed to be arbitrarily precise.
https://en.wikipedia.org/wiki/Probably_approximately_correct_learning?wprov=sfla1
Human facial detection has been at 99% accuracy since the 90s and OPs task I’d likely a lot easier since we can exploit time and location proximity data and know in advance that 10 pictures taken of Alice or Bob at one single party are probably a lot less variant than 10 pictures taken in different contexts over many years.
What OP is asking to do isn’t at all impossible-- I’m just not sure you’ll save any money on power and GPU time compared to buying another HDD.
Everything you just described is instruction. Everything from an input path and desired result can be tracked and followed to a conclusory instruction. That is not decision making.
Again. Computers do not make decisions.
Agree to disagree. Something makes a decision about how to classify the images and it’s certainly not the person writing 10 lines of code. I’d be interested in having a good faith discussion, but repeating a personal opinion isn’t really that. I suspect this is more of a metaphysics argument than anything and I don’t really care to spend more time on it.
I hope you have a wonderful day, even if we disagree.
It’s Boolean. This isn’t an opinion, it’s a fact. Feel free to get informed though.
Then it should be easy to find peer reviewed sources that support that claim.
I found it incredibly easy to find countless articles suggesting that your Boolean is false. Weird hill to die on. Have a good day.
https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=computer+decision+fairness&oq=computer+decison
LITERALLY from a “baby’s first 'puter” course: https://highered.mheducation.com/sites/007256380x/student_view0/part1/chapter2/reading_selection_quiz.html#:~:text=Although computers may appear to,humans who programmed the computers.&text=Humans are smarter than computers.&text=It is extremely time-consuming,that it makes correct decisions.
Or if you need something more ELI5: https://www.reddit.com/r/explainlikeimfive/comments/160lwe8/eli5_where_exactly_do_computers_make_decisions/
You seem very upset, so I hate to inform you that neither one of those are peer reviewed sources and that they are simplifying things.
“Learning” is definitely something a machine can do and then they can use that experience to coordinate actions based on data that is inaccesible to the programmer. If that’s not “making a decision”, then we aren’t speaking the same language. Call it what you want and argue with the entire published field or AI, I guess. That’s certainly an option, but generally I find it useful for words to mean things without getting too pedantic.
🙄
“Pedantic Asshole tries the whole ‘You seem upset’ but on the Internet and proceeds to try and explain their way out of being embarrassed about being wrong, so throws some idiotic semantics into a further argument while wrong.”
Great headline.
Computers also don’t learn, or change state. Apparently you didn’t read the CS101 link after all.
Also, another newsflash is coming in here, one sec:
“Textbooks and course plans written by educators and professors in the fields they are experts in are not ‘peer reviewed’ and worded for your amusement, dipshit.”
Whoa, that was a big one.