Hi guys! I think I’m over Joplin. Don’t get me wrong, it’s simple, it works, but… why is it Postgres db…. I have the server on a small box with like 250 GB of space and backing it up with kopia to Backblaze with free 10 GB, so I’m a bit storage cautious.
With each snapshot, it seems like a good chunk of the database changes, even when I don’t use Joplin that day. That results in kopia backing up those changed files, and backups keep growing. Right now the Joplin database is like 200Mb, BUT when I export the notes from the app… all of them weigh 2Mb… including images. Yes there is versioning of notes, but they shouldn’t be that big after one-two months lol.
I know I know, I’m being a bit weird about it, but I’m getting daily notifications about backups and I see how they grow each day.
Anyway, do you have any alternatives that have an app on iOS and on Linux? Or should I just use Apple Notes in the browser? Thanks
EDIT: The answer was easier than I thought. Just don’t back it up, it’s synced which means each device has a copy of it anyway so there is not really need for it, thanks @vvv@programming.dev !
Use Nextcloud AIO mastercontainer, set up joplin with Nextcloud sync (which is webdav). Use the builtin backup function in Nextcloud AIO container to backup nextcloud and the files it contains that are your joplin notes (and anything else you use nextcloud for).
I even use Nextcloud for its Gpoddersync app to keep my podcast subs/progress from Antennapod.
Joplin server also will use a file based storage instead of the db. And db then is only used for users
I use joplin as i share notes and collaborate with my wife. Obsidian etc dont do that. Id love to be on obsidian as i really like it but sharing notebooks is mandatory
I would’ve NEVER ever moved to Joplin if it wasn’t able to sync with WebDAV. I’m not into having a special daemon running on a server for that task, makes zero sense.
@jaykay@TCB13 Hetzner has low cost storage servers with SMB, SFTP & WebDAV access. Not quite self hosted but a hell of a lot better than [insert proprietary privacy invasive mega corp]
Please don’t follow joplins folder/notes view. It’s so stupid that folders and notes are in different panels on the left. Just make it a normal list.
I like joplin for its simplicity. No bells and whistles like obsidian.
Docker container would be awesome, but I don’t care for Google drive personally :) If you make the notes folder a volume I can bind to that would be great, as long as they’re normal files haha
Decent mobile app (more than happy to pay for this if it’s a one time fee)
Bonus for a OneNote / Evernote style Android widget. Being able to scroll through and quickly select from my most recent notes in the OneNote widget is really helpful.
WYSIWYG editor on mobile and desktop (why in God’s name does every Foss notes app insist I use a markdown language?) with bullet points, numbered lists, bold, italic, underline, strikethrough, and headings.
Checklists (as in, ability to add checkboxes to notes)
Ability to create an arbitrarily deep folder structure
Tags would be nice
Import from popular apps like OneNote, Evernote, or Joplin is basically essential at this point. A lot of us have way too fucking many notes to move by hand.
Can you not just backup the pg txn logs (with periodic full backups, purged in accordance with your needs?). That’s a much safer way to approach DBs anyway.
(exclude the online db files from your file system replication)
But jokes aside, it is mostly about syncing notes for the selfhosting part. You either go with the official offer, no self-hosting and costs money, or you use a community plug-in, self-hosted, or you use a third program like syncthing, selfhosted.
Syncthing is the way, I had tried setting on nextcloud but never could get it to store how I wanted, but syncthinf was ridiculously easy and should work for anything that uses a folder
There is a plugin for obsidian to work with syncthing, but it seems to still be in development, implementing through the app and selecting the folders also gave me a reason for syncing my camera as well, and was super easy, no portfowarding or anything required
I second obsidian. I was on the verge to jump onto logseq, but found its way of handling notes to be… different. I also felt a dislike of anytype where I don’t really have control over my notes. Obsidian clicked with me from the start and felt right. So I went with it, even though it’s not FOSS (which is usually a hard requirement from me).
Not every one stores the files as plain text files in markdown format like Obsidian. Logseq does I believe, but Joplin stores it all in database files which require an export should you decide to leave that app in favor of a other. With Obsidian you just point the new app at the folders full of .md files and away you go. That was the main selling point for me.
I don’t know where you’re getting that from. Here is my Joplin folder on my NC server, stuffed with md files from my notes. There are some database driven references in them if you do things like add pictures, and obviously the filename is a UID format, but it’s markdown all the way, baby.
Have you looked at the contents of those md files? In addition to creating its own hexadecimal file name, it appends the text with a bunch of metadata info. If you were to then take that folder of notes to any other markdown editor like Obsidian, it would be a mess to organize. That is why I’m a stickler for file format agnosticism. There is no vendor lock in and more importantly, no manipulation of the text filenames or contents.
Screenshot of my phone copy of the Obsidian vault directory as an example:
It is not a unique feature. But as a non-FOSS program its notes are not hidden behind proprietary filesystems, so any time you want you can still switch if they go in a direction thr user does not like.
I also switched from Joplin to Obsidian after about half a year. There’s an open-source plugin that lets you self-host a syncing server.
What I found paradoxical is how easy it is to mod and write plugins for Obsidian compared to Joplin. I would’ve thought that modifying the open-source candidate would’ve been easier, but nope.
Why do you need to back up that server data? The great thing about joplin, is that the full content of your notes (and history) is distributed, like a git repo. As long as you have one device left with your notes, everything else can be bootstrapped from there. If your sync server burns down, start a new one and sync your notes to it again.
Genuine question - doesn’t this leave you open to loss of data from database corruption or an app failure (or human error, accidentally deleting a bunch of pages, for example)?
I’ve used “sync as backup” a lot, and run into these kinds of issues (it’s my current OneNote “backup” strategy). I’m just not familiar enough with Joplin to know what risks this exposes.
Did you know that you can use Joplin on a standard webdav server? Basically it just takes up the space of the data itself.
I have it on a Caddy server and works like q charm synching between Windows and Android client
I think you need to learn more about how databases work. They don’t typically reclaim deleted space automatically for performance reasons. Databases like to write to a single large file they can then index into. Re-writing those files is expensive so left to the DBA (you) to determine when it should be done.
And how are you backing up the database? Just backing up /var/lib/postgres? Or are you doing a pg_dump? If the former then it’s possible your backups won’t be coherent if you haven’t stopped your database and it will contain that full history of deleted stuff. pg_dump would give you just the current data in a way that will apply properly to a new database should you need to restore
You can also consider your backup retention policy. How many backups do you need for how long?
You are right, I should. They are a bit more complicated than I anticipated, and apparently I’m doing everything wrong, haha. I have backups set up to go 2 years back, but I’m checking backblaze occasionally to check, so it shouldn’t be an issue. I have two months so far lol Thanks for the write-up :)
Setup backup hooks with velero and kopia on a HA postres cluster this week. Biggest DB is Lemmy and that shrinks by a factor of 10 using pgdunp with custom archive. Dumping is 100% the way to go!
Similarly I should do this for my sqlite applications, it looks like kopia can’t do incremental backups with them and thinking about it, it makes sebse, likely sane reasons you mentioned.
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !selfhosted@lemmy.world
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don’t control.
Rules:
Be civil: we’re here to support and learn from one another. Insults won’t be tolerated. Flame wars are frowned upon.
No spam posting.
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it’s not obvious why your post topic revolves around selfhosting, please include details to make it clear.
Don’t duplicate the full text of your blog or github here. Just post the link for folks to click.
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
No trolling.
Resources:
selfh.st Newsletter and index of selfhosted software and apps
Use Nextcloud AIO mastercontainer, set up joplin with Nextcloud sync (which is webdav). Use the builtin backup function in Nextcloud AIO container to backup nextcloud and the files it contains that are your joplin notes (and anything else you use nextcloud for).
I even use Nextcloud for its Gpoddersync app to keep my podcast subs/progress from Antennapod.
Joplin server also will use a file based storage instead of the db. And db then is only used for users
I use joplin as i share notes and collaborate with my wife. Obsidian etc dont do that. Id love to be on obsidian as i really like it but sharing notebooks is mandatory
This is why I’m moving to Joplin too
Why on earth are you using that? Just use WebDAV, you’ll only be required to have some WebDAV server such as Nginx and it will sync GB of notes without issues. https://joplinapp.org/help/apps/sync/webdav/ https://medium.com/learn-or-die/build-a-webdav-server-with-nginx-8660a7a7311
I would’ve NEVER ever moved to Joplin if it wasn’t able to sync with WebDAV. I’m not into having a special daemon running on a server for that task, makes zero sense.
I need to look into webDAV then :D
@jaykay @TCB13 Hetzner has low cost storage servers with SMB, SFTP & WebDAV access. Not quite self hosted but a hell of a lot better than [insert proprietary privacy invasive mega corp]
Yes you do ahaha
This is the way.
It works great with my self-hosted NextCloud!
I’m going to start developing my own alternative, is there any features that you really need/want?
I was planning on making this self hosted via docker with the option to save to Google drive.
https://github.com/ssebs/PadPal-Server
Please don’t follow joplins folder/notes view. It’s so stupid that folders and notes are in different panels on the left. Just make it a normal list.
I like joplin for its simplicity. No bells and whistles like obsidian.
Docker container would be awesome, but I don’t care for Google drive personally :) If you make the notes folder a volume I can bind to that would be great, as long as they’re normal files haha
@DichotoDeezNutz @jaykay Saving to a network drive _OTHER_THAN_ proprietary US mega corporations would be essential for me (eg SFTP, WEBDAV)
A mobile client is vital. I use mobile devices 95%
Here’s what I would be looking for;
Most of those are on the to-do list! I definitely like the Google keep style widget but want better UX when typing out bullet lists.
Thanks for the suggestions
Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I’ve seen in this thread:
3 acronyms in this thread; the most compressed thread commented on today has 10 acronyms.
[Thread #413 for this sub, first seen 8th Jan 2024, 22:35] [FAQ] [Full list] [Contact] [Source code]
Can you not just backup the pg txn logs (with periodic full backups, purged in accordance with your needs?). That’s a much safer way to approach DBs anyway.
(exclude the online db files from your file system replication)
Have you looked into either Obsidian or Logseq?
Obsidian is not open source, but uses Markdown for notes just like Logseq. Very popular overall.
Yeah, I’ve been on Obsidian before, but self-hosted syncing on iOS is a bit finicky.
I’ve heard good things about Logseq, but it’s certainly a waaay different approach to notes. I’ll have to read more about it. Thanks :)
How do you self-host Obsidian?
That’s the near thing about it… you don’t.
But jokes aside, it is mostly about syncing notes for the selfhosting part. You either go with the official offer, no self-hosting and costs money, or you use a community plug-in, self-hosted, or you use a third program like syncthing, selfhosted.
Yes that’s the one I was asking about.
Syncthing is the way, I had tried setting on nextcloud but never could get it to store how I wanted, but syncthinf was ridiculously easy and should work for anything that uses a folder
I’ll have to look into that. It doesn’t work like Joplin where I can just connect it to the same remote backup within the app, across devices?
There is a plugin for obsidian to work with syncthing, but it seems to still be in development, implementing through the app and selecting the folders also gave me a reason for syncing my camera as well, and was super easy, no portfowarding or anything required
I second obsidian. I was on the verge to jump onto logseq, but found its way of handling notes to be… different. I also felt a dislike of anytype where I don’t really have control over my notes. Obsidian clicked with me from the start and felt right. So I went with it, even though it’s not FOSS (which is usually a hard requirement from me).
Literally every note app uses markdown. I’m not sure why people point at that for Obsidian like it’s a unique feature.
Not every one stores the files as plain text files in markdown format like Obsidian. Logseq does I believe, but Joplin stores it all in database files which require an export should you decide to leave that app in favor of a other. With Obsidian you just point the new app at the folders full of .md files and away you go. That was the main selling point for me.
I don’t know where you’re getting that from. Here is my Joplin folder on my NC server, stuffed with md files from my notes. There are some database driven references in them if you do things like add pictures, and obviously the filename is a UID format, but it’s markdown all the way, baby.
Have you looked at the contents of those md files? In addition to creating its own hexadecimal file name, it appends the text with a bunch of metadata info. If you were to then take that folder of notes to any other markdown editor like Obsidian, it would be a mess to organize. That is why I’m a stickler for file format agnosticism. There is no vendor lock in and more importantly, no manipulation of the text filenames or contents.
Screenshot of my phone copy of the Obsidian vault directory as an example:
It is not a unique feature. But as a non-FOSS program its notes are not hidden behind proprietary filesystems, so any time you want you can still switch if they go in a direction thr user does not like.
I also switched from Joplin to Obsidian after about half a year. There’s an open-source plugin that lets you self-host a syncing server.
What I found paradoxical is how easy it is to mod and write plugins for Obsidian compared to Joplin. I would’ve thought that modifying the open-source candidate would’ve been easier, but nope.
Connect it to a DAV server and you dont have to learn a new software.
Gotta learn about that DAV thing everyone is talking about ig haha
removed by mod
If you mean the ‘export’ function, it’s not really the same as I’d have to do it manually every time
Why do you need to back up that server data? The great thing about joplin, is that the full content of your notes (and history) is distributed, like a git repo. As long as you have one device left with your notes, everything else can be bootstrapped from there. If your sync server burns down, start a new one and sync your notes to it again.
I’d rather backup a stack of plain text files, personally. So Obsidian for me.
Genuine question - doesn’t this leave you open to loss of data from database corruption or an app failure (or human error, accidentally deleting a bunch of pages, for example)?
I’ve used “sync as backup” a lot, and run into these kinds of issues (it’s my current OneNote “backup” strategy). I’m just not familiar enough with Joplin to know what risks this exposes.
There is a fail-safe switch in settings preventing the deletion of local copy if the remote is empty
Well that’s something anyway, though I wouldn’t rely on it - sometimes things happen.
I’m sure there’s a way to do proper backups of database data (e.g. Incrementals, full, etc), that would get the changes.
That’s always handled by other teams for systems I’ve deployed, so I’m joy familiar with current approaches.
True, should’ve thought of that. Well, at least this gives me a chance to explore and learn alternatives :P
Did you know that you can use Joplin on a standard webdav server? Basically it just takes up the space of the data itself. I have it on a Caddy server and works like q charm synching between Windows and Android client
Came here to say just that. The WebDAV synchronization target is great.
Yeah, I’m yet to play around with WebDAV or learning what it actually is haha Will look into it, thanks :)
Simplenote, Notesnook, Obsidian.
Obsidian sync isn’t free, and it’s easy to violate their license if you mix work and personal notes.
I think Joplin tends to be better than most. If Obsidian was licensed and charged differently I might change my mind.
Logseq
Free and open-source.
I think you need to learn more about how databases work. They don’t typically reclaim deleted space automatically for performance reasons. Databases like to write to a single large file they can then index into. Re-writing those files is expensive so left to the DBA (you) to determine when it should be done.
And how are you backing up the database? Just backing up /var/lib/postgres? Or are you doing a pg_dump? If the former then it’s possible your backups won’t be coherent if you haven’t stopped your database and it will contain that full history of deleted stuff. pg_dump would give you just the current data in a way that will apply properly to a new database should you need to restore
You can also consider your backup retention policy. How many backups do you need for how long?
You are right, I should. They are a bit more complicated than I anticipated, and apparently I’m doing everything wrong, haha. I have backups set up to go 2 years back, but I’m checking backblaze occasionally to check, so it shouldn’t be an issue. I have two months so far lol Thanks for the write-up :)
Setup backup hooks with velero and kopia on a HA postres cluster this week. Biggest DB is Lemmy and that shrinks by a factor of 10 using pgdunp with custom archive. Dumping is 100% the way to go!
Similarly I should do this for my sqlite applications, it looks like kopia can’t do incremental backups with them and thinking about it, it makes sebse, likely sane reasons you mentioned.
Seconding what others have already said. You should ABSOLUTELY NOT directly back up /var/lib/postgresql if that’s what you’re doing right now. Instead, use pg_dump: https://www.postgresql.org/docs/current/backup-dump.html
This should also give you smaller and probably more compressible backup sizes.