Faulty peripheral power supply killed my server a little over a day ago.
120 gigs of MySQL data just wouldn’t come up - backup is far from recent. My fault. Most corrupted tables were of course in Friendica.
After much nail chewing everything now appears operational again with minimum(?) data loss.
Ironically, if I would have had more services running in docker I might not have experienced such a fundamental outage. Since docker services usually tend to spin up their exclusive database engine you kind of “roll the dice” as far as data corruption goes with each docker service individually. Thing is, I don’t really believe in bleeding CPU computation cycles by running redundant database services. And since many of my services are already very long-serving they’ve been set up from source and all funneled towards a single, central and busy database server - thus, if that one experiences sudden outage (for instance power failure) all kinds of corruption and despair can arise. ;-)
Guess I should really look into a small UPS and automated shutdown. On top of better backup management of course! Always the backups.
Why so much? A simple daily timer that runs mysqlcheck + mysqldump + a backup of that would be enough for most people. Using a solid OS (Debian) and a filesystem such as BTRFS, ZFS or XFS will also save you from power loss related corruption. Why do people go SO overkill with everything?
Keep it simple, less services, less processes, less overhead, pick well written software and script the rest. Everything works out way better if you don’t overcomplicate things.
at least weekly mysqlcheck + mysqlddump and some form of periodic off-machine storing of that is something I’ll surely take to heart after this lil’ fiasco ;-) sound advice, thank you!
Personally I’d go for as big a UPS as I could afford, but I serve some public-facing stuff from my homelab and I live in an area with outdated infrastructure and occasional ice storms. I currently have a small UPS and have been too tired/overwhelmed to set up automated shutdown yet. It’s not too hard though, I’ve done it before. And even without that in place, my small UPS has kept things going thru a bunch of <10 minute outages.
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !selfhosted@lemmy.world
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don’t control.
Rules:
Be civil: we’re here to support and learn from one another. Insults won’t be tolerated. Flame wars are frowned upon.
No spam posting.
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it’s not obvious why your post topic revolves around selfhosting, please include details to make it clear.
Don’t duplicate the full text of your blog or github here. Just post the link for folks to click.
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
No trolling.
Resources:
selfh.st Newsletter and index of selfhosted software and apps
removed by mod
reading
Sound a lot like too much Docker :D
Ironically, if I would have had more services running in docker I might not have experienced such a fundamental outage. Since docker services usually tend to spin up their exclusive database engine you kind of “roll the dice” as far as data corruption goes with each docker service individually. Thing is, I don’t really believe in bleeding CPU computation cycles by running redundant database services. And since many of my services are already very long-serving they’ve been set up from source and all funneled towards a single, central and busy database server - thus, if that one experiences sudden outage (for instance power failure) all kinds of corruption and despair can arise. ;-)
Guess I should really look into a small UPS and automated shutdown. On top of better backup management of course! Always the backups.
Why so much? A simple daily timer that runs mysqlcheck + mysqldump + a backup of that would be enough for most people. Using a solid OS (Debian) and a filesystem such as BTRFS, ZFS or XFS will also save you from power loss related corruption. Why do people go SO overkill with everything?
Keep it simple, less services, less processes, less overhead, pick well written software and script the rest. Everything works out way better if you don’t overcomplicate things.
at least weekly mysqlcheck + mysqlddump and some form of periodic off-machine storing of that is something I’ll surely take to heart after this lil’ fiasco ;-) sound advice, thank you!
Personally I’d go for as big a UPS as I could afford, but I serve some public-facing stuff from my homelab and I live in an area with outdated infrastructure and occasional ice storms. I currently have a small UPS and have been too tired/overwhelmed to set up automated shutdown yet. It’s not too hard though, I’ve done it before. And even without that in place, my small UPS has kept things going thru a bunch of <10 minute outages.
And if the power in your area sucks, the power conditioning even a good small UPS provides is invaluable.
Holy cow, 120 gigs in a database?
Also remember, you don’t want a backup solution, you want a restoration solution :-)
Hear hear! You don’t own a backup if you’ve never restored it before. Words to live by both in corporate and self-hosting environments.