Issue Description: I have been having this issue with my raspberry pi running dietPi where it seems to lock up and I cannot SSH / access any of the services on it. The interesting part is that the interface seems to be up and I can still ping it on the local network but shows no video output. usually I get about 3-6 days before the issue appears again.
I am not really sure where to start to diagnose this issue. Any help would be appreciated!
Things Tried:
Reduced operating temp by getting a fan. I want to say this improved the length in between this issue appearing but don’t really have any hard evidence.
uninstalled unused services
Limited active torrents in QbitTorrent
EDIT: Small thing to mention is that the CPU load is usually really high - like not uncommon for the load to be between 8-10 but I have seen it as high as 24.
Temporary fix:
Power cycle - everything comes up again in less than a minute.
This is likely an issue with your SD card…Pi’s eat them up, even the good ones that supposedly can handle it get chewed up pretty quickly. It really sucks that they haven’t transitioned to emmc but insist on using SD cards.
I’ve had this happen when I had too many USB devices plugged into it. It was having power underrun, and acting unresponsive while trying to compensate. I solved it with a powered USB hub.
Edit: I’ve had pairing it with an off brand power brick cause the same problem, too. Apparently the 3 and later Pi really want better power quality regulation, and some of the cheapo bricks I had lying around - while providing the right Volts and Amps, didn’t control the variation well enough for the modern Pi computer.
That’s the weird part is that I don’t have any USB devices attached. I have Ethernet, power cable, and the fan on the case has pins going to some headers.
The case did come with another power supply so maybe I’ll try that and see if anything changes.
@AverageGoob I have this issue with one of my hosts as well. It appears to be a problem with the micro SD card. Same card, different pi = same problem. I’m currently working around it with a watchdog but will need to replace the card soon.
Are you running your OS from USB or from a micro SD card?
I upgraded to the Pi4 but I use this case. It has a daughter board that lets me use an m.2 SATA SSD over USB. But any USB to SATA adapter should work fine
You should get a scsi enabled adapter though, otherwise you may have to disable it in the kernel boot settings. And if you forget that it will run at like kbit/s.
@a_fancy_kiwi I agree, same here. This is the last pi that’s running off an SD card with services that do “significant” disk I/O. I have a few zeros that only really write to the card for OS updates. Their job is to collect data and send it via the network. I haven’t had issues with that kind of workload using micro SD cards.
Edit: For Pis with write workloads I’m using basic USB3 SSDs. Didn’t have good results with USB sticks though.
@AverageGoob The watchdog saves me from rebooting the host manually, but at the risk of data loss (though not more than a locked up SD card). I configured a custom script that writes to a file, when the card has problems, the watchdog kicks in. To keep the script from stressing the card even more, the script only writes to the file every few minutes.
As you said it’s only a workaround. I’ll move the stuff on the problematic host to a VM with SSD shortly.
@AverageGoob@lemmy.world Consider running something on it to monitor temps/memory/etc over time and push them to a separate device as it does so - that way you can measure things leading up to a crash, and know when it crashes (due to it suddenly stopping its reporting)
Have had this issue myself asking with other DD card related issues.
I can’t understand why the pi foundation persist with using SD as the only physically practical storage option.
They’re looking post the point of needing a way to snap on reliable EMMC storage, as a default, in a way that doesn’t leave a cable or something permanently plugged into a USB port.
Sure, USB is a fine option, but I hate that it’s only an option and not a designed default.
Most of us only need 8GB or so for the OS, 8GB or good quality durable EMMC should hardly cost anything.
Other tiny computers and even economy notebooks and Chromebooks already use this.
Yeah, I had the same issue. Sometimes it was the SD card, sometimes the network interface (not your case obviously), sometimes things connected to USB, sometimes it was running hot… I gave up and now I just run everything on an older Slimbook Zero. Yes, power consumption is higher (still pretty low) but so is stability.
Yeah it seems a lot of people are saying the SD card is the issue which wouldn’t surprise me. I do have some spare space on my proxmox server but it would just be a huge pain in the dick to move everything… But it’s looking like I may need to sadly.
Consider using a USB3 SSD as your boot drive if you want long term usage from your pi. The SD card is prone to failure relatively quickly on Raspian and is even worse on OSes that aren’t optimized for the PI directly.
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !selfhosted@lemmy.world
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don’t control.
Rules:
Be civil: we’re here to support and learn from one another. Insults won’t be tolerated. Flame wars are frowned upon.
No spam posting.
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it’s not obvious why your post topic revolves around selfhosting, please include details to make it clear.
Don’t duplicate the full text of your blog or github here. Just post the link for folks to click.
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
No trolling.
Resources:
selfh.st Newsletter and index of selfhosted software and apps
This is likely an issue with your SD card…Pi’s eat them up, even the good ones that supposedly can handle it get chewed up pretty quickly. It really sucks that they haven’t transitioned to emmc but insist on using SD cards.
Yeah seems like a weird choice to have a default storage type that is known in the community to be unreliable.
I’ve had this happen when I had too many USB devices plugged into it. It was having power underrun, and acting unresponsive while trying to compensate. I solved it with a powered USB hub.
Edit: I’ve had pairing it with an off brand power brick cause the same problem, too. Apparently the 3 and later Pi really want better power quality regulation, and some of the cheapo bricks I had lying around - while providing the right Volts and Amps, didn’t control the variation well enough for the modern Pi computer.
That’s the weird part is that I don’t have any USB devices attached. I have Ethernet, power cable, and the fan on the case has pins going to some headers.
The case did come with another power supply so maybe I’ll try that and see if anything changes.
@AverageGoob I have this issue with one of my hosts as well. It appears to be a problem with the micro SD card. Same card, different pi = same problem. I’m currently working around it with a watchdog but will need to replace the card soon.
Are you running your OS from USB or from a micro SD card?
I’d bet $1 it’s the SD card. My 3B+ used to have the same problem. Been running pis off some sort of SSD ever since, no issues.
Id be willing to try this. How do you have it connected? Just using an external USB attached one?
I upgraded to the Pi4 but I use this case. It has a daughter board that lets me use an m.2 SATA SSD over USB. But any USB to SATA adapter should work fine
You should get a scsi enabled adapter though, otherwise you may have to disable it in the kernel boot settings. And if you forget that it will run at like kbit/s.
@a_fancy_kiwi I agree, same here. This is the last pi that’s running off an SD card with services that do “significant” disk I/O. I have a few zeros that only really write to the card for OS updates. Their job is to collect data and send it via the network. I haven’t had issues with that kind of workload using micro SD cards.
Edit: For Pis with write workloads I’m using basic USB3 SSDs. Didn’t have good results with USB sticks though.
Pi 3B has dedicated bus for SD card but ethernet and usb share bandwidth. Enable zram, disable all swap and keep using sd card.
I am running it from an SD card. Did setting up the watchdog ultimately work for you? I did come across a watchdog as a possible workaround.
@AverageGoob The watchdog saves me from rebooting the host manually, but at the risk of data loss (though not more than a locked up SD card). I configured a custom script that writes to a file, when the card has problems, the watchdog kicks in. To keep the script from stressing the card even more, the script only writes to the file every few minutes.
As you said it’s only a workaround. I’ll move the stuff on the problematic host to a VM with SSD shortly.
Not sure if the rpi3 can use the 64bit version, or if it’s possible for it use an SSD like the the 4 can?
Pi 3b+ can you just need to enable it, look it up. Tried and tested.
@AverageGoob@lemmy.world Consider running something on it to monitor temps/memory/etc over time and push them to a separate device as it does so - that way you can measure things leading up to a crash, and know when it crashes (due to it suddenly stopping its reporting)
That is a good idea! Do you have any recommendations for remote logging?
Have had this issue myself asking with other DD card related issues.
I can’t understand why the pi foundation persist with using SD as the only physically practical storage option.
They’re looking post the point of needing a way to snap on reliable EMMC storage, as a default, in a way that doesn’t leave a cable or something permanently plugged into a USB port.
Sure, USB is a fine option, but I hate that it’s only an option and not a designed default.
Most of us only need 8GB or so for the OS, 8GB or good quality durable EMMC should hardly cost anything.
Other tiny computers and even economy notebooks and Chromebooks already use this.
Yeah, I had the same issue. Sometimes it was the SD card, sometimes the network interface (not your case obviously), sometimes things connected to USB, sometimes it was running hot… I gave up and now I just run everything on an older Slimbook Zero. Yes, power consumption is higher (still pretty low) but so is stability.
Yeah it seems a lot of people are saying the SD card is the issue which wouldn’t surprise me. I do have some spare space on my proxmox server but it would just be a huge pain in the dick to move everything… But it’s looking like I may need to sadly.
Consider using a USB3 SSD as your boot drive if you want long term usage from your pi. The SD card is prone to failure relatively quickly on Raspian and is even worse on OSes that aren’t optimized for the PI directly.
deleted by creator