I bought an Optiplex 5040, with an i5-6500TE, and 8 GB DDR3L RAM.
When I bought it, I installed Fedora Server on it. It got stuck every few days but I could never see the error. The services just stopped working, I couldn’t ssh into it, and connecting it to a monitor showed a black screen.
So, I thought let’s install Ubuntu Server, maybe Fedora isn’t compatible with all of its hardware. The same thing is happening, now, but I can see this error. Even when there’s nothing installed on it, no containers, nothing other than base packages, this happens.
I have updated the bios. I have tried setting nouveau.modeset=0 in the grub config file. I have tried disabling and enabling c-states. No luck till now.
Would really appreciate if anyone helps me with this.
UPDATE:
I cleaned everything and reapplied the thermal paste. I did not see any change in the thermals. It never goes over 55°C even under full load.
I reset the motherboard by removing that jumper thing.
I ran memtest86, which took over 2½ hours. It did not show any errors.
I ran a CPU stress test for over 15 hours, and nothing crashed.
I also ran the Dell’s diagnostic tool, available in the boot menu of the motherboard. The whole test took over 2 hours but did not show any errors. It tested the memory, CPU, fans, storage drives, etc.
Try running memtest86 for a few days to test memory. That is fairly easy to do though it involves booting from a flash drive. Web search should find info.
I’ve never seen this particular error, but CPU stall warnings seem like a fairly common thing. I wouldn’t jump straight to hardware fault, but it’s a possibility.
Try stress testing the CPU and RAM. See if you can get it to happen more frequently. Also see if you can disable that CPU core, either in the BIOS or in the OS, to see if the problem goes away.
I’m with catloaf. Consistent CPU soft locks point to a possible bad memory module or CPU.
Clear CMOS.
Try removing one memory module at a time.
See if there is an option to disable hyperthreading in bios.
Another thing to try is to remove the CPU, careful not to damage the LGA pins on the motherboard, and clean the CPU contacts with alcohol. Take care to ground yourself out and the case before handling the CPU out of socket.
Can you sudo dmesg | grep microcode and see if you have any errors?
If so, I’d be inclined to sudo dnf reinstall microcode_ctl then do a sudo dracut -f to regenerate the initramfs, and reboot. Be sure to have a working fallback kernel like LTS installed so you can recover if need be.
Edit: I just read you changed to Ubuntu. I can’t be arsed to figure out how Ubuntu does this stuff so that’s on you to figure out. Alternatively, install Arch and use intel-ucode package.
Set watchdog to reboot every day and juice it until the last drop before it definetly crashes.
Edit: that’s just a workaround if you really want try to fix it my answer would be restoring BIOS defaults, then clean install of the OS and then check it everything works fine, if the error persist try installing another OS if it still fail then go to the first step.
That server sounds a bit older in the teeth… Has new thermal paste been applied to the cpu? Even if the reported temps are under 90c, you might be getting hot spots causing glitches inside the package.
Worth trying a couple of different generations of kernel as well, both newer and older. You might be hitting a regression somewhere.
I had problems with soft locks because somehow the PSU was in corrupt state, maybe through a black out or something. The problem persisted through reboots and power offs, only cutting power helped.
Yeah, always check all of this stuff. Server hardware gets a lot more updates than like gamer board BIOS, companies invest high millions, even low billions in this stuff and they expect problems to be address promptly for that kind of cash.
Check for any peripherals or cards, too. RAID, backplanes, networking cards; drivers, firmware, anything.
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !selfhosted@lemmy.world
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don’t control.
Rules:
Be civil: we’re here to support and learn from one another. Insults won’t be tolerated. Flame wars are frowned upon.
No spam posting.
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it’s not obvious why your post topic revolves around selfhosting, please include details to make it clear.
Don’t duplicate the full text of your blog or github here. Just post the link for folks to click.
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
No trolling.
Resources:
selfh.st Newsletter and index of selfhosted software and apps
Time to add a cron job to auto reboot it once a day
Does it have a Nvidia GPU? If it doesn’t then the nouveau modest does nothing.
My guess it that you are using a badly support realtek device. However I would need to see the full dmesg
It’s a long shot, but I had something similar on one of mine servers once. It was fixed by installing
irqbalanceand starting that daemon at startup.This actually worked. The CPU has to get stuck, it will in a day of being turned on, or it will keep working for weeks.
Thanks a lot for this!
Haha completely forgot about this comment. But i’m very gald it helped! I remember struggling with that problem for a long time.
Try running memtest86 for a few days to test memory. That is fairly easy to do though it involves booting from a flash drive. Web search should find info.
I just ran it. It took over 2 hours to finish. Showed no errors. Is there a benefit of running it for a few days?
If the problem is intermittent then longer run has better chance of catching it, but 2 hours with no errors is a good sign, with regard to the memory.
I’ve never seen this particular error, but CPU stall warnings seem like a fairly common thing. I wouldn’t jump straight to hardware fault, but it’s a possibility.
https://docs.kernel.org/RCU/stallwarn.html
I’d lean toward bad hardware.
Try stress testing the CPU and RAM. See if you can get it to happen more frequently. Also see if you can disable that CPU core, either in the BIOS or in the OS, to see if the problem goes away.
I’m with catloaf. Consistent CPU soft locks point to a possible bad memory module or CPU.
Clear CMOS.
Try removing one memory module at a time.
See if there is an option to disable hyperthreading in bios.
Another thing to try is to remove the CPU, careful not to damage the LGA pins on the motherboard, and clean the CPU contacts with alcohol. Take care to ground yourself out and the case before handling the CPU out of socket.
Don’t try to clean CPU pins. That is a very bad idea
I mean speaking from experience, its resurrected a couple problematic CPUs for me. CPU pins no, pads on an LGA style CPU, sure.
It is much harder to mess up pads
The CPU in this has no pins, is just contacts on the chip. The pins are in the motherboard, like the new 7000 series Ryzen.
Clean pads not pins
I’d start here: https://askubuntu.com/questions/1264859/watchdog-bug-soft-lockup-cpu6-stuck-for-23s
Can you
sudo dmesg | grep microcodeand see if you have any errors?If so, I’d be inclined to
sudo dnf reinstall microcode_ctlthen do asudo dracut -fto regenerate the initramfs, and reboot. Be sure to have a working fallback kernel like LTS installed so you can recover if need be.Edit: I just read you changed to Ubuntu. I can’t be arsed to figure out how Ubuntu does this stuff so that’s on you to figure out. Alternatively, install Arch and use intel-ucode package.
Set watchdog to reboot every day and juice it until the last drop before it definetly crashes.
Edit: that’s just a workaround if you really want try to fix it my answer would be restoring BIOS defaults, then clean install of the OS and then check it everything works fine, if the error persist try installing another OS if it still fail then go to the first step.
Had the same issues, it was heat.
Cool down your server, add a fan or a cooler…
I added a usb-powered fan sucking cooler air from outside the server area directly blowing it on the chassis.
That fixed for me.
I cleaned everything and reapplied the thermal paste. That did not solve the problem. Also, the CPU is only of 35 watts and never goes over 55°C.
That server sounds a bit older in the teeth… Has new thermal paste been applied to the cpu? Even if the reported temps are under 90c, you might be getting hot spots causing glitches inside the package.
Worth trying a couple of different generations of kernel as well, both newer and older. You might be hitting a regression somewhere.
I cleaned everything and reapplied the thermal paste. That did not solve the problem. Also, the CPU is only of 35 watts and never goes over 55°C.
I had problems with soft locks because somehow the PSU was in corrupt state, maybe through a black out or something. The problem persisted through reboots and power offs, only cutting power helped.
Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I’ve seen in this thread:
4 acronyms in this thread; the most compressed thread commented on today has 8 acronyms.
[Thread #907 for this sub, first seen 6th Aug 2024, 10:15] [FAQ] [Full list] [Contact] [Source code]
What kind of drives do you have in your RAID? Is it SMR?
The boot drive is an SSD, which is not in any RAID. I have another HDD connected via SATA. Another HDD connected via USB.
So 2 HDDs one SATA and one via USB in RAID? Can you remove the RAID drives and test it out?
Also what size are the drives and what’s their capacity?
No, they are not in RAID either.
My bad. What are you using the drives for?
Films and shows, via Jellyfin.
Is it just you, any transcoding? Have you tried it without the USB drive?
removed by mod
Make sure the microcode package is up to date as well
Yeah, always check all of this stuff. Server hardware gets a lot more updates than like gamer board BIOS, companies invest high millions, even low billions in this stuff and they expect problems to be address promptly for that kind of cash.
Check for any peripherals or cards, too. RAID, backplanes, networking cards; drivers, firmware, anything.
I did reset it. It did not help. I ran memtest86 for over 2 hours and did a CPU stress test for over 15 hours. Nothing crashed during the testing.
Poorly supported hardware will also do this
An Optiplex 5040 should be well and thoroughly supported for 6+y now