My server (fedora) stops all podman containers after 2-3 hours since 3 days. I can start all containers again, and the same happens after a while. I do not know where to look for the problem.
In top, I found a oom message. I assume that the system runs out of memory and stops all services. How can I find the problem? I can’t find anything in the container logs.
I can see that systemctl status is always starting. It doesn’t become “running”. But I do not know how to proceed.
Install atop, basically ‘top’ on steroids with history… It defaults to capturing performance data every 5 minutes, I usually change it to 1 minute on production systems.
When I had the issue with mariadb demon been killed, I think either in dmesg or syslog there was an entry reading "Out of memory: Kill process… " or similar.
The issue with diagnosing memory issues is that it usually results in no memory available to handle the logging of such a problem when it happens.
I’ve found that the easieat approach is to set up a file as additional swap space, and swapon, then see if the problem disappears, either partially or fully.
If something you’re running has a memory leak then it doesn’t matter how much RAM you have.
You can try adding memory limits to your containers to see if that limits the splash damage. That’s to say you would hopefully see only one container (the bad one) dying.
So far I follow a bottom up strategy. I’ll keep adding containers each day (or after many hours) and wait for it to stop. I also looked up how to limit memory usage. It’s a great idea to limit all containers and see which one fails. thanks!
If you saw an OOM anything, it’s getting OOMkill’d by the kernel trying to keep the machine up. Check syslogs and dmesg, and it should say what was killed, and there’s your problem container. You probably have a memory leak, so just check your container stats every so often and see what is growing out of control with memory usage.
Enable swap regardless. Would also help to know what you’re running.
If you’re seeing an OOM killer messsage note that it doesn’t necessarily kill the problem process, by default the kernel hands out memory upon requestt, regardless of whether it has ram to back the allocation. When a process then writes to the memory (at some later time) and the kernel determines that there is no physical ram to store that write, it then invokes OOM Killer. This then selects a process and kills it. MySQL (and MariaDB) use large quantities of ram for cache, and by default the kernel lies about how much is available, so they often end up using more than the system can handle.
If you have many databases in containers, set memory limits for those containers, that should make all the databases play nicer together. Additionally , you may want to disable overcommit in the kernel, this will cause the kernel to return out of memory to a process attempting to allocate ram and stop lying about free ram to processes that ask, often greatly increasing stability.
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !selfhosted@lemmy.world
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don’t control.
Rules:
Be civil: we’re here to support and learn from one another. Insults won’t be tolerated. Flame wars are frowned upon.
No spam posting.
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it’s not obvious why your post topic revolves around selfhosting, please include details to make it clear.
Don’t duplicate the full text of your blog or github here. Just post the link for folks to click.
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
Install atop, basically ‘top’ on steroids with history… It defaults to capturing performance data every 5 minutes, I usually change it to 1 minute on production systems.
When I had the issue with mariadb demon been killed, I think either in dmesg or syslog there was an entry reading "Out of memory: Kill process… " or similar.
I’ll have a look, thx
The issue with diagnosing memory issues is that it usually results in no memory available to handle the logging of such a problem when it happens.
I’ve found that the easieat approach is to set up a file as additional swap space, and swapon, then see if the problem disappears, either partially or fully.
I’ve got way too much RAM for swap being useful at all. Good idea though.
How do you know that you have too much ram? Have you set up a monitoring solution like influxDB to track ram usage over time?
I observed it during resource hungry usage. I never had issues with it, not even close.
If something you’re running has a memory leak then it doesn’t matter how much RAM you have.
You can try adding memory limits to your containers to see if that limits the splash damage. That’s to say you would hopefully see only one container (the bad one) dying.
that’s neat. Tank you.
So far I follow a bottom up strategy. I’ll keep adding containers each day (or after many hours) and wait for it to stop. I also looked up how to limit memory usage. It’s a great idea to limit all containers and see which one fails. thanks!
There is no such thing as too much RAM…
Then you didn’t understand how the system uses swap.
https://chrisdown.name/2018/01/02/in-defence-of-swap.html
They could mean that they have swap but it’s not being used.
I’m just curious how much RAM you think that is.
If you saw an OOM anything, it’s getting OOMkill’d by the kernel trying to keep the machine up. Check syslogs and dmesg, and it should say what was killed, and there’s your problem container. You probably have a memory leak, so just check your container stats every so often and see what is growing out of control with memory usage.
Enable swap regardless. Would also help to know what you’re running.
I would start all containers except one. If everything works that one is the cause of the problem. Keep trying with a different container every time.
Are you running them from your user session? If so, when you log out it will stop your processes, unless you have enabled ‘linger’ mode.
If you’re seeing an OOM killer messsage note that it doesn’t necessarily kill the problem process, by default the kernel hands out memory upon requestt, regardless of whether it has ram to back the allocation. When a process then writes to the memory (at some later time) and the kernel determines that there is no physical ram to store that write, it then invokes OOM Killer. This then selects a process and kills it. MySQL (and MariaDB) use large quantities of ram for cache, and by default the kernel lies about how much is available, so they often end up using more than the system can handle.
If you have many databases in containers, set memory limits for those containers, that should make all the databases play nicer together. Additionally , you may want to disable
overcommitin the kernel, this will cause the kernel to return out of memory to a process attempting to allocate ram and stop lying about free ram to processes that ask, often greatly increasing stability.