I’m working in a system that generates 750 MILLION non-debug log messages a day (And this isn’t even as many as others).
Good luck grepping that, or making heads or tails of what you need.
We put a lot of work into making the process of digging through logs easier. The absolute minimum we can do it dump it into elastic so it’s available in Kibana.
Similarly, in a K8 env you need to get logs off of your pods, ASAP, because pods are transient, disposable. There is no guarantee that a particular pod will live long enough to have introspectable logs on that particular instance (of course there is some log aggregation available in your environment that you could grep. But they actually usefulness of it is questionable especially if you don’t know what you need to grep for).
These are dozens, hundreds, more problems that crop up as you scale the number of systems and people working on those systems.
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !programmerhumor@lemmy.ml
Post funny things about programming here! (Or just rant about your favourite programming language.)
Rules:
Posts must be relevant to programming, programmers, or computer science.
No NSFW content.
Jokes must be in good taste. No hate speech, bigotry, etc.
What the fuck is center even talking about? Is that shit a thing people do?
A good chunk of it is relating to the elastic search stack, yeah it’s a thing people do.
My life got so much better after we abandoned elasticsearch at work
Yeah, ofc it is.
I’m working in a system that generates 750 MILLION non-debug log messages a day (And this isn’t even as many as others).
Good luck grepping that, or making heads or tails of what you need.
We put a lot of work into making the process of digging through logs easier. The absolute minimum we can do it dump it into elastic so it’s available in Kibana.
Similarly, in a K8 env you need to get logs off of your pods, ASAP, because pods are transient, disposable. There is no guarantee that a particular pod will live long enough to have introspectable logs on that particular instance (of course there is some log aggregation available in your environment that you could grep. But they actually usefulness of it is questionable especially if you don’t know what you need to grep for).
These are dozens, hundreds, more problems that crop up as you scale the number of systems and people working on those systems.
This write-up can be the next KRAZAM skit