How one line of code caused a $60 million loss
engineercodex.substack.com
external-link
60,000 people lost full phone service, half of AT&T's network was down, and 500 airline flights were delayed

And it wasn’t COBOL.

I feel like anyone doing any automation with aws could hit this

@spudwart@spudwart.com
link
fedilink
English
910M

Damn, if only the systems your phone network were ran on hadn’t been forcibly closed source and scared good devs from interacting with you because of your sue-happy nature regarding BSD/Unix.

If one single line of code can make you lose $60M, surely you’ll ensure due review processes and independent QA and clear requirements and regular audits and a middle management not only doing KPI monitoring for a failing upper management. Right? Rrrright?

@BeardedGingerWonder@feddit.uk
link
fedilink
English
11
edit-2
10M

Hahahaha, you’re kidding right? I shit you not, I’ve literally seen a single line change almost cost a company £150MM during testing because “we need to test in prod because the guy we need to run the test hasn’t got access to the QA environment”

Best part was the actual change, there was a bug where a number that should’ve been divided by 100 was being multiplied by 100, the dev somehow managed to implement the fix in such a way that the number was multiplied by a further 100.

snowe
mod
link
fedilink
210M

Knight Capital Group lost $440 million in just 45 minutes due to a repurposed feature flag.

https://www.henricodolfing.com/2019/06/project-failure-case-study-knight-capital.html?m=1

Kinda makes the att one seem tiny in comparison.

Flying Squid
link
fedilink
4110M

AT&T made 120.74 billion in 2022. They can afford a lot of bad code.

MxM111
link
fedilink
710M

“60,000 people lost full phone service, half of AT&T’s network was down, and 500 airline flights were delayed”

Flying Squid
link
fedilink
1610M

I’m sure AT&T care about that due to their humanitarian nature.

MxM111
link
fedilink
210M

AT&T is not humanitarian non-profit company. It should worry about increasing its profits by providing people with good product so that people choose them over competition.

Natural monopoly and anti-competitive behavior would like to have a little chat with you in that dark alley over there…

MxM111
link
fedilink
110M

Yes, anti-monopoly laws are important to provide competitive market.

Flying Squid
link
fedilink
1110M

That hasn’t seemed to be their overall strategy considering how shitty their service is.

MxM111
link
fedilink
010M

I personally had better experience with them than Verizon. But whatever. The fact that they have customers today does meant that they provide competitive service. Today, you can easily switch provider.

Flying Squid
link
fedilink
310M

AT&T does a lot more than provide phone service.

Revenue is not profit.

Edit: Jesus. Their profit was $70b in FY 2022?! On $120b in revenue? Maybe I’m the one misunderstanding.

Flying Squid
link
fedilink
1010M

Do you think their profits will be dented by a $60 million loss?

With a +/-0.1% margin of error, the difference between $60M and $70B is $70B.

Recently, I paid ATT $500 to close my account and zero it out.

Fast forward 6 months, they send me a letter saying they need $300 more for cancellation fees or it will be sent to collections.

Write all the bad code you can, papi.

Sure it’s cancellation fees? This doesn’t seem legal.

Caveman
link
fedilink
610M

There are contract breaking fees attached to contracts sometime to prevent switching providers. I’m pretty sure everybody agrees it reduces competition.

Someone should write more of those lines

@rglullis@communick.news
creator
link
fedilink
English
-1710M

So you want more people to lose full phone service?

@Sanctus@lemmy.world
link
fedilink
English
1410M

No, we’re suffering, and most people just want these “people” with billions in revenue to pay their fair share instead of having twelve full time tax representatives whose only job is to make sure the public doesn’t see a dime after using public spaces and resources all year.

@rglullis@communick.news
creator
link
fedilink
English
-6
edit-2
10M

There are better and more effective ways to fight for justice besides just wishing that fines or that (a) do not really affect their bottom line, (b) can cause immense problems to the people who are mere customers and (c) will at most get one of their engineers fired but never point fingers to the manager who asked for the change to be expedited and pushed to production without review.

@Sanctus@lemmy.world
link
fedilink
English
910M

I’m going to be real with you, over the last 80 years corporations have created an environment where they can levy their large profits to manipulate the government into passing beneficial legislature. I don’t trust a single corporation anymore and I’d be more than happy to see them all burn, even if it means kissing our world goodbye. Every day I drive to work to make shit money with a degree I was told to get, my boss is pulling his 3rd home while he tells me I’m not worth 60k yet. But the average pay for my skills and degree is 65k, but nobody else will hire me. I’m incrementally closer to poverty each day and its maddening. Seeing such luxury while I’m six feet below the water fucks me up. I’m not asking for luxury, I’m asking to buy a house, to afford to provide well for my family. But that is even too much. So fuck it all. If I’m not allowed in the living club because of some birth lottery, or the universe coalescing to redirect an existing stream into my account, then I am going to be angry. There is nothing special about my boss besides the fact that he put down 30k from his GI bill when it mattered. Thats not a good fucking system, its the laws of the jungle but plastered in concrete. Fuck that.

@rglullis@communick.news
creator
link
fedilink
English
-510M

First, a bit of perspective: you were born in the richest country of the world and your salary alone is enough to put your family above the median national income. If you have a working spouse making half of that, your family would be around the 70th percentile. And again, we are talking about the richest country of the world. So, please stop complaining about losing some “birth lottery”.

Second, if the system has failed you, getting angry at some of the players can feel cathartic but is nothing more than a coping mechanism. It does little to nothing to actually find for a way to fix your problems. How can one help you to find a way to direct the energy from this anger to something productive?

ToRA
link
fedilink
English
8
edit-2
10M

Your argument of “other people have it worse” is pure ignorance at best.

Also, do you think one person has the ability to move the needle even a little towards fixing the issue? Getting angry is absolutely a step towards progress. We all need to be angry. We need to be angry enough to all work together and make a difference. Not everyone can spend the time needed to even attempt make a difference as an individual, but we can help rally together by ‘complaining’

This all started as a joke and you took it way too seriously and now you’re telling people they can’t feel bad because they aren’t suffering enough compared to others? Dude, cmon.

@rglullis@communick.news
creator
link
fedilink
English
-210M

My point is: getting angry is not enough, and a competition to see who seems the angriest or says the most disparaging thing ends up becoming performative action.

What I’d like to see is people showing real skin in the game. Telcos in the US are running a racket and abuse their de-facto monopoly? Then how about we start looking into companies that provide mesh networks like Fon?

To put in real terms: I know for a fact that I could offer you unlimited phone calls for ~$15 month and have a healthy sustainable operation, but I need at least 100 people to sign up to this just to setup the basic infra. Everyone would be able to make and receive phone and texts, but they would lose some convenient features provided by larger carriers. Are you “so angry” at the big corporations that you’d be willing to go get these 100 people, or is your anger just enough to fuel some occasional rant on an internet forum?

@Sanctus@lemmy.world
link
fedilink
English
410M

I’m sorry, I think about how I am lucky on a global scale, but on a local scale I was just one bad day away from being broke and I just had that day. So it sucks, I hone my skills every day, game dev every night, study IT skills for my main job, its just not fast enough to put me in a house, or feel like I’m at least treading water instead of swapping bills around every month. Thank you for the discussion and the question. But my issue with how we live is random people like my boss, who consistently tells me to end India’s scam attempts, decides my salary. If he doesn’t know I can’t combat the entire nation of India for under 60k a year, why does he get to say how much I’m worth? The system rewards people who already have money, thats it. Everyone else can subsist off of goodwill and prayers. These systems we made are a feedback loop to return wealth to the wealthy and keep it there.

I work for the Atlantic and Pacific Telegraph Company, and I think I talk for everyone when I say: yes, why not

Would get more people upset that they don’t have an alternative (assuming this affects one of those cable monopoly communities)

@cynar@lemmy.world
link
fedilink
English
4610M

Interestingly, this software was not tested. Testing was actually bypassed as per management’s request because the code change was small.

I don’t think much needs to be said about this…

meseek #2982
link
fedilink
1210M

;

There I fucked up your entire codebase.

Create a post

Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!

Cross posting is strongly encouraged in the instance. If you feel your post or another person’s post makes sense in another community cross post into it.

Hope you enjoy the instance!

Rules

Rules

  • Follow the programming.dev instance rules
  • Keep content related to programming in some way
  • If you’re posting long videos try to add in some form of tldr for those who don’t want to watch videos

Wormhole

Follow the wormhole through a path of communities !webdev@programming.dev



  • 1 user online
  • 1 user / day
  • 1 user / week
  • 1 user / month
  • 1.11K users / 6 months
  • 1 subscriber
  • 1.21K Posts
  • 17.8K Comments
  • Modlog