Amazon's total all-in cost per dev is almost certainly >$300k per year, so more like >$7 million if that's an accurate number.
As far as how it gets approved, that's not enough money to really register at Amazon's scale. I've seen waste similar to that at companies much, much smaller than Amazon that goes entirely un-noticed.
> As far as how it gets approved, that's not enough money to really register at Amazon's scale. I've seen waste similar to that at companies much, much smaller than Amazon that goes entirely un-noticed.
Maybe I'm being dense, but this still doesn't register for me. If 4+ calendar years of work for a single team continues with no noticeable progress, then there's an organizational issue, no matter the size of the company.
Of course there is, but that's not the point. Every large organisation has organisational issues and nobody in the history of humanity has found a way of preventing them yet. The only thing you can do when the group sizes get large is to try and keep the problem under control relative to how bad it is at other large companies.
As far as I could tell, the "reason" given by the OP for this waste was attributed to Amazon's scale. Could you explain how company size and broken corporate structure are correlated? Or explain why 20+ person-years of engineering time wasted isn't a symptom of broken corporate structure?
That there is an organizational issue. Of course there is an organizational issue. There will always be organizational issues.
> Or explain why 20+ person-years of engineering time wasted isn't a symptom of broken corporate structure?
Oh, it's definitely an organizational issue here, that's just not the main point. If you're willing to call this a "broken corporate structure," then you should revise your opinion of every company with more than 50 employees downward sharply. This is a normal corporate structure, not a broken one.
> Could you explain how company size and broken corporate structure are correlated?
When there are enough people, you will always end up with conflicting incentives somewhere. A larger organization will have more of these. In this case, it seems to be a clear case of competing priorities: IT wants to secure systems, and developers want to be maximally productive. I'm sure some folks in IT view Amazon paying ~$7M as a fair cost to remove known vulnerable tools from their infrastructure.
It's unlikely that this went entirely unnoticed for 4 years, and I would guess that when the problem was understood, someone did the math on staying the course, versus starting over or reversing course.
And after all the internal calculations were complete (which factor in way more than technical effort by a small dev team), they intentionally stayed the course.
This is probably an extreme example because an internal wiki that is used company-wide will have an incredibly complex and hairy stakeholder matrix (both business and technical), which means consensus is very challenging.
At some point the cost of the meetings required to work out a solution (both in time spent, and in time not spent on core work by the stakeholders) probably exceeds the cost of just letting a small team brute-force their way through a bad solution.
Cost here isn't just money, it's focus, morale, disruption, conflict, risk, etc..
Keep in mind that you have the benefit of hindsight, and the comfort of looking at it from the outside, with a very incomplete picture of all the constraints and pressures at play.
But having worked in and around these kinds of teams and companies for most of my 25-year career, I can tell you that it's "often", in my opinion/experience.
It's pretty easy to have a project look like progress is being made in the moment, especially when presenting to the director+ level, where a manager is presenting on 4-5 different projects. It's even easier with projects that aren't important or highly-visible (as internal projects tend to be).
>that's not enough money to really register at Amazon's scale.
This viewpoint doesn't really match up with how enterprises operate.
It's possible to function at amazon's scale without meticulous accounting and financial operations. The reason for this is that is extremely easy for wasteful spending to propagate throughout the enterprise if unchecked.
Waste only appears after a project fails...it's easy to point out waste after the fact, but it's downright impossible to call it out as it's happening.
As far as how it gets approved, that's not enough money to really register at Amazon's scale. I've seen waste similar to that at companies much, much smaller than Amazon that goes entirely un-noticed.