T-Mobile COO Mike Sievert announces Netflix partnership (GeekWire Photo/Kevin Lisota)
T-Mobile has sent us the following statement:
This is not a cyberattack. It is an internal technical issue that has temporarily impacted some platforms. We’re working quickly to resolve it.
T-Mobile
In addition, they followed up stating that the issue is now resolved
Click here for our other latest update, including a possible cause.
Original coverage below.
It looks like T-Mobile has been hit with a serious system outage.
Customers attempting to login via the web or the app were met with an error message on Thursday. The website redirected attempts to login to a dedicated “outage” site.
Even more worrying, according to a tip sent in to us here at The Mobile Report, T-Mobile Money, the company’s banking service, was also down. Customers were apparently unable to access their accounts during this outage. Indeed, attempting to login to T-Mobile Money also resulted in a redirect to the outage page.
In addition, according to multiple sources, internal systems were also completely down, with employees unable to do pretty much anything for customers. As of 6:30PM Eastern, there were a few reports of some internal systems beginning to recover, and around 8PM they appeared to be mostly back up.
There were even reports of customer accounts being suspended due to non-payment because they were unable to login to pay their bills, and customer support were unable to do anything about it either.
This is a developing story. We’ve not heard anything official from T-Mobile about the outage, but we’ll keep an eye on things and update when we learn more.
Update 1: Systems are beginning to be restored, and we may know why it happened.
A source on Reddit has stated that the cause was a “rogue admin ID” deleting internal programs from servers.
A script was executed that deleted every single namespace managed by the Conducktor platform. A namespace is essentially an abstraction of a cluster of EC2 instances that are leased from AWS. Conducktor manages the leasing, organization, networking configuration, and API orchestration to handle deployment and configuration of AWS stuff in general but its primarily EC2 instances, K8S configuration, some Redis, Elsaticache, Routing/Load Balancers, etc. It’s a lot. Too much to list and it gets complicated in a hurry and I don’t know how to succinctly summarize it. Maybe “Giant magical AWS wrapper” ?
But the overall is that this means that every team that owned an application or service, that deployed to AWS via Conducktor, had their stuff nuked. Conducktor is very widely used in Digital for APIs and applications. So most UI applications that are served from a webserver, APIs running on a java server, etc., were impacted as those servers themselves were deployed to EC2 instances managed by Conducktor. This is why this was such a widespread problem across channels (Retail, Care/Telesales, Web and App) as well as across lines of business (Prepaid, Postpaid, Business, Tmobile Money – which nobody knows exists nor should they, etc etc).
Quoting from a guy on the bridge, this was done by “A rogue admin ID” …so …I dunno, that smells really bad to me. Like, someone’s going to jail kinda bad.
Anonymous Reddit user
Obviously, this sounds really bad. If this is to be believed, either a rogue employee/former employee or a bad actor accessed high-level internal systems and ran a script, perhaps custom and purpose-made, to deliberately cripple T-Mobile systems.
These claims are unofficial, of course, and should be taken with a grain of salt. We’ll continue to post updates as we learn more.
Update 2: Systems both external and internal at T-Mobile appear to be mostly restored. As mentioned in our previous update, it is possible a bad actor intentionally disabled T-Mobile’s systems, though that is still at this point unconfirmed.
So far, and most importantly, it seems that no customer data was exposed. This is also unconfirmed, but based on information we’ve seen that appears to be the case. We will, of course, update if this changes.
