iPhone 4 Ordering and Session Switching

Users are reporting issues trying to reserve the ability to purchase (pre-order) the latest iPhone 4 on June 24th, when they go on sale, caused by what is basically a complete overwhelming of the systems designed to take the orders. AT&T’s web interface at brick and mortar shops is failing to the point where orders are being taken with pen and paper, and the Apple web site is acting clunky. But the most serious issue people are reporting is that upon logging into AT&T online to place the order, another user’s information is coming up.

AT&T eventually just disabled online access for its users. Gizmodo received these messages from their readership amongst around 8 complaints:

From: Eric Paul Mertens
Date: Tue, Jun 15, 2010 at 11:19 AM
Subject: AT&T iphone pre-order wrong account

This morning while trying to pre-order the iPhone 4 through AT&T website, my login brought me to a ‘website 
unavailable' screen. After a refresh it brought me to the phone upgrade page logged in under a different 
account, 

some dude from Lakewood OH!

From: Michael
Date: Tue, Jun 15, 2010 at 11:09 AM
Subject: AT&T security breach

Hello,

I am not sure if this e-mail is going to the correct place but this morning when trying to log in to my at&t 
account I entered my information and ended up in someone else's account with access to all their information. 
I feel as though someone could now be logged in to my account. To me, this seems like a huge security 
breach and at&t is shrugging me off as if it is no big deal and I feel that it is. I just thought with the
recent i-pad breach that this one is an even bigger one and I wanted to bring it to your attention.

Thanks,
Michael Staropoli

And one user was kind enough to provide a screenshot to drive home the point.

From: john king
Date: Tue, Jun 15, 2010 at 2:04 PM
Subject: ATT WEBSITE LOGS ME IN AS ANOTHER CUSTOMER
To: tips@gizmodo.com

I LOGGED IN AS ME AND IT BROUGHT UP A MARY ???? BIG PROBLEM
-JPK

System Upgrade?

Gizmodo received a report from a 3rd party put forth that an untested (in their opinion) system change is probably responsible for these issues.

I work at a 3rd party order processing facility—what AT&T refers to as a 3CC. We process business-to-business, business-to-customer Wireline Indirect, and ACME/PAC (what AT&T calls their iPhone program internally). Agents use AT&T programs called Phoenix, Telegence, Compass, Ordertrack and myCSP to process orders.

Over the weekend there was a major fraud update that went down on all of AT&T’s systems, from Saturday overnight to Sunday early morning. All systems were down and agents were unable to use any systems.

The issues people are seeing at AT&T stores and online are most likely related to this update that went wrong.

I do know that there was absolutely NO TESTING of this system done before the launch of the new iPhone. I know it’s just heresay at this point, but I can confirm that there was a major outage over the weekend that impacted all ordering systems and programs, and I can confirm that there were multiple systems being upgraded/updated, with some updates being related to fraud.

At this point, I can say that the system that AT&T uses to send automated orders to be processed is as of this very moment down completely. Our facility is unable to process any orders by phone or by automation.

(Regarding the identity problem) Whenever we see people who are logging in and seeing other customer’s account info, it is an issue with the databases that contain customer information. Orders that contain any information like this can cross customer information, and cause a customer be able to see other accounts by logging out and logging back in. This means that when they log in a few times, it gives them different customer account info every time. It’s a rare occurrence, but it has happened in the past.

You might want to advise people to not get the upgrade at this point as it may be a doorway to a major privacy breach.

This analysis is thin and speculative at best. It seems to be focused on B2B platforms, whereas the problems are reported in the retail web site. It is possible to have a problem introduced this way, but theoretically it would probably be more wide spread.

Nah…

So if we look at this, without any other information, how do we decide that one user logging in and seeing another user’s information is probably not the result of a weekend systems’ upgrade? Because we’ve seen this behavior before, a lot. When you stress test a web site, its not uncommon to see functions that return and read user sessions get garbled, and web sites start to return pages for the wrong user session.

When you log into a web site a session gets created and some sort of persistence mechanism is returned to maintain the session (usually a session cookie, but there are other less used methods available). Every “logged in page” reads this session identifier to determine whether the user is logged in and uses it to return the right information. Further complexity is usually introduced into large web sites, where some sort of load balancing is taking place, and therefore a user’s session has to be found amongst data centers, servers, and so forth.

As Christian points out below, one example is that session cookies are sometimes made unique based on a time stamp which in many sites will only go down to milliseconds. Thus when a site receives too many concurrent requests, it starts issuing duplicate session cookies.

When you overload the capacity of programs that read, manage, and create sessions, bad stuff happens like sessions getting crossed. Since the AT&T site was probably under a severe and unusually high server load today, the site went haywire (in our technical opinion).

How do you prevent this from happening? Add occasional and event driven stress testing to your quality assurance processes, you will find a number of unusual and difficult to solve problems result. At the very least you will know how your web application acts under unusually high loads, and thus not be surprised when the Apple fanboys come calling for Steve’s latest masterpiece.

Related Posts:

• Security and IT Pros, I need your help
• May’s Patch Tuesday
• First Patch Tuesday of 2010
• Regular or Decaf? Tool launched to combat COFEE
• Six Bulletins in Last Patch Tuesday of 2009

Filed Under: Security

Tags: apple, AT&T