The IT Nerd

Microsoft CEO Steve Ballmer made his first public comment about the Sidekick circus yesterday in Network World. Here’s what he said:

“It is something we are going to have to address and explain to customers our method and process and quality approach and what went wrong in that case and how we are making sure that it does not happen again,”

But it gets better than that. Here’s what else he said:

“It is not clear there was data loss,” he said. “Initially we thought there was. We are working hard to get all the users’s data back in the Sidekick case. I think we believe we will get all user data back at this juncture.”

Oh really? If there was no data loss, why tell users that there was data loss after you and T-Mobile stalled on saying anything for a week? Why put users through that sort of stress if there wasn’t data loss of some sort? As for regaining customer trust, try this response from the chair throwing one:

“People will want to know, is our approach different for SharePoint Online, is our approach different for the enterprise infrastructure. I think we have good answers, but I know we are going to continue to upgrade our processes and have to upgrade how we talk about this stuff, because we are going to get more questions.”

He better hope that his approach is different for the other Microsoft cloud products, because I’m guessing that people think that they’re suspect at the moment because of this fiasco.

Steve my boy, you need better answer than these as they’ve done little to make people feel warm and fuzzy all over.

At least Microsoft is making progress with this nightmare. According to this post from Microsoft, Sidekick owners can use a recovery tool on T-Mobile’s Web site to get their contacts back:

The Danger / Microsoft team continues to work around the clock and has completed its latest round of rigorous tests. We are now ready to make the first phase of the content restoration process available to you, starting with personal contacts.

This data restoration effort is only necessary for the minority of customers who lost data from their Sidekick devices.

I don’t know about the “minority of customers who lost data from their Sidekick devices” as there’s a lot of pissed off Sidekick users out there, but at least there’s a light at the end of this tunnel. The question is, how long will it take to get everything else back?

I guess Microsoft must be feeling the heat as they came out with a statement yesterday apologizing for the amount of time it is taking to get Sidekick data back online so that Perez Hilton can stop bashing them on his Twitter feed:

The Danger / Microsoft team is continuing to work around the clock on the data restoration process.  We apologize that this is taking so long, but we want to make sure we are doing everything possible to maintain the integrity of your data.

There’s a timeframe of “this week” in the statement to get everything online, and for their sake  I hope they make it. If they do get the data back on line and everybody is happier, then that will take the wind out of the sails of the lawsuits that were filed last week. That would make life easier for Microsoft and T-Mobile I would think.

Here’s another plot twist in a story that has had a number of plot twists. T-Mobile posted this message on their Sidekick forum this morning:

We are pleased to report that we have recovered most, if not all, customer data for those Sidekick customers whose data was affected by the recent outage. We plan to begin restoring users’ personal data as soon as possible, starting with personal contacts, after we have validated the data and our restoration plan. We will then continue to work around the clock to restore data to all affected users, including calendar, notes, tasks, photographs and high scores, as quickly as possible.

I guess this means that they had a backup after all? Or does this mean that the sabotage angle of this story was bogus? Or perhaps they found who did it and “encouraged” them to help resolve the issue? Seriously, you have to wonder how they went from all the data being lost to most if not all the data being recovered in such a short space of time. We’ll never get answers to those questions I suspect.

In any case the proof is in the pudding as they say, so we’ll see if everything comes back up for users. It would likely go a long way to mitigating those lawsuits that were filed yesterday. Sadly for T-Mobile and Microsoft, it will do nothing to stop users from running to buy a Blackberry or iPhone on another carrier. Nor will it stop the damage to their respective reputations. The damage has already been done on those fronts.

From the “what took them so long” department comes the first lawsuits over the Sidekick outage being filed. One of them highlights the fact that there was apparently no backup of users data. Something that I’ve touched on before:

T-Mobile and its service providers ought to have been more careful the use of backup technology and policies to prevent such data loss” said Ira P. Rothken, an attorney working on that case. “We are hopeful that T-Mobile and the rest of the defendants will do the right thing, use this as an opportunity to redesign the system as a new standard for cloud computing storage, and provide full compensation for the data loss.”

Another highlights the fact that the data that was lost has a high value, which is something else that I’ve touched on in the past:

Another suit, filed on behalf of Maureen Thompson “and all others similarly situated” seeks unspecified damages for Thompson and others who have lost data as a result of the recent Sidekick problems.

According to her lawyer, Thompson owns a Sidekick used primarily by her daughter, an aspiring model, singer, and songwriter who used her Sidekick to store personal and business contacts, appointments, and even irreplaceable song lyrics not stored anywhere else. The lawyer said that Thompson bought the device “primarily because T-Mobile promised that any data would be protected and available no matter what happened to the phone.”

“T-Mobile’s initial efforts to reimburse Sidekick users are a step in the right direction, but fail to sufficiently compensate Sidekick users for this disastrous loss of data,” Thompson attorney Jay Edelson said in a statement. “T-Mobile and Microsoft promised to safeguard the most important data their customers possess and then apparently failed to follow even the most basic data protection principles. What they did is unthinkable in this day and age.”

Wait. Didn’t Perez Hilton point out on his Twitter feed that T-Mobile told him that he didn’t had to worry about backing up his data because they did it for him? I bet T-Mobile is wishing they never made that claim. Speaking of T-Mobile, I wonder how they and Danger/Microsoft are responding to this? Oh wait. They’re quoted in this article too:

Microsoft declined to comment on the lawsuit, but, a representative said on Wednesday that the company is “obviously very sorry for the inconvenience that this situation has caused Sidekick users, and we are working around the clock in an effort to recover and restore the data for any affected users. While it is still too early to say for sure, we announced on Monday that our engineering teams were increasingly optimistic.”

For its part, T-Mobile said in a statement that it “does not comment on pending litigation.”

Seeing as some users were getting their data back, I guess there’s reason for Microsoft to hope. But I suspect it won’t be enough for them to avoid this lawsuit I suspect.

Meanwhile, the mass defections to other carriers that have Blackberries and iPhones continues while this plays out in court. You can also expect that more lawsuits will be filed shortly as well.

Here’s a plot twist that I never saw coming. “Insiders” within the Microsoft Danger/Pink group have alleged via AppleInsider that the Sidekick outage must have been caused by Sabatoge and further complicated by internal politics within Microsoft:

“If this was an ordinary sort of failure, the service would have come back within a day, so once again, all signs point to sabotage. If they erased the server hard drives, they would have to reinstall the OS on each affected server, then reload all of the server-side software and start everything back up, and who knows how many people are remaining at Danger who even know how to do all of that? Once again, there is no-one on the Microsoft side who is going to know how to do any of this.

Charming. I’ve always said that every business is one disgruntled employee away from a major disaster. If this is true, this pretty much proves that point. But it gets better. Apparently the word is that T-Mobile is irate at Microsoft:

“T-Mobile has an SLA (Service Level Agreement) with Danger/Microsoft, which is a standard legal document for these types of relationships, one that requires Danger/MS to reimburse T-Mobile with defined monetary penalties if the service goes down for longer than x minutes, etc. I have no clue about the details, but clearly a week-plus outage plus permanent loss of all user data stored in the cloud (leaving only the user data stored on the devices themselves, which will completely vanish if the device is shut down improperly or crashes!) is the worst possible violation of the SLA conceivable, and essentially guarantees a very nasty lawsuit against Microsoft, regardless of whatever forensic and legal investigations they are doing to try to find the culprit,” one of the insiders explained.

“T-Mobile is now getting blamed for something which isn’t their fault at all, and a million plus customers are now seriously considering leaving for the iPhone or elsewhere. I’m also thinking that a class-action lawsuit on behalf of those users who lost all of their data (contacts, notes, emails, SMS’s, tasks, calendar entries) is now quite likely, and once again T-Mobile is going to be caught in the crossfire, even though the servers were all run by Danger/Microsoft and not T-Mobile.”

Even though I’ve criticized T-Mobile in the past, I hope that T-Mobile puts the wood to Microsoft as this is a major screw up on their part. In any case, I encourage you to read the whole story as it really shows how screwed up this situation this is.

It also means that it can only get worse from here.

There’s a story floating around that Danger/Microsoft that the massive Sidekick fail is due to…. Get this… Not having a backup:

Microsoft was upgrading their SAN (Storage Area Network aka the thing that stores all your data) and had hired Hitachi to come in and do it for them. Typically in an upgrade like this, you are expected to make backups of your SAN before the upgrade happens. Microsoft failed to make these backups for some reason. We’re not sure if it was because of the amount of data that would be required, if they didn’t have time to do it, or if they simply forgot. Regardless of why, Microsoft should know better. So Hitachi worked on upgrading the SAN and something went wrong, resulting in it’s destruction. Currently the plan is to try to get the devices that still have personal data on them to sync back to the servers and at least keep the data that users have on their device saved.

We’ve heard this from what appears to be several sources and it seems to hold weight. Needless to say it all boils down to one thing: Microsoft did not have a working backup.

You’ve got to be kidding me! The world’s biggest software company DIDN’T HAVE A WORKING BACKUP? What kind of two bit operation is this? Am I to believe that a sole computer consultant like me has the ability to restore data from a year ago to any of the computers in my house at will (Because I’ve tested that ability to make sure that I could do so if I needed to. Such as when my place was broken into and my MacBook Pro was stolen) and Microsoft DIDN’T HAVE A WORKING BACKUP? If they can’t be trusted with a bunch of smartphones that store their data on a central server, why should I trust Microsoft service like Office Live or any of their cloud based services? No wonder Perez Hilton is ticked because clearly these jokers haven’t got their act together.

Oh, in case you haven’t figured it out, this disaster is the end of the Sidekick as nobody is ever going to own one of these phones again. And I would wonder if any other smartphone with a Microsoft OS on it would sell. I think they’re all damaged goods at this point.