My first experience in War Room

Prologue - We've been working on a project for the last four months. It's finished now and going to be installed today.

War Room - It's a place where people from different groups, involved directly or indirectly in the development of a project, gather on the eve of a project installation. It has been renamed to Ops (operations) Room, but people still like to call it the War Room due to nature of the work that goes inside. It's a huge conference room fully equipped with gadgets. Large LCD panels are placed on the three walls of the room. At the center, there is a large table where around 20 people can sit and connect there laptops to work.

The Project - It's a re-write of an existing application, which facilitates users to apply online for a Wellsfargo Credit Card. The existing application needs user to fill out 17 pages to complete the application, which makes a lot of users to drop out in between. The new application has simplified this process and reduced it to a single page and that is why the name Next Stage/Generation Credit Card application.
The project has a very high visibility in Higher Management of Wellsfargo, but it has been in News from the day of its inception for all the wrong reasons. The Go Live date itself says it all ..June 13, 2008 ..Friday the 13th.


Date: Thursday June 12, 2008
Location: War Room

4.00 PM: People have started gathering in the War Room.
4.15 PM: People from all the groups involved are either in the War Room or on Call.
4.30 PM: Officially starts the installation process. First announcement is made "Starting phase one".
5.00 PM: Beginning to realize the importance of the project now when who's who of Wellsfargo, whom we've only seen copied in some of the mails, are walking in and wishing us luck for a successful install, which is not a normal practice.
5.15 PM: All the teams are busy in doing their respective jobs. Through out the evening we keep hearing voices like:
"Shutting down servers at Arizona".
"Data Center Cassie Hill is up and running".
"All the servers are in sync now".
"Application is up on first set of servers. Rotating servers now".
This would be a daily job for these guys, but all this sounds fascinating to me.

5.30 PM: Things are going smooth..I go for a coffee break.
6.00 PM: Me still out of office..actually on a long walk..thinking about the things that are coming to an end with this project..long working hours..late night stays in office..my stay in US..n so on.
6.30 PM: Reach back to office. Two people are playing golf outside the War Room..practicing a putt..I walk in with a smile..things must be really going smooth. Dinner is already served.

7.00 PM: The application starts coming up partially on Production (web). Everyone is cheering for every new page that comes up.
7.30 PM: Business Users start testing the application.
8.00 PM: The whole application is up and seems to be working fine. Heard a voice "Routing live traffic to the new application now". Everyone's excited.

9.30 PM: Live Users are hitting the application across the globe..everything is working fine. There is a minor issue of some user applications not reaching the back end system, but the number is too low..we start analyzing the problem.
10.00 PM: Got mail from Business Manager saying the application has been successfully installed on Production on 6/12 and thanking hundred odd people from different groups involved.
11.00 PM: The teams done with their job start leaving. We are also thinking of leaving as we are supposed to be in office early next morning to monitor the application for any post-launch issues.
11.30 PM: Phase two starts, which involves starting the backup servers. We are still analyzing the issue of some user applications not reaching to the back end system.


Date: Friday June 13, 2008
Location: War Room

12.00 AM: Date changes and maybe our fate also. The number of lost user applications has increased in the past couple of hours..total 38 till now..and it has raised alarm considering that these are off-peak hours. Various teams are on hot line now to trouble shoot the problem.
12.15 AM: Not able to find the root cause yet. Time seems to be ticking faster. If there has to be a back out, it must start soon as normal traffic would start coming in within 3-4 hours, from the East Coast users who are three hours ahead of us.

12.30 AM: No concrete information available yet. Back out seems unavoidable.
12.45 AM: Top management is on call now to make Go/No-Go decision.
1.00 AM: Business is not ready to go ahead with the issue in Production. Heard the words, which we never wanted to, "Backing out all the changes. Start phase one now".

1.30 AM: Only teams required for back out are on call now. Other people are sitting around..tired of whole day's work..chatting..
Manager 1: "Accha hua meri biwi India gayi hui hai" (As it is already very late for a married man to be out).
Manager 2: "Accha hua meri biwi abhi Canada mein hai"
Me: Just popped in and said "Accha hua meri biwi hi nahin hai"
Both the managers turned and gave me that is-it-the-time-to-crack-that-stupid-joke look and I was like ..looking somewhere else.

3.00 AM: Back out is successful. Traffic is routed back to the old application. We leave the office.

3.00 PM: Just came out of Fault Analysis meeting. Relieved to know issue is not on our side, but on one of the back end systems that we are using. Also, out of the 38 lost applications, 13 have reached to the back end system. Still analyzing the remaining ones.

6.00 PM: Got confirmation that all the 38 (supposedly) lost applications had reached to the back end system. It was a false alarm and the back out was not at all required.
Come Monday and it'll start a series of meetings, escalations n blame games.

What an end to the efforts that we've put-in in the last four months!!

Though ended on a sad note, but it has been quite an experience to see the launch of an application. Inside the War Room, I was feeling as if I was a part of a team launching a rocket. Saying that I don't mean that I've been a part of a rocket launching team before and considering the kind of trial n error coding I do, it is in national interest that I shouldn't be, in future too.



P.S. - I've tried to keep it as non-techie as possible, but there are some technical terms, which can't be avoided. Believe me, we use these terms in our normal conversation and they are not just thrown in to make it all high sounding.

P.P.S. - I know some people are scratching their heads.. thinking what was all this about..at the end of it, we are just adding a few more pages on the already overcrowded web. I'd have agreed with you, but they are paying me $$ only for doing this.

P.P.P.S. - I know most of you are thinking that had I spent half the time, which I spent in writing all this, in actually working on the project, we might not be in this mess.