I'm Production Support... which is why I'm up at 4am.

As I mentioned in my first post I do Production Support for a large company. Unlike "Developer" or "Project Manager" - both widely used and understood titles - this probably needs some explanation. In our case "Production Support" is the team that's called when production applications have problems. I'm one of several "Incident Managers" who manage the incident by engaging dedicated production support resources, other enterprise support teams, application development teams or whoever else is needed to restore service.

My team supports nearly 400 custom applications - both mainframe and distributed. The mainframe is most likely similar to any other mainframe: an illogical progression of code over nearly three decades with tendrils into almost every moving part of the company. In theory we've "standardized" on either AIX/WebSphere or Windows/Dot Net on the distributed side. But that was only recently and of course no corporate money goes to rewriting working applications. So in practice we've got an unholy soup of engines, languages and protocols. If it's ever said "Hello World" it's probably in an application we support.

So my day consists of working up processes, investigating prior incidents and keeping my fingers crossed that things will hold together just long enough for me to get just a little bit ahead. They never do. When incidents arise they may be fast and easy or slow and hard. The easy ones might be sent to another team, have known solutions or not severe enough for major escalation. The hard ones... they can take many forms. There's the "happens every other day for months" or the "must be fixed before we sleep" and even the rare "Holy Shit, if this doesn't get fixed we'll lose millions!"

Incidents often take hundreds of man hours to fully solve - 10 to 30 people on conference calls for literally days on end. Much of the actual boot-on-the-ground work is done by a large offshore team (a mixed blessing at best and something I'll be talking about frequently) and across many offices here in the States. Conference calls backed by instant messaging are the only real option for collaboration and more often than not English is a second language making communication a non-trivial issue.

The work is hard the hours are long and I'm positive that my situation is far from unusual.

I've got Things to say... Do I Have I Time to Write Them Down?

I've been wanting to start a blog for some time now. I have the hubris to think that some of the things that happen to me might be interesting to a select group of people. I'm not sure if I'll force myself to keepup with things (I've enough work that if this starts feeling like it I'll probably stop) but I'm going to try.

Most of the things I'd like to say revolve around my job. Although my family and my PlayStation are definately more interesting to me I doubt they will be to others. But I'm fairly confident that others are doing the same kind of work that I am and I think that we could help each other.

I work production support for a very large financial services company. My team manages production problems on several hundred distinct applications which themselves are run from several thousand individual servers and a sprawling mainframe. We, like most large companies, have dozens of teams and initiatives which, together, keep things running.

Although I've only been in this position for somewhat over a year, I've been working in this industry for over 15 years. I have experience! I have anecdotes! I (think) I (might) have something interesting to say.