It's free, but it does not rock

The problems ESPN has been having with its 2007 incarnation of fantasy baseball may go down in the software testing hall of infamy.

I've been playing in ESPN leagues since college with my collegiate buddies, and up until this year it had been a pay service. We stuck with it out of habit, despite free offerings from Yahoo and CBS Sportsline; despite the cost, it was always fairly reliable. Their player listings were up to date, and the website always behaved the way I expected it to.

In an apparent bid to complete with the free games, ESPN decided to jump on the free bandwagon, while at the same time giving their system a major overhaul. There was a whole lot of publicity and ballyhoo, including a funny and expensive-looking ad campaign featuring, among other things, Peter Gammons in a wig. And the look of the new site is great; it's well-designed and sleek, it largely makes sense, and it takes advantage of modern browser technologies, with real-time updates and a tab-based user interface. I had a couple issues with selecting the wrong player during the draft, but the interface is only partially to blame there.

Other than that it all looked pretty solid, until the season rolled around. Some of the bugs I noticed (and I'm sure I didn't hit them all):

- On opening day, roster changes were locked; there was no way to activate players for the following day(s). As a result there was no way to pick up free agents.
- Players who were dropped from a roster are supposed put on waivers for a couple days; if nobody else claims them, they become free agents. In our league, dropped players were never clearing waivers; days after they were supposed to be free agents, they were still marked as being on waivers.

Other issues reported here and there on the intertubes: there were persistent problems with roster moves, live scoring updates, and general site accessibility throughout the first week of the season. Some leagues were not seeing the players they drafted appear on their rosters. There may have even been other problems that haven't been publicized (ESPN hasn't provided a detailed description of all the bugs they've fixed since the start of the season, just that there are "problems").

Earlier this week, ESPN decided it had to take a nuclear bug-fix option: it rolled everyone's roster back to opening day, and made all scoring retroactive to that active roster only. In other words, all transactions were wiped out, so any points gained or lost by roster moves that players had made between the start of the season and April 12 were erased. Which really hurts if you're the kind of player who spends a lot of time adjusting his team (like, um, me... sometimes). I had made a few moves to shore up weak spots in my roster, and they've been wiped out by ESPN's time traveling shenanigans.

Supposedly it wasn't the increased user traffic that caused the problems. ESPN hasn't said how many more players it has had to accommodate since going gratis, but traffic to the site never seemed to slow it down. The problems instead appear to stem from data processing glitches. Which begs the question: what kind of testing did they do before the season?

A fantasy baseball game on the scale of ESPN's has quite a few data processing hurdles to overcome:
- There are 8-12 teams per league, which adds up to thousands of different teams and roster combinations
- There are hundreds of baseball players available to choose from, and the pool of available players changes during the season due to injuries, call-ups from the minors, etc.
- The system has to keep up with all baseball players' statistics, (in real time)
- Relevant fantasy team statistical totals are calculated (in real time)
- League standings are updated based on these statistical totals (in real time)

It was the real-time stuff that was new to ESPN this year, but I don't think that's what was causing the issues, either. In previous years, you could get a real-time box score for your team, but the league standings were not updated live; they were processed late at night after all the day's action had completed. This year, with the addition of real-time updates, the standings appeared to be updating accurately during the day; you could even leave the standings or team box-score page up, and it would update without having the refresh the page.

Instead, the bulk of their processing problems appeared to come from a simple lack of good functional testing. I can only speculate without any details or some kind of public post-mortem on what went wrong, but it sure looks like they didn't bother or just didn't have time to simulate a real season. When nobody can even make a roster move on the first day of the season, and players never clear waivers on the date they're supposed to, that's a pretty good sign to me that they didn't adequately test the system using simulated data and players.

And so you see why my chosen profession, software tester, is something like the red-headed stepchild of software engineering. It's important, because you can catch high-publicity problems like these before an angry public encounters them. But it's often ignored or blown off; after all, you could work as a tester on a big system like ESPN's fantasy baseball, have an infinite amount of time to work on it, and still never catch all the bugs. That makes scheduling time and resources for testing difficult, especially when you're trying to push out a flashy, highly publicized new system under a tight deadline.

It doesn't help that one of the Web 2.0-type philosophies toward testing seems to be, "let the users do it." The teams that create web-based applications tend to not devote a ton of resources to dedicated testers; this is perhaps one of the reasons why GMail, which was originally introduced three years ago, is still marked as being in "Beta" (implying there's still more testing to do).

I'm really curious about what kind of testing ESPN did. Hopefully we'll find out in some kind of ugly, public tell-all. Those are the best kinds. In the meantime, I can't believe I'm stuck with Jorge Julio, AGAIN. I keep trying to drop Jorge Julio, but THEY KEEP PULLING HIM BACK IN.