Tip of the Iceberg

Category domino
Chris, Nathan and I were discussing Worst Practices earlier today, and I was reminded of a bonehead approach I used maybe five or six years ago. I may submit this one to the site, but in the meantime here it is for your bemusement.

At the time, I was supporting roughly 250 different applications, many of which contained "action items" in one form or another. Many of these also included a daily scheduled agent to send reminders to assignees to close items that were overdue... and they'd send one email per item. As a result, most users just ignored the notices because they were so inundated by them. In fact, at one point a user reported that he wasn't receiving certain messages that he'd been expecting; we found them in his junk mail folder because he'd gotten so many action item notices that he added the user we signed all mail agents with to his "Block Mail From" list.

Finally, one of the users came up with a brilliant idea: centralize the notification. One email per week, listing all open items. It was easy enough - set up a database to store application profiles (defining application-specific criteria for inclusion in the report, which fields represented the assignee, due date and brief description, etc.) and an agent to do the scanning and send the notices. To get a feel for how the user community would respond, we disabled the daily notification from 8 or 9 databases and added report profiles for them to the new centralized process. The response was undeniable: from then on, every week there'd be a flood of items closed the day the report went out, because people were finally paying attention instead of being inundated. So, every few weeks we'd absorb another database's notifications into the centralized report. Happy ever after......

......until I started hearing from a couple users saying they hadn't gotten their report that week. I looked into it, and could tell that the notices were going out. Asked a few others if they'd received theirs, and they had. At the time it didn't occur to me that the people who had received the report had last names ending in B or C and the ones who hadn't had last names ending in V or W... see where this is heading? As we added more and more databases to the report, less and less people would receive a copy each week. Finally, a pattern was obvious: only the first half of the alphabet was being processed. The agent was exceeding the time limit. Out of sheer morbid curiosity, I bumped the off-hours time limit to 6 hours... the agent completed successfully in just over 4. Ew. Having not needed to touch the code since its original creation, I glanced at it to see why it was taking so long to run, and it was immediately apparent (sweet, sweet hindsight) why this agent didn't scale for crap: it was way too loopy.

There were about 900 people in the address book, and 38 application profiles in the report database. The code was looping through the address book, and for each user it would loop through the application profiles. For each profile it would get a handle on the corresponding database and loop through its items, checking each to see if the user was the assignee. Since each application probably had an average of about 50 open items at any given time, that's roughly 1.7 million document scans. No wonder it was taking so long.

I know there are plenty of developers who still don't use classes because they've never seen a need for them, but in this case the benefit of switching to a class structure was painfully obvious. I refactored the code to scan each database once. For each item, if the assignee already had a report in progress, it would append the item data to it; otherwise it would start a report and then append the item data. Once it had scanned all the items, it would do one last loop through all the in-memory reports, adding signature information to each and sending them to their respective recipients. From then on, instead of taking 4+ hours to run, it took about 2 minutes.

In retrospect, I'm glad that happened, because ever since that incident I've been more sensitive to how drastic the difference can be when code is working harder than it should have to.