I'm at CodeMash today (stop by the Inedo booth if you're there!), so I thought it'd be a great time for this classic. Rutherford, Price, Atkinson, Strickland, and Associates Dentistry, Inc was originally published on January 30th, 2008.
Although the partnership talks had begun years before, July 19th was their agreed-upon D-Day, wherein the four separate practices would officially combine to be Rutherford, Price, Atkinson, Strickland, and Associates Dentistry, Inc. In the months leading up to D-Day, and after much bickering and debate, the four dentists got everything ready from new signage to new logoed toothbrushes. The only thing that remained was combining their computer systems. That task was left to Aaron B, an IT consultant who had the pleasure of working with each office through many of the “ugly years.”
Fortunately for Aaron, each of the dentists used the same practice management system: Beaglesoft’s Practice EnterprisePlus. It certainly wasn’t the best software, but it was among the most expensive. Perhaps more-importantly, Beaglesoft offered all sorts of outrageously-priced add-ons that the dentists could buy to one-up one another. For example, a wand-shaped oral camera required a $7,000-per-site “camera driver,” in addition to the ungodly amount the camera cost in the first place. When Aaron plugged the camera into his laptop (which didn’t have any Beaglesoft software running), it was recognized as a plug-and-play camera and immediately started streaming video. Not that it mattered though; as soon as Dr. Price had his installed, the other three dentists had to get one as well.
“While our prices might seem high,” a Beaglesoft rep once told Aaron, “keep in mind that you’re paying for quality. Our products are rigorously tested to work in today’s and tomorrow’s high-tech dental office.”
And for that reason, Aaron wasn’t too worried about Beaglesoft’s portion of the D-Day migration. They assured him on several occasions that their latest and greatest – Beaglesoft Practice EnterprisePlus Elite with Networking – could network a “virtually unlimited” number of practices. The four Aaron was linking together was “chump change” compared to what the system could do.
When Friday, July 16th – the weekend before D-Day – had finally come, the dentists were ready. They closed their offices at noon and, per Beaglesoft’s instructions, initiated the migration process. Over the next twelve hours, so the plan went, each practice’s system would upload its data to the Central Server at Dr. Strickland’s office. Naturally, none of the other dentists were too thrilled about having a “Central Server”, especially one at Dr. Strickland’s.
Aaron arrived at Dr. Strickland’s office early Saturday morning to find a surprising message on the server: “Migration Completed Successfully.” He ran through some initial smoke tests and it appeared that the migration did, in fact, complete successfully. After a trip to the other three dentist offices, Aaron verified that he could access any patient’s file from any office. He called up the four dentists to share the good news: come Monday, they should be in business.
Monday came and, shortly thereafter, the four offices were out of business. The system had completely grinded to a halt. Every click of the mouse was met with a several-minute delay, and every delayed response was met with more clicking. Aaron, who happened to be on-site “just in case,” immediately suspected the newly-installed T1 lines.
Aaron called up the phone company. They ran a few diagnostics on their end, only to find that each office’s T1 line was completely pegged. Most certainly, the technician claimed, the problem was on their end. Perhaps a router gone haywire?
Aaron checked and rechecked the switches, the hardware, the ports, and the routers. He rebooted once, twice, and thrice. Everything seemed functional, aside from the fact that the Central Server was firing packets off non-stop.
Not sure what else to do, Aaron bridged his laptop between the Central Server and its switch. Within seconds, he logged hundreds of megabytes of data, far too much for anyone to go through in the middle of such a crisis. He had no choice but to take the “satellite” dentists offline to investigate the problem. They grew suspicious of this and, of course, Dr. Strickland, and demanded that there was foul-play involved.
With only a couple users accessing the Beaglesoft system, Aaron was able to get a handle on the traffic. As he assuaged the other dentists over the phone, Aaron noticed that a lot of the data seemed to be coming from SQL Server. Specifically, it was from queries like this:
SELECT * FROM Patients
Digging further, Aaron figured out that, whenever a user wanted to look up a patient, the program would run “SELECT * FROM Patients” query, returning the entire Patients table to the client computer.
What’s worse, the query would run any time a character was typed in the patient search box. Searching for just his first name – A-A-R-O-N – resulted in five SELECT * queries.
What’s still worse, the same method of client-side filtering was used for appointments. It wouldn’t just get, say, today’s appointments. Or this week’s. Or, say, any that haven’t happened yet. It would query for every appointment that they ever had or would have in the future. That’s about 100,000 rows.
And since each appointment involved a patient, it’d have to fire off queries for each appointment to download and filter information about the patient.
It was apparent that Beaglesoft’s “rigorous testing” of “Practice EnterprisePlus Elite with Networking” involved, perhaps, a single computer and two, maybe three patient records. He immediately called Beaglesoft to report their issues and a demand a resolution.
(an actual screenshot from Beaglesoft’s install directory)
Within a few hours, three of Beaglesoft’s finest were on a plane to New Portlandopolis. When they arrived, two of them split off to work on “de-migrating” the system into the original four databases. The other Beaglesoftie, a product manager, worked on “damage control” – and boy did she have a lot of damage to control.
By that time – ten hours into Rutherford, Price, Atkinson, Strickland, and Associates Dentistry, Inc’s first day – the dentists were at each other’s throats. Dr. Price blamed the mess on Dr. Strickland who was “online the entire time, ” while Dr. Strickland was convinced that Dr. Atkinson had somehow “spiked the T1s,” while Dr. Rutherford believed that Dr. Price “wanted to retire, and was bringing everyone else down.” Eventually, Dr. Atkinson stormed out and tore down the new "Rutherford, Price, Atkinson, Strickland, and Associates Dentistry, Inc" sign. As he stomped the sign into pieces, he vowed never to work together again. Things pretty much went downhill from there.
After things cooled down a bit with the dentists, the product manager met with Aaron. Angered that his future prospects looked like a repeat of the “ugly years”, he lambasted Beaglesoft’s latest and greatest, and asked why, oh why, they couldn’t have done some client-side caching. Or, at the very least, use the magical WHERE clause.
She was astonished by Aaron’s technical knowledge and eagerly asked more questions on “WHERE clauses and other optimization techniques.” Near the end of their conversation, she actually offered Aaron a job as a Lead Developer at Beaglesoft.
Aaron ended up declining the position. He figured that they’d never be willing to tar and feather the existing development staff. That, and after the Beaglesoft Fiasco of 2004 (as it’s called today), he’d have a lot of cleaning up and intra-dentist diplomacy to do. Besides, how could he miss taking part in the latest exciting chapter in the dentistry history of New Portlandopolis?
No comments:
Post a Comment