| || |
Performance Eng: WAN Unfriendly app War Stories
OK. We have all seen them. They work well on a fast local network but then some well meaning manager walks through a department and says, "Hey, that looks great. We should have that everywhere..."
And the result is everyone feels pain (except for the few local people). Toss in your stories here of the observed behavior and the source of the problem.
I'll begin. Early in my career I worked for the worlds largest software company as part of their support services organization. The group I was a part of had built a nice little incident tracking system for customer follow up. The situation described above came to be and someone decided this would be good on every support persons desktop.
Out it went. It lengthened every support call, as calls had to be opened through the interface. It was slow. It was Big. It was ugly. It was on a server that was so overloaded it needed to have a person caring for it literally every hour of every day. But it gave great management reports and customer tracking. It made the job of supporting customers more difficult instead of less.
The ultimate cause, design. In a small group it worked to the groups advantage to be able to observe complete histories, see trends, spot consistent issues, but in a large group that large set of data flows at the inception of a support incident, particularly with an overloaded database a couple of timezones away lead to horrible local performance issues. The system even took on a nickname amongst its user base that cannot be repeated in this forum without it being masked out. The nickname very much reinforced the slowness of the system.
It took a lot of system engineering work to rebuild the solution to where it was not such a performance pig. Hardware got faster, the data flows got leaner, the pipes got larger. Eventually the tool got out the way, but only after a lot of production pain.
Lessons learned: Expand through pilots. Examine the requirements and see if these requirements need to be changed before you change the deployment model. Networks are not your friends, so don't be so generous with them. Be kind to your system critical DBAs!
Food for thought on this shortest day of the year.
Replace ineffective offshore contracts, LoadRunnerByTheHour
. Starting @ $19.95/hr USD.
Put us to the test, skilled expertise is less expensive than you might imagine.
Twitter: @LoadRunnerBTH @PerfBytes