Database

Flashback: The Lesson I Learned

Michael Mikhailidi

• 2 min read
Flashback: The Lesson I Learned
Photo by Markus Spiske / Unsplash

This piece is not about features or products. It's about lessons that we learn or not. The one I learned some twenty-three years ago was: "Don't ask for troubles and do one thing at a time."

At the end of the last century (I love how it sounds), I worked as a lead Oracle DBA for the big regional bank back in my home country. With the proper training from Oracle University and extensive hands-on experience, I knew everything until the server migration weekend and the revelation Monday.

To set a stage,  it was a time when businesses realized that you could move from proprietary and costly platforms to Intel machines and save big-time on support, licenses, and spare parts. So we have tested the newest Oracle 8i release on an Intel box with SCO Unix and reported to the CIO that we are happy with the performance. Banks didn't run 24/7; still, nobody wanted to go through the series of changes. There were platform compatibility issues, so we decided that we would finish everything and be ready for the following Monday if we start Saturday night by Sunday afternoon. By everything, I mean:

  • Move database files from AIX partitions to the SCO Unix files.
  • Upgrade database from 7.3.4 to 8i
  • Upgrade banking core application to support new database version and address a few PL/SQL compatibility issues.
  • Upgrade client applications and database clients to match database version

The migration went smoothly, and we were happy with the smoke test results. I can't say the same about our ~ 200 clerks and managers because the system hung dead, and nobody can't do anything. Operations went to the manual mode; we tried to figure out what was happening with an unknown operating system and not many hints from the database or system logs. We have fallen back to the original servers; the bank worked late-night hours to catch up on the missing operations and send out regulatory reports.

Tuesday morning, I got a call from the CIO's office. He summoned my manager and me for the rundown. First time in my life, I expected to be fired on the spot. But, to my surprise, the meeting was productive and professional, and that was the place and time where I learned my main lesson:

Set big goals but make small steps.

Since then, I don't make too many changes unless I have no other choice.