Oracle Upgrade
At work we run a dotNet web application over an Oracle Database for our Central Reservations system. When we launched the site we launched the database on a dual 2.4Ghz Xeon system with 6 internal hard drives and 4Gb of RAM.
A little background on the system. The new dotNet application was written to replace an aging and poorly written cold fusion CMS, an equally aged and similarly bad vb6 application.
On top of replacing these applications, were a host of enhancements and new features that could not be done in the old systems.
At first things went really smoothly. The system was humming along, which considering it was an un monitored, unadministered Oracle server was pretty good.
It wasn’t long before the box began to suffer.
At first it wasnt so bad, and we found workarounds for the problems that we were encountering. However it became more and more evident that the only solutions that would permanently fix the problems:
- Revist the code and spend some time fixing up some of the problem areas that had been identified.
- Stop users from connecting to the live database with MS Access to do reports that they had created.
- Spend GOBS of money on new hardware and more Oracle licenses.
Having identified several areas that were affecting performance in the code and in the reports that the users were creating we, the developers, thought that there was no way they would not let us fix some of the issues.
How wrong could we be. Instead of giving us some time to fix some design and implementation issues in the system, upper management went for the “GOBS OF MONEY” option. This is a trend that I have seen time and time again with this company, and whilst it pisses me off to see the money wasted like this, I love the chances that I get to play with cool new toys.
This turned into one of those chances. A few of us went to IBM in Brisbane to check out some cool hardware. This was followed with a variety of different upgrade options being planned out to find the most suitable for our situation. In the end it came down to 2 options:
- Quad processor IBM x460 (or equivalent) with attached fibre channel storage and Oracle Enterprise Server.
- 2 * Dual processor x346 (or equivalent) with attached fibre channel san storage and Oracle Enterprise Server with RAC.
Obviously for toy factor, and to add to my Oracle resume I was keen on the RAC option, but never thought that the $$ would be approved considering the level of investment that would require. However after a protracted negotiation with our contact at Oracle we were able to secure a good price and we went with option 2.
In the intervening time we had installed a couple of DELL servers and we were quite impressed with their performance so we decided to get DELL to quote on the Oracle requirements as well. The price that they came back with was too good to refuse and we ended up going with the DELL servers for the deployment. It also helped that we managed to wrangle 2 free 42U racks from them as well, but all’s fair and all that.
By the time it came time to do the cut across to the RAC install the company was being seriously hindered by the existing system. The issues that we had were causing the system to become almost completely unresponsive several times daily, with the only solution to this a bounce of either IIS or Oracle. The worst that I saw it was the Load on the Oracle server reached a wopping 100 in top. I tried to screen shot it when I saw it, but it dipped back under 100 almost straight away and I missed it.
The whole installation and migration process was fairly straight forward. We only had a few hicups, and 1 seriously annoying bug that we came across that I will post about later. The rollout and switch across also went fairly smoothly, with the exception of all the sequences resetting themselves (kinda my stupid fault, but we wont go into that) and everyone is really happy with the level of performance that we are now seeing.
My only concern is that if we do not get into the code and clean it up, something that is slowly (and I mean really slowly) being done, we will be at a similar position in about 6 to 12 months from now.