Oracle is full!
Had my first “issue” with our Oracle server at work today. The archive log destination had filled up. I was aware that I had to keep an eye on this, and I had been actively monitoring its growth. We had one massive logging day the day after launching our new systems at work so I was paying particular attention to it.
We are running Oracle 10G and we had configured the archive log destination to have a capacity of 50Gb. There was more room than this on the volume that we were logging to, but we didn’t want to use the entire drive to the detriment of the other components that also use it. The colume of logs reached 40Gb and I moved a majority of them to another location to clear space on the archive log destination. It turns out that this was not enough for Oracle, or at least not 10G.
Apparently Oracle 10G uses a catalogue of the contents of the archive log, something that I was not aware of. This meant that when I moved the 35Gb of logs that I moved the Oracle instance thought they were still occupying space in the archive log destination. I found a process in the EM that went through and checked all the archived logs against the configured retention policy and listed any that were obsolete or expired. I assumed that this would sort out how much of the archive log destination was being used and make the server see the correct values.
I was wrong.
The server happily kept logging away until it reached its predefined maximum of 50Gb and then it just stopped. Of course it stopped at about 3pm on Saturday, or at least that was when I got the phone call. After some uncertainty I managed to talk our Network Admin through the process of increasing the size of the archive log destination and everything started up again. I was not impressed that Oracle couldn’t tell that the archived logs had been moved, but at least the problem was resolved with as little pain as possible.