Oracle is full!
Had my first “issue” with our Oracle server at work today. The archive log destination had filled up. I was aware that I had to keep an eye on this, and I had been actively monitoring its growth. We had one massive logging day the day after launching our new systems at work so I was paying particular attention to it.
We are running Oracle 10G and we had configured the archive log destination to have a capacity of 50Gb. There was more room than this on the volume that we were logging to, but we didn’t want to use the entire drive to the detriment of the other components that also use it. The colume of logs reached 40Gb and I moved a majority of them to another location to clear space on the archive log destination. It turns out that this was not enough for Oracle, or at least not 10G.
Apparently Oracle 10G uses a catalogue of the contents of the archive log, something that I was not aware of. This meant that when I moved the 35Gb of logs that I moved the Oracle instance thought they were still occupying space in the archive log destination. I found a process in the EM that went through and checked all the archived logs against the configured retention policy and listed any that were obsolete or expired. I assumed that this would sort out how much of the archive log destination was being used and make the server see the correct values.
I was wrong.
The server happily kept logging away until it reached its predefined maximum of 50Gb and then it just stopped. Of course it stopped at about 3pm on Saturday, or at least that was when I got the phone call. After some uncertainty I managed to talk our Network Admin through the process of increasing the size of the archive log destination and everything started up again. I was not impressed that Oracle couldn’t tell that the archived logs had been moved, but at least the problem was resolved with as little pain as possible.
Alistair Said,
April 2, 2005 @ 12:41 am
I’m still surprised that when you moved the logs away, that Oracle didn’t pick up that the catalogue and the file system were not in sync.
What do you suppose a reasonable reason would be for not automatically cleaning the catalogue when logs are moved away? I appreciate that they have the functionality to manually do it, however at the same time; I’m not seeing an obvious reason for a catalogue to say ‘Yes, we have 40Gb of logs present’ when in fact the file system is saying ‘No, we actually have 2Gb at the moment’.
Me. Said,
April 3, 2005 @ 8:17 pm
The processes that can be run manually to clean up the archive log destination and the catalogue can also be scheduled to run. Needless to say this has been done, and we should not see this problem repeat itself any time soon.
The only exception would be if the volume of transactions reaches that of the boxes a friend of mine looks after. Currently they write approximately 2Gb of logs every 15 minutes.
*eeeek*