I could spend all day trying to reiterate their reasoning for maintaining open connections forever, not re-connecting them, or even their decision to take down their entire application server (not physical server but the windows services that encompass their application) when connections begin to fail.  I’ll allow you all to come to your own conclusions as to the competence of this company that shall remain nameless.

After 4 years of open “TAR’s” with Oracle, numerous consultants, configuration changes, and eventually a plan to migrate these instances to 10g on Linux we believe we stumbled upon the root cause.  We are all aware, I hope, of Windows Perfmon/Performance Counters and how very helpful they can be in giving insight into your running applications.  Well it turns out that Oracle exposes performance counters, but the underlying implementation of these counters login and extract the counter data directly from the target instances.  In order for Oracle to do this it provides a set of configurations for specifying the login, instance information, etc for the counters to use.  Here comes the kicker – that login must be VALID.  Yes, if it’s not valid and you have a tool which try to open these performance counters pretty often like MOM your Oracle process will, over time, leak Virtual Bytes and the Oracle Process will crash, causing a Split Brain.

Long story short, if you’re interested in crashing an Oracle 9i instance, try to login over and over with an invalid login.  Now would you call that a bug?  A pretty glaring one if you ask me.

Here is a link to the configuration settings article which details what we had to do to resolve this issue.