AskTom

Hi,

my OMS suddenly stopped working. I wasn’t able to upload data from any monitored servers.
Agents told me that everuthink is ok but number of pendind messages were quite high.

oracle@db01:~> /opt/app/oracle/product/10.2/agent10g/bin/emctl status agent
Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
Copyright (c) 1996, 2007 Oracle Corporation.  All rights reserved.
—————————————————————
Agent Version     : 10.2.0.4.0
OMS Version       : 10.2.0.5.0
Protocol Version  : 10.2.0.4.0
…
Started at        : 2009-10-25 02:14:15
Started by user   : oracle
Last Reload       : 2009-10-26 13:21:34
Last successful upload                       : 2009-12-07 11:34:01
Total Megabytes of XML files uploaded so far :  1343.77
Number of XML files pending upload           :     1916
Size of XML files pending upload(MB)         :    46.33
Available disk space on upload filesystem    :    39.89%
Collection Status                            : Disabled by Upload Manager
Last successful heartbeat to OMS             : 2009-12-07 11:33:13
—————————————————————
Agent is Running and Ready
oracle@db01:~> /opt/app/oracle/product/10.2/agent10g/bin/emctl upload agent
Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
Copyright (c) 1996, 2007 Oracle Corporation.  All rights reserved.
—————————————————————
EMD upload error: Upload was successful but collections currently disabled - disk full

Other corresponding messages:
EMD upload error: Upload timed out before completion.
EMD upload error: Failed to upload file D0096435.xml: HTTP error.
Response received: ERROR-100|No space left on device

I found a lot of core files in OMS_HOME:
-rw——-   1 oracle oinstall 16265216 2009-12-07 14:22 core.24493
-rw——-   1 oracle oinstall 16265216 2009-12-07 14:22 core.24533
-rw——-   1 oracle oinstall 16265216 2009-12-07 14:22 core.24582
-rw——-   1 oracle oinstall 16265216 2009-12-07 14:22 core.24588
…

Solution:
I fount the problem. My access_log in Apache had more than 2GB size.
oracle@beta:/opt/app/oracle/product/10/oms10g/Apache/Apache/logs> du -hs .
2.3G    .
oracle@beta:/opt/app/oracle/product/10/oms10g/Apache/Apache/logs> ls -ltr access_log
-rw-r—–  1 oracle oinstall 2147483647 2009-12-04 21:23 access_log

Kindly delete them and upload data from agents. That’s all.

Metalink:
Problem: Core Dumps: Grid Control OMS generates huge amount of core.xxx files causing the system to halt [ID 419999.1]
How We Corrected a Disabled Agent Due to Disk Full [ID 754458.1]
Agent Upload To Oms Fails With Collections Disabled Due To Disk Full [ID 317242.1]

Regards,

Tom