AskTom

Hi,

my OMS suddenly stopped working. I wasn’t able to upload data from any monitored servers.
Agents told me that everuthink is ok but number of pendind messages were quite high.

oracle@db01:~> /opt/app/oracle/product/10.2/agent10g/bin/emctl status agent
Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
Copyright (c) 1996, 2007 Oracle Corporation.  All rights reserved.
—————————————————————
Agent Version     : 10.2.0.4.0
OMS Version       : 10.2.0.5.0
Protocol Version  : 10.2.0.4.0
…
Started at        : 2009-10-25 02:14:15
Started by user   : oracle
Last Reload       : 2009-10-26 13:21:34
Last successful upload                       : 2009-12-07 11:34:01
Total Megabytes of XML files uploaded so far :  1343.77
Number of XML files pending upload           :     1916
Size of XML files pending upload(MB)         :    46.33
Available disk space on upload filesystem    :    39.89%
Collection Status                            : Disabled by Upload Manager
Last successful heartbeat to OMS             : 2009-12-07 11:33:13
—————————————————————
Agent is Running and Ready
oracle@db01:~> /opt/app/oracle/product/10.2/agent10g/bin/emctl upload agent
Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
Copyright (c) 1996, 2007 Oracle Corporation.  All rights reserved.
—————————————————————
EMD upload error: Upload was successful but collections currently disabled - disk full

Other corresponding messages:
EMD upload error: Upload timed out before completion.
EMD upload error: Failed to upload file D0096435.xml: HTTP error.
Response received: ERROR-100|No space left on device

I found a lot of core files in OMS_HOME:
-rw——-   1 oracle oinstall 16265216 2009-12-07 14:22 core.24493
-rw——-   1 oracle oinstall 16265216 2009-12-07 14:22 core.24533
-rw——-   1 oracle oinstall 16265216 2009-12-07 14:22 core.24582
-rw——-   1 oracle oinstall 16265216 2009-12-07 14:22 core.24588
…

Solution:
I fount the problem. My access_log in Apache had more than 2GB size.
oracle@beta:/opt/app/oracle/product/10/oms10g/Apache/Apache/logs> du -hs .
2.3G    .
oracle@beta:/opt/app/oracle/product/10/oms10g/Apache/Apache/logs> ls -ltr access_log
-rw-r—–  1 oracle oinstall 2147483647 2009-12-04 21:23 access_log

Kindly delete them and upload data from agents. That’s all.

Metalink:
Problem: Core Dumps: Grid Control OMS generates huge amount of core.xxx files causing the system to halt [ID 419999.1]
How We Corrected a Disabled Agent Due to Disk Full [ID 754458.1]
Agent Upload To Oms Fails With Collections Disabled Due To Disk Full [ID 317242.1]

Regards,

Tom

Hi,

today i wanted to backup our database but I found that samebody delete saem archivelog which wasn’t backed up.
When I tried to backup database I have got an error below.
RMAN-06059: expected archived log not found.
So how to solve this issue?

[oracle@centos53 dbs]$ rman target /
…
connected to target database: ORCL11G (DBID=806862530)

RMAN> backup database archivelog all delete input;

Starting backup at 11-NOV-09
…
archived log +DATA/orcl11g/archivelog/2009_11_08/thread_1_seq_58.275.702372629 not found or out of sync with catalog
trying alternate file for archived log of thread 1 with sequence 58
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of backup command at 11/11/2009 02:50:45
RMAN-06059: expected archived log not found, lost of archived log compromises recoverability
ORA-19625: error identifying file /u01/app/oracle/product/11.1.0/db_1/dbs/arch1_58_699978250.dbf
ORA-27037: unable to obtain file status
Linux Error: 2: No such file or directory
Additional information: 3

Solution
1) crosscheck archivelog all;
2) backup archivelog all skip inaccessible delete input;
3) backup database archivelog all delete input;

It’s done.

Regards,
Tom

Hi,

today when I tried to backup my repository database I have faced strange error.

MAN> backup database
2> ;

Starting backup at Jul 08 2009 08:04:27
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=152 device type=DISK
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
input datafile file number=00005 name=/opt/app/oracle/oradata/emrep/mgmt.dbf
input datafile file number=00001 name=/opt/app/oracle/oradata/emrep/system01.dbf
input datafile file number=00003 name=/opt/app/oracle/oradata/emrep/sysaux01.dbf
input datafile file number=00002 name=/opt/app/oracle/oradata/emrep/undotbs01.dbf
input datafile file number=00006 name=/opt/app/oracle/oradata/emrep/mgmt_ecm_depot1.dbf
input datafile file number=00007 name=/opt/app/oracle/oradata/emrep/tools
input datafile file number=00004 name=/opt/app/oracle/oradata/emrep/users01.dbf
channel ORA_DISK_1: starting piece 1 at Jul 08 2009 08:04:28
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on ORA_DISK_1 channel at 07/08/2009 08:40:54
ORA-19502: write error on file “/backup/alpha/rdbms/emrep/backupset/1gkjjq8c_1_1″, block number 262017 (block size=8192)
ORA-27072: File I/O error
Linux Error: 2: No such file or directory
Additional information: 4
Additional information: 262017
Additional information: 1040383

I did few checks and everything seem to be ok:
- correct rights on directory ( directory exist )
- I run backup as oracle
- There were enought free space on the volume

Solution:
=========
update kernel on OS
update nfutils

I am running on SUSE 10 and db version is 11.1.0.7

Regards,
Tom