AskTom

Hi,
today one of databases stopped. The reason was that ASM disk group were out of space.

Alert.log
=========
Mon Mar  1 11:57:45 2010
Errors in file /opt/app/oracle/admin/replica1/bdump/replica1_arc1_15884.trc:
ORA-19816: WARNING: Files may exist in db_recovery_file_dest that are not known to database.
ORA-17502: ksfdcre:4 Failed to create file +DATA
ORA-15041: diskgroup space exhausted

Sqlplus:
========

SQL> select a.name DiskGroup, b.disk_number Disk#, b.name DiskName, b.total_mb, b.free_mb, b.path, b.header_status
2  from v$asm_disk b, v$asm_diskgroup a
3  where a.group_number (+) =b.group_number
4  order by b.group_number, b.disk_number, b.name
5  /

DISKGROUP       DISK# DISKNAME                 TOTAL_MB      FREE_MB PATH       HEADER_STATU
————— —– ——————– ———— ———— ———————————– ————
DATA 0 DATA_0000                 148,256            0 /dev/raw/raw1                       UNKNOWN
***************                            ———— ————
sum                                             148,256            0

SYS@replica1 SQL> show parameter recovery

NAME                                 TYPE        VALUE
———————————— ———– ——————————
db_recovery_file_dest                string      +DATA
db_recovery_file_dest_size           big integer 106290M
recovery_parallelism                 integer     0

SOLUTION:
=========
backup and delete ald archivelog by RMAN command

RMAN> backup archivelog all delete input;

Starting backup at Mar 01 2010 11:59:06
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
…

ls -ltr
…
-rw-r—–  1 oracle oinstall 1612922880 2010-02-17 10:16 38l67qh7_10_1
-rw-r—–  1 oracle oinstall   24561664 2010-02-17 10:16 39l67uoj_1_1
-rw-r—–  1 oracle oinstall 2147483648 2010-03-01 12:05 3cl7fmcu_1_1
-rw-r—–  1 oracle oinstall 2147484160 2010-03-01 12:05 3cl7fmcu_2_1

Regards,

Tom

Hi,

there are two ways how to get information about disk group usage.

Sqlplus
——–

SQL> set lines 255
SQL> col path for a35
SQL> col Diskgroup for a15
SQL> col DiskName for a20
SQL> col disk# for 999
SQL> col total_mb for 999,999,999
SQL> col free_mb for 999,999,999
SQL> compute sum of total_mb on DiskGroup
SQL> compute sum of free_mb on DiskGroup
SQL> break on DiskGroup skip 1 on report -
>
SQL> set pages 255
SQL>
SQL> select a.name DiskGroup, b.disk_number Disk#, b.name DiskName, b.total_mb, b.free_mb, b.path, b.header_status
2  from v$asm_disk b, v$asm_diskgroup a
3  where a.group_number (+) =b.group_number
4  order by b.group_number, b.disk_number, b.name
5  /

set lines 122
set pages 66
DISKGROUP       DISK# DISKNAME                 TOTAL_MB      FREE_MB PATH       HEADER_STATU
————— —– ——————– ———— ———— ———————————– ————
DATA                0 DATA                      238,747       37,328 ORCL:DATA  MEMBER
***************                            ———— ————
sum                                             238,747       37,328

ASMCMD tool
———–

oracle@antarctic:~> asmcmd
ASMCMD> lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB Req_mir_free_MB  Usable_file_MB  Offline_disks  Name
MOUNTED  EXTERN  N         512   4096  1048576    238747    37293 0           37293              0  DATA/

Regards,

Tom

[root@localhost /]# /u01/app/oracle/product/11.1.0/asm/root.sh
Running Oracle 11g root.sh script…

The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME=  /u01/app/oracle/product/11.1.0/asm

Enter the full pathname of the local bin directory: [/usr/local/bin]:
Copying dbhome to /usr/local/bin …
Copying oraenv to /usr/local/bin …
Copying coraenv to /usr/local/bin …

Creating /etc/oratab file…
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
/etc/oracle does not exist. Creating it now.
/u01/app/oracle/product/11.1.0/asm/bin/crsctl.bin: error while loading shared libraries: /u01/app/oracle/product/11.1.0/asm/lib/libnnz11.so: cannot restore segment prot after reloc: Permission denied
/u01/app/oracle/product/11.1.0/asm/bin/clscfg.bin: error while loading shared libraries: /u01/app/oracle/product/11.1.0/asm/lib/libnnz11.so: cannot restore segment prot after reloc: Permission denied
Configuration for local CSS has been initialized
Failed configure CSS. Fix the problem and rerun this script from destination Oracle home with ‘reset’ argument
Finished product-specific root actions.
[root@localhost /]# cd /u01/app/oracle/product/11.1.0/asm/bin/
[root@localhost bin]# ./localconfig delete
./localconfig: line 778: /etc/init.d/init.cssd: No such file or directory
Cleaning up Network socket directories

SOLUTION:

While installing CentOS 5 linux if you haven’t paid much attention then chances are you probably have missed window where installation program ask to enable/disable SElinux.
SElinux can run as enforcing or in permissive mode. If in /etc/selinux/config file selinux is set as permissive or enforcing then change it to disabled. You would need to reboot your CentOS server to disable selinux. This change wouldn’t work without rebooting your linux box.

[root@localhost bin]# grep SELINUX /etc/selinux/config
# SELINUX= can take one of these three values:
SELINUX=enforcing
# SELINUXTYPE= type of policy in use. Possible values are:
SELINUXTYPE=targeted

You must disable SELINUX.

Correct
[root@localhost bin]# grep SELINUX /etc/selinux/config
# SELINUX= can take one of these three values:
SELINUX=disabled
# SELINUXTYPE= type of policy in use. Possible values are:
SELINUXTYPE=targeted

Regards,

Tom