JUST SEEMS A LITTLE SLOW TO ME

Over the past week or two I just had a gut feeling that one of our databases i/o subsystem was performing slower than it should have been. This is Oracle 11gR2 on Linux 64 bit using ASM with San storage (8 HBA’s). The DASD is RAID 5 sliced into 1.1 tb Lun’s. There are 15 of these Lun’s presented to ASM as Volume’s (asm lib was used).

When I found a window of lower utilization on the server we ran some “Read” tests against the disks. We were getting 2.5 gbs/second and peaked at 2.9 gbs/second. That told me that my gut feeling was not coming from the i/o sub-system.

Some digging reveled the following:
/dev/oracleasm/disks> ls -ltrfa
brw-rw---- 1 oracle dba 253, 28 Sep 26 10:04 VOL9
brw-rw---- 1 oracle dba 253, 30 Sep 26 10:04 VOL8
brw-rw---- 1 oracle dba 253, 32 Sep 26 10:04 VOL7
 brw-rw---- 1 oracle dba 253, 24 Sep 26 10:04 VOL6
brw-rw---- 1 oracle dba      8, 29 Sep 26 10:04 VOL5
brw-rw---- 1 oracle dba      8, 31 Sep 26 10:04 VOL4
brw-rw---- 1 oracle dba      8, 33 Sep 26 10:04 VOL3
brw-rw---- 1 oracle dba      8, 19 Sep 26 10:04 VOL2
brw-rw---- 1 oracle dba 253, 22 Sep 26 10:04 VOL14
brw-rw---- 1 oracle dba 253, 23 Sep 26 10:04 VOL13
brw-rw---- 1 oracle dba 253, 25 Sep 26 10:04 VOL12
brw-rw---- 1 oracle dba 253, 26 Sep 26 10:04 VOL11
brw-rw---- 1 oracle dba 253, 27 Sep 26 10:04 VOL10
brw-rw---- 1 oracle dba     8, 20 Sep 26 10:04 VOL1

Take note of the column with the numbers 253 and 8 in it. This is the Agent (Major) number. A 253 is a multi-path’ing device. However, an 8 is a single-path device. We changed these all to 253’s and now the world is a little better place.

Beware if you are RAC & ASM and are planning on upgrading to 11R2

We are starting to use ACFS some and while doing some reading I came across the following information as a Burleson comment and wanted to pass it along. Thx ASM Cluster Filesystem (ACFS) There is a serious vulnerability for Oracle RAC databases noted in the Oracle 11g Release 2 RAC documentation. This note indicates a scenario where it's possible for an entire RAC cluster to fail. This exposure is only for RAC databases that are using Automatic Storage Management (ASM). When using RAC, Oracle has announced that the key clusterware files, the OCR and Votedisks can now be stored in Oracle Automated Storage Management (ASM). However, research by expert DBA's indicate that it may well be premature to use this feature. According to Oracle's documentation, if a single node is lost, then the entire cluster may go down if that node happens to be the master node. The entire cluster only fails if the Oracle ASM instance on the OCR master node fails. If the majority of the OCR locations are in Oracle ASM and if there is an OCR read or write access, then the crsd stops and the node becomes inoperative. In order to identify the OCR master node, search the crsd.log file for the line -- "I AM THE NEW OCR MASTER" -- with the most recent time stamp. Oracle has announced that new installations of 11g Release 2 (as opposed to upgrades) will not allow the installation of the OCR and Votedisks on RAW. So the best solution at this time is to install these files on a shared, non-ASM file system such as OCFS2. Clusterware upgraded to 11g Release 2 may still be on RAW.