Sunday, February 05, 2006

Oracle VMware RAC Q&A!

Replying to Sampl's several emails I realized that I had pointed out at the architecture of 3 500mb disks and 3 asm disks (preferebly 10G all). But given the length of the article and not being able to post everything on it, I gave a general advice on what diss to use for OCFS and what for ASM. Still the print screens did not really come close to it.Sampl was using the article step by step and I saw (he pointed to me actually here step 8 onwards) that I hadn't done justice there. So here I'll post the prints for more clarity.This is a sample *.vmx file from my Windows 2003 RAC

config.version = "7"
virtualHW.version = "3"
scsi0.present = "TRUE"
scsi0.virtualDev = "lsilogic"
memsize = "828"
scsi0:0.present = "TRUE"
scsi0:0.fileName = "vm1.vmdk"
scsi0:0.deviceType = "plainDisk"
ide1:0.present = "TRUE"
ide1:0.fileName = "auto detect"
ide1:0.deviceType = "cdrom-raw"
floppy0.fileName = "C:\TEMP\vmscsi1-"
Ethernet0.present = "TRUE"
usb.present = "FALSE"
displayName = "vm1"
guestOS = "winNetEnterprise"
priority.grabbed = "normal"
priority.ungrabbed = "normal"
powerType.powerOff = "default"
powerType.powerOn = "default"
powerType.suspend = "default"
powerType.reset = "default"

Ethernet0.virtualDev = "vmxnet"
Ethernet1.present = "TRUE"
Ethernet1.connectionType = "bridged"
Ethernet1.virtualDev = "vmxnet"

ide1:0.startConnected = "TRUE"
floppy0.fileType = "file"
Ethernet0.addressType = "generated"
Ethernet1.addressType = "generated"
uuid.location = "56 4d e3 3b ab 45 e1 24-dc 75 db 3e 1a 26 17 70"
uuid.bios = "56 4d e3 3b ab 45 e1 24-dc 75 db 3e 1a 26 17 70"
ethernet0.generatedAddress = "00:0c:29:26:17:70"
ethernet0.generatedAddressOffset = "0"
ethernet1.generatedAddress = "00:0c:29:26:17:7a"
ethernet1.generatedAddressOffset = "10"
tools.syncTime = "TRUE"

disk.locking = "FALSE"
diskLib.dataCacheMaxSize = "0"
diskLib.dataCacheMaxReadAheadSize = "0"
diskLib.dataCacheMinReadAheadSize = "0"
diskLib.dataCachePageSize = "4096"
diskLib.maxUnsyncedWrites = "0"

scsi1.present = "TRUE"
scsi1.virtualDev = "lsilogic"
scsi1.sharedBus = "VIRTUAL"
scsi1:1.present = "TRUE"
scsi1:1.mode = "independent-persistent"
scsi1:1.fileName = "E:\shared\ocr.vmdk"
scsi1:1.deviceType = "plainDisk"

scsi1:2.present = "TRUE"
scsi1:2.mode = "independent-persistent"
scsi1:2.fileName = "E:\shared\votingdisk.vmdk"
scsi1:2.deviceType = "plainDisk"

scsi1:3.present = "TRUE"
scsi1:3.mode = "independent-persistent"
scsi1:3.fileName = "E:\shared\spfileasm.vmdk"
scsi1:3.deviceType = "plainDisk"

scsi1:4.present = "TRUE"
scsi1:4.mode = "independent-persistent"
scsi1:4.fileName = "E:\shared\asm1.vmdk"
scsi1:4.deviceType = "plainDisk"

scsi1:5.present = "TRUE"
scsi1:5.mode = "independent-persistent"
scsi1:5.fileName = "E:\shared\asm2.vmdk"
scsi1:5.deviceType = "plainDisk"

scsi1:6.present = "TRUE"
scsi1:6.mode = "independent-persistent"
scsi1:6.fileName = "C:\shared\asm3.vmdk"
scsi1:6.deviceType = "plainDisk"

scsi0:0.redo = ""
scsi1:1.redo = ""
scsi1:2.redo = ""
scsi1:3.redo = ""
scsi1:4.redo = ""
scsi1:5.redo = ""
scsi1:6.redo = ""

Ethernet0.connectionType = "hostonly"

And this is the configuration of my disks. Check out the prints here.

Six Disks all laid up as I mentioned in the architecture

Here this Q drive is being formatted with OCFS, and used for OCR and Votingdisk storage

Here you can clearly see that the OCR primary is grayed out so we pick the other 500mb vmdk disk and use it for OCR mirror (1 is enough for OCR mirror) and for placing another copy of Votingdisk

And here you can see that the third 5oomb disk (which we will primarily use for our ASMs spfile, we will put yet another copy of, the installer asks us for the third one)

So clearly as you can see above all the OCFS formatted 500mb disks are assigned with a Drive letter AND you can see the format type Data(CFS). The rest remains untouched. This we pick on as we go about installing our ASM instance and RAC instance using DBCA.

Another user had problems with clusterware not being able to see the disks. I suggest that you go follow these steps to install the disks and leave them as raw.

PS: Blogger is having some trouble with file uploads so I'm using my own gallery here. I'll try Flickr later for some prints on disk creation.

PS2: Looks like they moved to more powerful servers :-)


Aamer said...

Hi Tarry,

I am facing an odd error during CRS install. CRS assistant fails with the folowing error after having formatted the OCR and Vote disk drives with OCFS.

Step 5: Starting up CRS stack on all nodes
rac2 service OracleEVMService in improper PENDING state, err(997)
rac1 service OracleEVMService in improper PENDING state, err(997)

On the remote node Both OracleEVM and OracleCS services are started but on the local Node OracleEVM service seems to hangs. Plus both nodes start to reboot after the failed CRS installation.

Is there any known workaround for this Issue?



Aamer said...

Any ideas about that OracleEVM service failing folks!

Tarry said...

Ok Aamer,

So you had no memory problems/limitations? And checking the cluvfy.bat before installation was also fine?

Aamer said...

I had no memory related issues during the pr installation check performed with cluvfy.bat. The only check which failed was for VIPs. I have dedicated 524MB RAM for each of my VMware machines and 512MB is the minimum requirement I believe.

Tarry said...

I'm not sure if the mem limit is 512Mb, Aamer but it's a lot below the acceptable level.

The OUI will giev you warnings of low mem.

Can you try to install it on a machine with some more capacity? Also since the VMware Server (a free version of GSX server) has been set free. Can you try it? (I'm assuming that you're using Workstation)

oraclRAC said...

Hi Tarry,

I was able to install RAC successfully. Thanks a lot for all the help and guidance.

When I was installing Database, suddenly the node from which I was installing crashed i.e., I got a blue system crash error. But the error said that the system has crashed because of some newly installed hardware or software, and to remove it. It said that if it was the first time, then I could restart.

I restarted and it was working fine. The database was also installed properly and I was able to continue with the other installations.

Do you know why I got the error? I got the same error on the same node twice while installing Database and DBCA. Both the times I restarted and it worked fine. Is it going to cause any problem?

Tarry said...

That is great news. The "blue screen" of death could also be an issue with VMware and Windows installation.

What version is it 2003 standard or enterprize?

By the way, did you try getting the free VMware Server? It's the free GSX version? Try it out :-)

elshaerr said...

Hi Tarry,

I have upgraded VMWare workstation to 5.5.1 and setup the W2003 Nodes as you have recommended. First when I start the workstation, I get a message "Clustering is not supported for VMWare Workstation. This setting will be ignored" I suppose that says it all, but just to go forward with the 10GR2 RAC install, it will confirm the previous message with an Oracle install error "Error in creating directory Q:\csdata\crs" and I am locked her. Do you have any suggestions to go forward.
Alwyse thanks for your help.


Tarry said...


How about these settings

scsi1.present = "TRUE"
scsi1.virtualDev = "lsilogic"
scsi1.sharedBus = "VIRTUAL"

The last line says about a typical virtual HBA(Host Bus Adapter) and this should allow the disks to be shared.

Also I've experienced several times if I restarted the machines together then many of my shared disks settings changed from

scsi1:1.present = "TRUE"
scsi1:1.mode = "independent-persistent"
scsi1:1.fileName = "E:\shared\ocr.vmdk"
scsi1:1.deviceType = "plainDisk"


scsi1:1.present = "FALSE"
scsi1:1.mode = "independent-persistent"
scsi1:1.fileName = "E:\shared\ocr.vmdk"
scsi1:1.deviceType = "plainDisk"

Try restarting machines one by one.

oraclRAC said...

Hi Raji,

When I was installing RAC, initially I had faced with the same problem i.e., "Error in creating directory Q:\csdata\crs"

In the "Specify Disk Configuration" window, if I tried to change the drive letter (i.e., to Q:, R:, S: as in the sample given by Tarry) other than the default drive letter that is assigned, I used to get this error. If I accept the default drive letters, I was able to continue with the installation successfully. Try this out.

orclrman said...

Hi Tarry,

I'm trying to install Oracle database v10.2 on Raw devices. I am using Windows 2000 Professional on VMware gsx 3.2.0.

To create raw devices I followed the same procedure as you had given for creating Raw devices for ASM during RAC installation. After creation, I made them RAW through Disk Managment. I rebooted the system. (In Windows 2000, so automount is enabled by default)

The Oracle software is installed in c:\ The ORACLE_BASE and ORACLE_HOME has correct paths.
When I run the dbca, I choose the option Raw devices for storage type. As I continue with the installation, it fails during "Raw device Validation Check" for pfile and again for control file.

It is taking the Path: {ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/
I tried to change the path to: \Device\Harddisk9\Partition1\
But still it is giving the same error.

1. I would like to know if the way I'm creating Raw device is correct?
2. Does the Raw device mean the same as for ASM?
3. How should I install the database using Raw devices option?

Please reply as soon as possible.

Thank you

sunil said...

Step 5: Starting up CRS stack on all nodes
rac2 service OracleEVMService in improper PENDING state, err(997)
rac1 service OracleEVMService in improper PENDING state, err(997)

OracleEVM and OracleCS are not started on any of the node and when i try to start them it gives me an error 1053.
Any one with any idea highly appriciated



preallocate all space to disks.

give same passwords to both machines.

i think all other is ok...