BACKUP Failure - TAPE Issue

When I run Backup,

i got the following error.

Starting backup at 
executing global script: ll1
current log archived
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03015: error occurred in stored script ll1

Additional information: 2
ORA-27211: Failed to load Media Management Library
ORA-19554: error allocating device, device type: SBT_TAPE, device name: 
RMAN-03002: failure of backup command at 


when I check,

]$sbttest test
The sbt function pointers are loaded from oracle.static library.
libobk.so could not be loaded. Check that it is installed

Backup has not happened after database upgrade.

Solution:


After upgarde DB will have new home.

In the new home path,  libobk.so should be linked to backup vendor supplied rman library.

Then we have made a link,

]$ cd $ORACLE_HOME/lib
]$ln -s /usr/lib/libora.so libobk.so


Now if i check,

]$ls -lrt lib*
lrwxrwxrwx 1 oracle oinstall     libobk.so -> /usr/lib/libora.so



After that backup was running fine. Even sbttest libora.so succeeded.

returning error ORA-16191

Hi,

Issue - Standby not in sync

Version - oracle 11g

Error in alert log file -

PING[ARC1]: Heartbeat failed to connect to standby ‘STANDBY’. Error is 16191.
Sat Aug 06 02:06:52 2011
Error 1017 received logging on to the standby
------------------------------------------------------------
Check that the primary and standby are using a password file
and remote_login_passwordfile is set to SHARED or EXCLUSIVE,
and that the SYS password is same in the password files.
returning error ORA-16191

If you face the above error in your 11g standby setup, then do not waste your time with investigating unnecessary things. First check listener status at both sites and then check remote_login_passwordfile.
It should be shared or exclusive.

Most probably the above said things would be fine.

So just copy your primary password file to standby passwordfile location.($ORACLE_HOME/dbs).

Now check your RFS & MRP process it would be running.

If not just restart the MRPO process. :)

To Stop MRP

SQL>alter database recover managed standby database cancel

To Start MRP (If your environment is real time apply then use second command to restart MRP)

SQL>alter database recover managed standby database disconnect from session;

SQL>alter database recover managed standby database using current logfile disconnect from session;

Thank u..

RAC Nodes Time Difference issue bcoz of NTP in 11gR2

In One of our client Time difference between the nodes were keep on increasing drastically.

I fed up by setting that manually between the nodes.

Finally Got a permanent Solution.

We have configured time synchronization using NTP network protocol. But late we have decided oracle to manage the time sync.

For this we have to stop NTP & enable CTSS thats it.. :)

CTSS - Cluster time synchronization Service.

 Time Synchronization – Steps

(Needs to be tested on a test Setup  -  Not tested)

1.       Stop the database

Oracle]$srvctl stop database –d erpdb –o immediate

2.Stop all cluster related services on both the nodes (stopping HAS would stop all the services)

Grid]$crsctl stop has

3.Stop ntp process and remove all the configuration files related to ntpd

# /sbin/service ntpd stop

#/sbin/service ntpd status

#chkconfig ntpd off

#mv  /etc/ntp.conf   /etc/ntp.conf_bkp

Remove the ntpd.pid file.

#rm  /var/run/ntpd.pid

Or  #mv /var/run/ntpd.pid /var/run/ntpd.pid_bkp

4.Now set the date by using date command in Linux by keeping 3 seconds difference between both Nodes

For Example:

Node1]date –s   12:00:00

Node]date –s  12:00:03

       5.As of Now ctss is in observer Mode.

       Reboot Both Nodes, Start high availability Service,

      6. Now check ctss status

      Grid]$crsctl check ctss

      It should be in Active Mode

      7.Start the database and related Services.

Creating a Manual Standby from RMAN Backup and switching it to primary

Please find the steps that I have followed,

1.Move Rman backup to Standby Server,

sql>startup nomount;

>rman target /

rman > restore standby controlfile from '/backup/rman/hhhhh';

rman> sql "alter database mount standby database"

rman > catalog backuppiece '/backup/rman/hhhh';

rman > catalog backuppiece '/backup/rman/hhhh';

run

{

set newname for datafile 4 to '/oradata1/db/data/users01.dbf';

set newname for datafile 3 to '/oradata1/db/data/undotbs01.dbf'; 

set newname for datafile 2 to '/oradata1/db/data/sysaux01.dbf';        

set newname for datafile 1 to '/oradata1/db/data/system01.dbf';

set newname for datafile 5 to '/oradata1/db/data/undotbs02.dbf';

set newname for datafile 6 to '/oradata1/db/data/indx_1.dbf';    

set newname for datafile 7 to '/oradata1/db/data/16k.dbf';

set newname for datafile 8 to '/oradata1/db/data/bs.dbf';

set newname for datafile 9 to '/oradata1/db/data/er.dbf';        

set newname for datafile 10 to '/oradata1 db/data/prod.dbf'; 

set newname for datafile 11 to '/oradata2/db/data/prod_2.dbf';

set newname for datafile 12 to '/oradata2/db/data/prod_3.dbf';

set newname for datafile 13 to '/oradata2/db/data/prod_4.dbf';

set newname for datafile 14 to '/oradata2/db/data/rman.dbf';

set newname for datafile 15 to '/oradata2/db/data/om.dbf';

set newname for datafile 16 to '/oradata2/db/data/om_1.dbf';

set newname for datafile 17 to '/oradata2/db/data/ys.dbf';

set newname for datafile 18 to '/oradata2/db/data/example.dbf';

set newname for datafile 19 to '/oradata2/db/data/prod_5.dbf';

set newname for datafile 20 to '/oradata2/db/data/01.dbf';

set newname for datafile 21 to '/oradata2/db/data/system02.dbf';

set until scn 2281756927;

Restore database;

Switch datafile all;

Recover database;

}

sql> Recover standby database;

2.To open the database,

sql> Alter database recover managed standby database finish force;

sql>Alter database commit to switchover to primary;

sql>alter database open;

(While opening the database, it will try to create redologs on the specified path in controlfile.

If that path physically does not exist, it will fail.. We can create new redolgs here by specifying rename command -SQL)

>lsnrctl start

Restoration of database from RMAN Backup

1.Install Oracle 10g(10.2.0.5) on RHEL 5.6(Required For My setup, You can install as per your Requirement

 2. Create user Oracle and set environmental variables

  export ORACLE_SID=TEST

               3.Create pfile with necessary parameters like  db_name  ,control_files,

                  Background_dump_dest

  4.Start the database in nomount Stage with the pfile.

  >Startup nomount

  5.Start RMAN and connect to the target instance

   $rman TARGET / NOCATALOG

   6.Restore the controlfile from backup piece.

                RMAN>Restore controlfile from ‘Path with backup piece Name’

                7.Catalog all the backup pieces

               RMAN>catalog backuppiece ‘Path with backup piece Name’

              Take controlfile trace and check whether all physical location for datafiles available,

              If not add set newname & switch datafile all commands to the below rman script by            

              Mentioning old path & New Path

              8.Restore and Recover.

              RMAN> run {

              2> allocate channel c1 type disk;

              3> restore database;

              4> recover database;

              5> release channel c1;

              6> }

               9. Open the database in Resetlogs option

              >Alter database open resetlogs;

Diskgroup addition 11gR2 Rac on Windows

Adding  Disk group on 2 - Node 11gr2 RAC on Windows

-->Control Panel\All Control Panel Items\Administrative Tools -->Computer Management --> Disk Management

Create a partition with required size from the Free Space.

In My setup Disk 1 is the shared storage between the Nodes. From the available 7.5 GB am going add one partition with 2GB of Size.

Dont assign drive letter.

.

Do not Format the Volume.

Refresh the Disk Management in other Node, So that the same will appear there too. In case if it has got some default drive letter in other Node, remove the drive letter.

Once the partitions are completed, I used the ASMTOOLG.EXE utility to stamp each partition with an ASM label so that Oracle can recognize these partitions as candidate disks for the ASM instance. I executed the ASMTOOLG.EXE program.

This step is mandatory, otherwise asmca utility will not identify your asmdisk.

When we add label, we could find the candidate device with 2GB of size as we created. Select the same disk and then We can add the prefix as our wish.

Once it is done Use asmca to create diskgroup using those disks.  --> Create Diskgroup

It will display the disk, which we have created. Since we have only one disk, I have used external redundancy.

Check the same in other node. Check whether the diskgroup mounted or not.

Its Done.

Issues Faced,

While trying to stamp asm header, found below error,

Error from Node 2. When I check the status of the diskgroup, it was unmounted. logged in to +ASM2 instance and then mounted those diskgroups,

>alter diskgroup dd2 mount;

Everything worked fine.

Online Redo corruption

Hi,

One of my development db was affected because of Online Redo corruption.

Since it is non - critical db I have applied this solution and got my db back.

If you face Online Redo corruption, Please raise SR with oracle before trying this soln.


DB would go upto Mount Stage.

You can fing LGWR error in alertlog file.

Soln.

Create a pfile

>create pfile from spfile;

In Pfile add the undocumented oracle parameter

_allow_resetlogs_corruption=true

then start the db using the edited Pfile.

Now It needs an fake recovery..

So

>startup Mount;

>recover database;

>alter database open resetlogs;

then give checpoint

>alter system checkpoint

>alter system switch logfile

recreate the corruputed redolog file by dropping the existing one.

Remove that undocumented parameter from pfile.

 

Then Found the database is not consistent.. and instance was terminated automatically..

 

In alertlog

 

Found ORA-600

 

The issue was because of some uncommited transactions (corruption) in the undo tablespace

 

So, created New undo tablespace and then assigned undo_tablespace into new undo tablespace, restarted the instance.

 

Finally Issue got cleared..

 

Experts comments welcome...