1、备库standby异常报错
昨天凌晨磁盘空间突然暴涨导致oracle备库异常，报警，后了过来清理掉磁盘的备份文件，去了之后，归档日志能同步过来了，但是启动备库standby发现mrp没有启动，后台报错如下，google了，说这种情况要重新再做备库（不知道是否还有其它修复办法呢？）：错误日志报错信息如下：
Completed: alter database recover managed standby database using current logfile disconnect from session
Wed May 31 10:44:48 2017
Dumping diagnostic data in directory=[cdmp_20170531104448],requested by (instance=1,osid=43411 (PR00)),summary=[incident=600239].
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors with log /oracle/app/oracle/archivelogs/1_18799_940127349.dbf
MRP0: Background Media Recovery terminated with error 600
Errors in file /oracle/app/oracle/diag/rdbms/powerdes_s1/powerdes/trace/powerdes_pr00_43411.trc:
ORA-00600: internal error code,arguments: [2619],[18799],[],[]
Managed Standby Recovery not using Real Time Apply
Recovery interrupted!
Errors in file /oracle/app/oracle/diag/rdbms/powerdes_s1/powerdes/trace/powerdes_pr00_43411.trc:
ORA-00600: internal error code,[]
Errors in file /oracle/app/oracle/diag/rdbms/powerdes_s1/powerdes/trace/powerdes_mrp0_43408.trc (incident=600231):
ORA-00600: internal error code,[]
ORA-10877: error signaled in parallel recovery slave
ORA-10877: error signaled in parallel recovery slave
ORA-10877: error signaled in parallel recovery slave
ORA-10877: error signaled in parallel recovery slave
ORA-10877: error signaled in parallel recovery slave
2、使用duplicate target 进行恢复备库
duplicate target database for standby nofilenamecheck dorecover;
Starting Duplicate Db at 05-JUN-17
release channel c1;
release channel c2;
using target database control file instead of recovery catalog
allocated channel: ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: SID=4708 device type=DISK
contents of Memory Script:
{
set until scn 14430447592;
restore clone standby controlfile;
}
executing Memory Script
executing command: SET until clause
Starting restore at 05-JUN-17
using channel ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: starting datafile backup set restore
channel ORA_AUX_DISK_1: restoring control file
channel ORA_AUX_DISK_1: reading from backup piece /oracle/backup/data/controlfiles/c-3391761643-20170605-00
channel ORA_AUX_DISK_1: ORA-19870: error while restoring backup piece /oracle/backup/data/controlfiles/c-3391761643-20170605-00
ORA-19505: Failed to identify file "/oracle/backup/data/controlfiles/c-3391761643-20170605-00"
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
这里恢复失败，google了下，没有找到合适的办法，咋办呢？需要换种思路去恢复了，duplicate此路不通了。
3、尝试使用最原始的rman备份直接restore、recover的方式
copy控制文件、备份文件，备份文件已经有了，所以不需要再次生成，但是控制文件需要再次生成一下，于是在主库上生成最新的备份的控制文件：
RMAN> backup current controlfile for standby format '/home/oracle/ctlfile4.bak';
Starting backup at 05-JUN-17
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=948 device type=DISK
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
including standby control file in backup set
channel ORA_DISK_1: starting piece 1 at 05-JUN-17
channel ORA_DISK_1: finished piece 1 at 05-JUN-17
piece handle=/home/oracle/ctlfile4.bak tag=TAG20170605T223329 comment=NONE
channel ORA_DISK_1: backup set complete,elapsed time: 00:00:01
Finished backup at 05-JUN-17
Starting Control File and SPFILE Autobackup at 05-JUN-17
piece handle=/oracle/backup/data/controlfiles/c-3391761643-20170605-02 comment=NONE
Finished Control File and SPFILE Autobackup at 05-JUN-17
RMAN>
scp -r 2017-06-05 101.251.31.131:/oracle/
scp /home/oracle/ctlfile4.bak 101.251.31.131:/home/oracle/
接下来基本全在备库上进行操作，进行restore standby controlfile恢复控制文件
RMAN> restore standby controlfile from '/home/oracle/ctlfile4.bak';
Starting restore at 05-JUN-17
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=4708 device type=DISK
channel ORA_DISK_1: restoring control file
channel ORA_DISK_1: restore complete,elapsed time: 00:00:01
output file name=/oracle/app/oracle/oradata/powerdes/control01.ctl
output file name=/oracle/app/oracle/fast_recovery_area/powerdes/control02.ctl
Finished restore at 05-JUN-17
RMAN>
启动到mount状态
RMAN> alter database mount;
database mounted
released channel: ORA_DISK_1
RMAN
注册备份集合
RMAN> catalog start with '/oracle/2017-06-05';
Starting implicit crosscheck backup at 05-JUN-17
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=4709 device type=DISK
Crosschecked 147 objects
Finished implicit crosscheck backup at 05-JUN-17
Starting implicit crosscheck copy at 05-JUN-17
using channel ORA_DISK_1
Crosschecked 2 objects
Finished implicit crosscheck copy at 05-JUN-17
searching for all files in the recovery area
cataloging files...
no files cataloged
searching for all files that match the pattern /oracle/2017-06-05
List of Files Unknown to the Database
=====================================
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9901.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9899.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9910.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9912.bak
File Name: /oracle/2017-06-05/rman_backup.log
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9908.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9904.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9905.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9906.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9909.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9903.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9902.bak
File Name: /oracle/2017-06-05/full_POWERDES_20170605_9911.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9900.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9907.bak
Do you really want to catalog the above files (enter YES or NO)? YES
cataloging files...
cataloging done
List of Cataloged Files
=======================
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9901.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9899.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9910.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9912.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9908.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9904.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9905.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9906.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9909.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9903.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9902.bak
File Name: /oracle/2017-06-05/full_POWERDES_20170605_9911.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9900.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9907.bak
List of Files Which Where Not Cataloged
=======================================
File Name: /oracle/2017-06-05/rman_backup.log
RMAN-07517: Reason: The file header is corrupted
RMAN>
博客作者黄杉（mchdba），博客源地址：http://www.jb51.cc/article/p-kdgzwcet-bav.html，谢绝转载。
开始restore database操作
RMAN> restore database;
Starting restore at 05-JUN-17
using channel ORA_DISK_1
channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00001 to /home/oradata/powerdes/system01.dbf
channel ORA_DISK_1: restoring datafile 00002 to /home/oradata/powerdes/sysaux01.dbf
channel ORA_DISK_1: restoring datafile 00003 to /home/oradata/powerdes/undotbs01.dbf
channel ORA_DISK_1: restoring datafile 00004 to /home/oradata/powerdes/users01.dbf
channel ORA_DISK_1: restoring datafile 00005 to /home/oradata/powerdes/powerdesk01.dbf
channel ORA_DISK_1: restoring datafile 00006 to /home/oradata/powerdes/plas01.dbf
channel ORA_DISK_1: restoring datafile 00007 to /home/oradata/powerdes/pl01.dbf
channel ORA_DISK_1: restoring datafile 00008 to /home/oradata/powerdes/help01.dbf
channel ORA_DISK_1: restoring datafile 00009 to /home/oradata/powerdes/adobelc01.dbf
channel ORA_DISK_1: restoring datafile 00010 to /home/oradata/powerdes/sms01.dbf
channel ORA_DISK_1: restoring datafile 00011 to /home/oradata/powerdes/plcrm01.dbf
channel ORA_DISK_1: restoring datafile 00012 to /home/oradata/powerdes/powerdesk02.dbf
channel ORA_DISK_1: restoring datafile 00013 to /home/oradata/powerdes/datagm01.dbf
channel ORA_DISK_1: restoring datafile 00014 to /home/oradata/powerdes/plimp01.DBF
channel ORA_DISK_1: restoring datafile 00015 to /home/oradata/powerdes/dwetl01.DBF
channel ORA_DISK_1: restoring datafile 00016 to /home/oradata/powerdes/dw02.DBF
channel ORA_DISK_1: restoring datafile 00017 to /home/oradata/powerdes/timdba01.DBF
channel ORA_DISK_1: restoring datafile 00018 to /home/oradata/powerdes/users02.dbf
channel ORA_DISK_1: restoring datafile 00019 to /home/oradata/powerdes/system02.dbf
channel ORA_DISK_1: restoring datafile 00020 to /home/oradata/powerdes/powerdesk03.dbf
channel ORA_DISK_1: restoring datafile 00021 to /home/oradata/powerdes/timdba02.dbf
channel ORA_DISK_1: reading from backup piece /oracle/2017-06-05/full_POWERDES_20170605_9911.bak
channel ORA_DISK_1: piece handle=/oracle/2017-06-05/full_POWERDES_20170605_9911.bak tag=TAG20170605T033303
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete,elapsed time: 01:29:37
Finished restore at 05-JUN-17
RMAN>
开始rocover database操作
RMAN> recover database;
Starting recover at 05-JUN-17
using channel ORA_DISK_1
starting media recovery
archived log for thread 1 with sequence 20421 is already on disk as file /oracle/app/oracle/archivelogs/1_20421_940127349.dbf
archived log for thread 1 with sequence 20422 is already on disk as file /oracle/app/oracle/archivelogs/1_20422_940127349.dbf
archived log for thread 1 with sequence 20423 is already on disk as file /oracle/app/oracle/archivelogs/1_20423_940127349.dbf
archived log for thread 1 with sequence 20424 is already on disk as file /oracle/app/oracle/archivelogs/1_20424_940127349.dbf
archived log for thread 1 with sequence 20425 is already on disk as file /oracle/app/oracle/archivelogs/1_20425_940127349.dbf
archived log for thread 1 with sequence 20426 is already on disk as file /oracle/app/oracle/archivelogs/1_20426_940127349.dbf
archived log for thread 1 with sequence 20427 is already on disk as file /oracle/app/oracle/archivelogs/1_20427_940127349.dbf
archived log for thread 1 with sequence 20428 is already on disk as file /oracle/app/oracle/archivelogs/1_20428_940127349.dbf
archived log for thread 1 with sequence 20429 is already on disk as file /oracle/app/oracle/archivelogs/1_20429_940127349.dbf
archived log for thread 1 with sequence 20430 is already on disk as file /oracle/app/oracle/archivelogs/1_20430_940127349.dbf
archived log for thread 1 with sequence 20431 is already on disk as file /oracle/app/oracle/archivelogs/1_20431_940127349.dbf
archived log for thread 1 with sequence 20432 is already on disk as file /oracle/app/oracle/archivelogs/1_20432_940127349.dbf
archived log for thread 1 with sequence 20433 is already on disk as file /oracle/app/oracle/archivelogs/1_20433_940127349.dbf
archived log for thread 1 with sequence 20434 is already on disk as file /oracle/app/oracle/archivelogs/1_20434_940127349.dbf
channel ORA_DISK_1: starting archived log restore to default destination
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=20357
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=20358
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=20359
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=20360
channel ORA_DISK_1: reading from backup piece /oracle/2017-06-05/arch_POWERDES_20170605_9912.bak
channel ORA_DISK_1: piece handle=/oracle/2017-06-05/arch_POWERDES_20170605_9912.bak tag=TAG20170605T051030
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete,elapsed time: 00:00:15
archived log file name=/oracle/app/oracle/archivelogs/1_20357_940127349.dbf thread=1 sequence=20357
archived log file name=/oracle/app/oracle/archivelogs/1_20358_940127349.dbf thread=1 sequence=20358
archived log file name=/oracle/app/oracle/archivelogs/1_20359_940127349.dbf thread=1 sequence=20359
archived log file name=/oracle/app/oracle/archivelogs/1_20360_940127349.dbf thread=1 sequence=20360
unable to find archived log
archived log thread=1 sequence=20361
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of recover command at 06/05/2017 22:17:07
RMAN-06054: media recovery requesting unknown archived log for thread 1 with sequence 20361 and starting SCN of 14425991740
RMAN> recover database until scn 14425991740;
Starting recover at 05-JUN-17
using channel ORA_DISK_1
starting media recovery
Oracle Error:
ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01152: file 1 was not restored from a sufficiently old backup
ORA-01110: data file 1: '/home/oradata/powerdes/system01.dbf'
media recovery complete,elapsed time: 00:00:01
Finished recover at 05-JUN-17
RMAN>
最后开始启动传输并且应用归档日志
sql> alter database recover managed standby database using current logfile disconnect from session;
Database altered.
sql>
4、开始检查主库备库数据一致性
查看归档日志应用情况：
sql> select sequence#,applied from v$archived_log order by sequence# asc;
SEQUENCE# APPLIED
---------- ---------
20379 YES
20380 YES
20381 YES
20382 YES
20383 YES
20384 YES
20385 YES
20386 YES
20387 YES
20388 YES
20389 YES
SEQUENCE# APPLIED
---------- ---------
20390 YES
20391 YES
20392 YES
20393 YES
20394 YES
20395 YES
20396 YES
20397 YES
20398 NO
20399 NO
20400 NO
查看后台alert日志，正在进行应用
Media Recovery Log /oracle/app/oracle/archivelogs/1_20391_940127349.dbf
Mon Jun 05 22:24:00 2017
Media Recovery Log /oracle/app/oracle/archivelogs/1_20392_940127349.dbf
Mon Jun 05 22:24:10 2017
Media Recovery Log /oracle/app/oracle/archivelogs/1_20393_940127349.dbf
Mon Jun 05 22:24:22 2017
Media Recovery Log /oracle/app/oracle/archivelogs/1_20394_940127349.dbf
Mon Jun 05 22:24:33 2017
Media Recovery Log /oracle/app/oracle/archivelogs/1_20395_940127349.dbf
Mon Jun 05 22:24:47 2017
Media Recovery Log /oracle/app/oracle/archivelogs/1_20396_940127349.dbf
Mon Jun 05 22:25:00 2017
Media Recovery Log /oracle/app/oracle/archivelogs/1_20397_940127349.dbf
查看备库主库log信息是保持一致的，如下所示：
sql> archive log list;
Database log mode Archive Mode
Automatic archival Enabled
Archive destination /oracle/app/oracle/archivelogs
Oldest online log sequence 20430
Next log sequence to archive 20435
Current log sequence 20435
sql>
sql> archive log list;
Database log mode Archive Mode
Automatic archival Enabled
Archive destination /oracle/app/oracle/archivelogs
Oldest online log sequence 20430
Next log sequence to archive 0
Current log sequence 20435
sql>
至此，利用rman备份直接restore、recover的方式恢复了dataguard的standby库。
仔细思考下，这个问题以前一直没有遇到过，磁盘满了后，按照道理来说，清理磁盘后，可以直接恢复得，但是却不能恢复，原因猜测在于启用了实时应用归档日志导致。这样就导致了重新再次传输加载的时候，混乱了。去查看以前的dataguard搭建记录http://www.jb51.cc/article/p-ozjbvbst-bav.html，发现有类似的操作记录：
（2）添加current logfile参数，使得应用当前正在读写，还没有完成归档的日志 sql> alter database recover managedstandby database using current logfile disconnect from session;
