合 OGG-00446 Could not find archived log for sequence

2022年11月10日

0 306 1

Tags：故障处理 OGG OGG-00446

👉 本文共约12762个字，系统预计阅读时间或需48分钟。

前言部分
导读和注意事项
本文简介
相关知识点扫盲
OGG的告警日志路径
故障分析及解决过程
故障环境介绍
故障发生现象及报错信息
故障分析及解决过程
确认归档文件是否存在
第一种办法：改变抽取进程的时间
第二种办法：数据初始化
source端准备
target 准备
验证是否可以正常同步数据
故障处理总结
OGG-00446 Could not find archived log for sequence 53586 thread 1 under alternative destinations.

前言部分

导读和注意事项

各位技术爱好者，看完本文后，你可以掌握如下的技能，也可以学到一些其它你所不知道的知识，~O(∩_∩)O~：

① OGG-00446 Could not find archived log for sequence 的解决方法（重点）

② OGG基本的维护、排错操作

本文简介

同事说有一套OGG的环境挂掉了，不能同步了，让处理一下，查看了进程状态和错误日志后分析得出是source端的归档日志不见了，也没有备份，最后只能重新初始化数据来解决了。

故障分析及解决过程

故障环境介绍

项目	source db	target db
db 类型	rac环境	rac环境
db version	10.2.0.5.0	11.2.0.3.0
db 存储	RAW	ASM
ORACLE_SID	oraXPAD	oraXPAD
db_name	oraXPAD	oraXPAD
主机IP地址：	22.188.131.27 22.188.131.47	22.188.132.82 22.188.132.85
OS版本及kernel版本	AIX 5.3	AIX 6.1
OS hostname	ZTGXPADDB1	ZT1XPADRDB1

故障发生现象及报错信息

source 端：

 root@ZTGXPADDB1:/# ps -ef|grep gg
    root 3248278 3211312   1 09:15:45  pts/0  0:00 grep gg
  oracle 3719330 3751980   1   Dec 30      -  7:29 /gg/ogg/extract PARAMFILE /gg/ogg/dirprm/ggspump.prm REPORTFILE /gg/ogg/dirrpt/GGSPUMP.rpt PROCESSID GGSPUMP USESUBDIRS
  oracle 3723472 3706978   0   Aug 03      -  0:03 /oracle/app/oracle/product/10.2.0/crs/bin/evmlogger.bin -o /oracle/app/oracle/product/10.2.0/crs/evm/log/evmlogger.info -l /oracle/app/oracle/product/10.2.0/crs/evm/log/evmlogger.log
  oracle 3751980       1   0   Aug 03      - 21:00 ./mgr PARAMFILE /gg/ogg/dirprm/mgr.prm REPORTFILE /gg/ogg/dirrpt/MGR.rpt PROCESSID MGR PORT 7809
  oracle 3772504 3751980   3   Dec 30      - 18:42 /gg/ogg/extract PARAMFILE /gg/ogg/dirprm/ggsext.prm REPORTFILE /gg/ogg/dirrpt/GGSEXT.rpt PROCESSID GGSEXT USESUBDIRS
root@ZTGXPADDB1:/# cd /gg/ogg/
oracle@ZTHXPADDB2:/gg/ogg$ ggsci

Oracle GoldenGate Command Interpreter for Oracle
Version 11.1.1.1.2 OGGCORE_11.1.1.1.2_PLATFORMS_111004.2100 
AIX 5L, ppc, 64bit (optimized), Oracle 10.2 on Oct  5 2011 02:32:51

Copyright (C) 1995, 2011, Oracle and/or its affiliates. All rights reserved.

GGSCI (ZTHXPADDB2) 1> info all

Program     Status      Group       Lag           Time Since Chkpt

MANAGER     RUNNING                                           
EXTRACT     STOPPED     GGSEXT      00:00:00      21915:44:49 
EXTRACT     RUNNING     GGSPUMP     00:00:00      unknown 
GGSCI (ZTHXPADDB2) 5> view report GGSEXT 

***********************************************************************
                 Oracle GoldenGate Capture for Oracle
     Version 11.1.1.1.2 OGGCORE_11.1.1.1.2_PLATFORMS_111004.2100 
  AIX 5L, ppc, 64bit (optimized), Oracle 10.2 on Oct  5 2011 02:52:14

Copyright (C) 1995, 2011, Oracle and/or its affiliates. All rights reserved.

                    Starting at 2020-08-03 07:47:21
***********************************************************************

Operating System Version:
AIX
Version 5, Release 3
Node: ZTGXPADDB1
Machine: 00C207D44C00
                         soft limit   hard limit
Address Space Size   :    unlimited    unlimited
Heap Size            :    unlimited    unlimited
File Size            :    unlimited    unlimited
CPU Time             :    unlimited    unlimited

Process id: 2113596

Description: 

***********************************************************************
**            Running with the following parameters                  **
***********************************************************************
EXTRACT ggsext
SETENV (NLS_LANG="AMERICAN_AMERICA.ZHS16GBK")
Set environment variable (NLS_LANG=AMERICAN_AMERICA.ZHS16GBK)
SETENV (ORACLE_SID="oraXPAD1")
Set environment variable (ORACLE_SID=oraXPAD1)
USERID goldengate, PASSWORD **********
DISCARDFILE ./dirrpt/ggsext.dsc, APPEND, MEGABYTES 100
--DDL INCLUDE MAPPED
--DDLOPTIONS ADDTRANDATA NOCROSSRENAME REPORT
TRANLOGOPTIONS RAWDEVICEOFFSET 0
EXTTRAIL ./dirdat/st
TRANLOGOPTIONS ALTARCHIVELOGDEST PRIMARY INSTANCE oraXPAD1 /gg/sarch1, ALTARCHIVELOGDEST INSTANCE oraXPAD1 /gg/sarch2, ALTARCHIVELOGDEST INSTANCE oraXPAD2 /gg/sarch2, ALTARCHIVELOGDEST INSTANCE oraXPAD2 /gg/sarch1
--TRANLOGOPTIONS ALTARCHIVELOGDEST PRIMARY INSTANCE oraXPAD1 /gg/sarch1, ALTARCHIVELOGDEST INSTANCE oraXPAD2 /gg/sarch2
WILDCARDRESOLVE DYNAMIC
dynamicresolution
THREADOPTIONS MAXCOMMITPROPAGATIONDELAY 60000
--GETTRUNCATES
TABLE XPADB.BASE_ACTIONPOWER;
TABLE XPADB.BASE_BANK;
TABLE XPADB.BASE_BANKMERGE;
TABLE XPADB.BASE_BANKTREE;
TABLE XPADB.BASE_BRCHBANKCTRL;
TABLE XPADB.BASE_CERTIFICATE;
《《《《。。。。。。。。篇幅原因，有省略。。。。。。。。》》》》
TABLE XPADB.RICH_TRANSFERTRADE;
TABLE XPADB.RICH_LISTING;
TABLE XPADB.RICH_DELISTING;
TABLE XPADB.RICH_CUSTPROOFPROPERTY;
TABLE XPADB.BASE_FEESCALE;
TABLE XPADB.BASE_FEEDISCOUNT;

TABLE t.t2;

Bounded Recovery Parameter:
BRINTERVAL = 4HOURS
BRDIR      = /gg/ogg

CACHEMGR virtual memory values (may have been adjusted)
CACHEBUFFERSIZE:                         64K
CACHESIZE:                                8G
CACHEBUFFERSIZE (soft max):               4M
CACHEPAGEOUTSIZE (normal):                4M
PROCESS VM AVAIL FROM OS (min):          16G
CACHESIZEMAX (strict force to disk):  13.99G

2020-08-03 07:47:22  INFO    OGG-01639  BOUNDED RECOVERY:  ACTIVE: for object pool 1: p1916948_Redo Thread 1.

2020-08-03 07:47:22  INFO    OGG-01640  BOUNDED RECOVERY: recovery start XID: 0.0.0.

2020-08-03 07:47:22  INFO    OGG-01641  BOUNDED RECOVERY: recovery start position: SeqNo: 2598, RBA: 89822224, SCN: 2850.524344649 (12241181138249), Timestamp: 2018-07-01 04:09:42.000000.

2020-08-03 07:47:22  INFO    OGG-01642  BOUNDED RECOVERY: recovery end position: SeqNo: 2598, RBA: 89823232, SCN: 2850.524344649 (12241181138249), Timestamp: 2018-07-01 04:09:42.000000.

2020-08-03 07:47:22  INFO    OGG-01639  BOUNDED RECOVERY:  ACTIVE: for object pool 2: p1916948_Redo Thread 2.

2020-08-03 07:47:22  INFO    OGG-01640  BOUNDED RECOVERY: recovery start XID: 0.0.0.

2020-08-03 07:47:22  INFO    OGG-01641  BOUNDED RECOVERY: recovery start position: SeqNo: 2382, RBA: 245488144, SCN: 2850.524344719 (12241181138319), Timestamp: 2018-07-01 04:09:41.000000.

2020-08-03 07:47:22  INFO    OGG-01642  BOUNDED RECOVERY: recovery end position: SeqNo: 2382, RBA: 245489152, SCN: 2850.524344719 (12241181138319), Timestamp: 2018-07-01 04:09:41.000000.

2020-08-03 07:47:22  INFO    OGG-01643  BOUNDED RECOVERY: CANCELED: for object pool 2: p1916948_Redo Thread 2.

2020-08-03 07:47:22  INFO    OGG-01579  BOUNDED RECOVERY: VALID BCP: CP.GGSEXT.000000102.

2020-08-03 07:47:22  INFO    OGG-01629  BOUNDED RECOVERY: PERSISTED OBJECTS RECOVERED: 1.

Database Version:
Oracle Database 10g Enterprise Edition Release 10.2.0.5.0 - 64bi
PL/SQL Release 10.2.0.5.0 - Production
CORE    10.2.0.5.0      Production
TNS for IBM/AIX RISC System/6000: Version 10.2.0.5.0 - Productio
NLSRTL Version 10.2.0.5.0 - Production

Database Language and Character Set:
NLS_LANG = "AMERICAN_AMERICA.ZHS16GBK" 
NLS_LANGUAGE     = "AMERICAN" 
NLS_TERRITORY    = "AMERICA" 
NLS_CHARACTERSET = "ZHS16GBK" 

2020-08-03 07:47:24  INFO    OGG-00546  Default thread stack size: 196608.

2020-08-03 07:47:24  INFO    OGG-00547  Increasing thread stack size from 196608 to 1048576.

2020-08-03 07:47:24  INFO    OGG-01513  Positioning to (Thread 1) Sequence 2598, RBA 89822224, SCN 2850.524344649.

Source Context :
  SourceModule            : [er.main]
  SourceID                : [/scratch/aime2/adestore/views/aime2_staxk11/oggcore/OpenSys/src/app/er/rep.c]
  SourceFunction          : [extract_start_point(time_elt_def *, time_elt_def *)]
  SourceLine              : [13026]

2020-08-03 07:47:44  ERROR   OGG-00446  Could not find archived log for sequence 2598 thread 1 under alternative destinations. SQL <SELECT MAX(sequence#)  FROM v$log WHERE thread# = :ora_thread>. Last alternative log tried /gg/sarch2/1_2598_704996932.arc, error retrieving redo file name for sequence 2598, archived = 1, use_alternate = 0Not able to establish initial position for sequence 2598, rba 89822224.

2020-08-03 07:47:44  ERROR   OGG-01668  PROCESS ABENDING.

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

root@ZTGXPADDB1:/# ps -ef|grep gg

root 3248278 3211312 1 09:15:45 pts/0 0:00 grep gg

oracle 3719330 3751980 1 Dec 30 - 7:29 /gg/ogg/extract PARAMFILE /gg/ogg/dirprm/ggspump.prm REPORTFILE /gg/ogg/dirrpt/GGSPUMP.rpt PROCESSID GGSPUMP USESUBDIRS

oracle 3723472 3706978 0 Aug 03 - 0:03 /oracle/app/oracle/product/10.2.0/crs/bin/evmlogger.bin -o /oracle/app/oracle/product/10.2.0/crs/evm/log/evmlogger.info -l /oracle/app/oracle/product/10.2.0/crs/evm/log/evmlogger.log

oracle 3751980 1 0 Aug 03 - 21:00 ./mgr PARAMFILE /gg/ogg/dirprm/mgr.prm REPORTFILE /gg/ogg/dirrpt/MGR.rpt PROCESSID MGR PORT 7809

oracle 3772504 3751980 3 Dec 30 - 18:42 /gg/ogg/extract PARAMFILE /gg/ogg/dirprm/ggsext.prm REPORTFILE /gg/ogg/dirrpt/GGSEXT.rpt PROCESSID GGSEXT USESUBDIRS

root@ZTGXPADDB1:/# cd /gg/ogg/

oracle@ZTHXPADDB2:/gg/ogg$ ggsci

Oracle GoldenGate Command Interpreter for Oracle

Version 11.1.1.1.2 OGGCORE_11.1.1.1.2_PLATFORMS_111004.2100

AIX 5L, ppc, 64bit (optimized), Oracle 10.2 on Oct 5 2011 02:32:51

GGSCI (ZTHXPADDB2) 1> info all

Program Status Group Lag Time Since Chkpt

MANAGER RUNNING

EXTRACT STOPPED GGSEXT 00:00:00 21915:44:49

EXTRACT RUNNING GGSPUMP 00:00:00 unknown

GGSCI (ZTHXPADDB2) 5> view report GGSEXT

***********************************************************************

Oracle GoldenGate Capture for Oracle

Version 11.1.1.1.2 OGGCORE_11.1.1.1.2_PLATFORMS_111004.2100

AIX 5L, ppc, 64bit (optimized), Oracle 10.2 on Oct 5 2011 02:52:14

Starting at 2020-08-03 07:47:21

***********************************************************************

Operating System Version:

AIX

Version 5, Release 3

Node: ZTGXPADDB1

Machine: 00C207D44C00

soft limit hard limit

Address Space Size : unlimited unlimited

Heap Size : unlimited unlimited

File Size : unlimited unlimited

CPU Time : unlimited unlimited

Process id: 2113596

Description:

***********************************************************************

** Running with the following parameters **

***********************************************************************

EXTRACT ggsext

SETENV (NLS_LANG="AMERICAN_AMERICA.ZHS16GBK")

Set environment variable (NLS_LANG=AMERICAN_AMERICA.ZHS16GBK)

SETENV (ORACLE_SID="oraXPAD1")

Set environment variable (ORACLE_SID=oraXPAD1)

USERID goldengate, PASSWORD **********

DISCARDFILE ./dirrpt/ggsext.dsc, APPEND, MEGABYTES 100

--DDL INCLUDE MAPPED

--DDLOPTIONS ADDTRANDATA NOCROSSRENAME REPORT

TRANLOGOPTIONS RAWDEVICEOFFSET 0

EXTTRAIL ./dirdat/st

TRANLOGOPTIONS ALTARCHIVELOGDEST PRIMARY INSTANCE oraXPAD1 /gg/sarch1, ALTARCHIVELOGDEST INSTANCE oraXPAD1 /gg/sarch2, ALTARCHIVELOGDEST INSTANCE oraXPAD2 /gg/sarch2, ALTARCHIVELOGDEST INSTANCE oraXPAD2 /gg/sarch1

--TRANLOGOPTIONS ALTARCHIVELOGDEST PRIMARY INSTANCE oraXPAD1 /gg/sarch1, ALTARCHIVELOGDEST INSTANCE oraXPAD2 /gg/sarch2

WILDCARDRESOLVE DYNAMIC

dynamicresolution

THREADOPTIONS MAXCOMMITPROPAGATIONDELAY 60000

--GETTRUNCATES

TABLE XPADB.BASE_ACTIONPOWER;

TABLE XPADB.BASE_BANK;

TABLE XPADB.BASE_BANKMERGE;

TABLE XPADB.BASE_BANKTREE;

TABLE XPADB.BASE_BRCHBANKCTRL;

TABLE XPADB.BASE_CERTIFICATE;

《《《《。。。。。。。。篇幅原因，有省略。。。。。。。。》》》》

TABLE XPADB.RICH_TRANSFERTRADE;

TABLE XPADB.RICH_LISTING;

TABLE XPADB.RICH_DELISTING;

TABLE XPADB.RICH_CUSTPROOFPROPERTY;

TABLE XPADB.BASE_FEESCALE;

TABLE XPADB.BASE_FEEDISCOUNT;

TABLE t.t2;

Bounded Recovery Parameter:

BRINTERVAL = 4HOURS

BRDIR = /gg/ogg

CACHEMGR virtual memory values (may have been adjusted)

CACHEBUFFERSIZE: 64K

CACHESIZE: 8G

CACHEBUFFERSIZE (soft max): 4M

CACHEPAGEOUTSIZE (normal): 4M

PROCESS VM AVAIL FROM OS (min): 16G

CACHESIZEMAX (strict force to disk): 13.99G

2020-08-03 07:47:22 INFO OGG-01639 BOUNDED RECOVERY: ACTIVE: for object pool 1: p1916948_Redo Thread 1.

2020-08-03 07:47:22 INFO OGG-01640 BOUNDED RECOVERY: recovery start XID: 0.0.0.

2020-08-03 07:47:22 INFO OGG-01641 BOUNDED RECOVERY: recovery start position: SeqNo: 2598, RBA: 89822224, SCN: 2850.524344649 (12241181138249), Timestamp: 2018-07-01 04:09:42.000000.

2020-08-03 07:47:22 INFO OGG-01642 BOUNDED RECOVERY: recovery end position: SeqNo: 2598, RBA: 89823232, SCN: 2850.524344649 (12241181138249), Timestamp: 2018-07-01 04:09:42.000000.

2020-08-03 07:47:22 INFO OGG-01639 BOUNDED RECOVERY: ACTIVE: for object pool 2: p1916948_Redo Thread 2.

2020-08-03 07:47:22 INFO OGG-01640 BOUNDED RECOVERY: recovery start XID: 0.0.0.

2020-08-03 07:47:22 INFO OGG-01641 BOUNDED RECOVERY: recovery start position: SeqNo: 2382, RBA: 245488144, SCN: 2850.524344719 (12241181138319), Timestamp: 2018-07-01 04:09:41.000000.

2020-08-03 07:47:22 INFO OGG-01642 BOUNDED RECOVERY: recovery end position: SeqNo: 2382, RBA: 245489152, SCN: 2850.524344719 (12241181138319), Timestamp: 2018-07-01 04:09:41.000000.

2020-08-03 07:47:22 INFO OGG-01643 BOUNDED RECOVERY: CANCELED: for object pool 2: p1916948_Redo Thread 2.

2020-08-03 07:47:22 INFO OGG-01579 BOUNDED RECOVERY: VALID BCP: CP.GGSEXT.000000102.

2020-08-03 07:47:22 INFO OGG-01629 BOUNDED RECOVERY: PERSISTED OBJECTS RECOVERED: 1.

Database Version:

Oracle Database 10g Enterprise Edition Release 10.2.0.5.0 - 64bi

PL/SQL Release 10.2.0.5.0 - Production

CORE 10.2.0.5.0 Production

TNS for IBM/AIX RISC System/6000: Version 10.2.0.5.0 - Productio

NLSRTL Version 10.2.0.5.0 - Production

Database Language and Character Set:

NLS_LANG = "AMERICAN_AMERICA.ZHS16GBK"

NLS_LANGUAGE = "AMERICAN"

NLS_TERRITORY = "AMERICA"

NLS_CHARACTERSET = "ZHS16GBK"

2020-08-03 07:47:24 INFO OGG-00546 Default thread stack size: 196608.

2020-08-03 07:47:24 INFO OGG-00547 Increasing thread stack size from 196608 to 1048576.

2020-08-03 07:47:24 INFO OGG-01513 Positioning to (Thread 1) Sequence 2598, RBA 89822224, SCN 2850.524344649.

Source Context :

SourceModule : [er.main]

SourceID : [/scratch/aime2/adestore/views/aime2_staxk11/oggcore/OpenSys/src/app/er/rep.c]

SourceFunction : [extract_start_point(time_elt_def *, time_elt_def *)]

SourceLine : [13026]

2020-08-03 07:47:44 ERROR OGG-00446 Could not find archived log for sequence 2598 thread 1 under alternative destinations. SQL <SELECT MAX(sequence#) FROM v$log WHERE thread# = :ora_thread>. Last alternative log tried /gg/sarch2/1_2598_704996932.arc, error retrieving redo file name for sequence 2598, archived = 1, use_alternate = 0Not able to establish initial position for sequence 2598, rba 89822224.

2020-08-03 07:47:44 ERROR OGG-01668 PROCESS ABENDING.

source端告警日志：

oracle@ZT1XPADRDB1:/gg/ogg$ tail -f ggserr.log

2020-08-03 07:47:27 INFO OGG-01226 Oracle GoldenGate Capture for Oracle, ggspump.prm: Socket buffer size set to 27985 (flush size 27985).

2020-08-03 07:47:27 INFO OGG-01055 Oracle GoldenGate Capture for Oracle, ggspump.prm: Recovery initialization completed for target file ./dirdat/tt002073, at RBA 1153.

2020-08-03 07:47:27 INFO OGG-01478 Oracle GoldenGate Capture for Oracle, ggspump.prm: Output file ./dirdat/tt is using format RELEASE 10.4/11.1.

2020-08-03 07:47:27 INFO OGG-01026 Oracle GoldenGate Capture for Oracle, ggspump.prm: Rolling over remote file ./dirdat/tt002074.

2020-08-03 07:47:27 INFO OGG-01053 Oracle GoldenGate Capture for Oracle, ggspump.prm: Recovery completed for target file ./dirdat/tt002074, at RBA 1119.

2020-08-03 07:47:27 INFO OGG-01057 Oracle GoldenGate Capture for Oracle, ggspump.prm: Recovery completed for all targets.

2020-08-03 07:47:44 ERROR OGG-00446 Oracle GoldenGate Capture for Oracle, ggsext.prm: Could not find archived log for sequence 2598 thread 1 under alternative destinations. SQL <SELECT MAX(sequence#) FROM v$log WHERE thread# = :ora_thread>. Last alternative log tried /gg/sarch2/1_2598_704996932.arc, error retrieving redo file name for sequence 2598, archived = 1, use_alternate = 0Not able to establish initial position for sequence 2598, rba 89822224.

2020-08-03 07:47:44 ERROR OGG-01668 Oracle GoldenGate Capture for Oracle, ggsext.prm: PROCESS ABENDING.

2020-12-29 15:37:48 ERROR OGG-01117 Oracle GoldenGate Command Interpreter for Oracle: Received signal: Program interrupt (2).

2020-12-29 15:37:48 ERROR OGG-01668 Oracle GoldenGate Command Interpreter for Oracle: PROCESS ABENDING.

 target 端：
 oracle@ZT1XPADRDB1:/gg/ogg$ ggsci

 Oracle GoldenGate Command Interpreter for Oracle
 Version 11.1.1.1.2 OGGCORE_11.1.1.1.2_PLATFORMS_111004.2100 
 AIX 5L, ppc, 64bit (optimized), Oracle 11g on Oct  5 2011 00:37:03

 Copyright (C) 1995, 2011, Oracle and/or its affiliates. All rights reserved.

 GGSCI (ZT1XPADRDB1) 1> info all

 Program     Status      Group       Lag           Time Since Chkpt

 MANAGER     RUNNING                                           
 REPLICAT    ABENDED     GGSREP      30088:08:59   3577:00:35  
 GGSCI (ZT1XPADRDB1) 4> info all

 Program     Status      Group       Lag           Time Since Chkpt

 MANAGER     RUNNING                                           
 REPLICAT    ABENDED     GGSREP      30088:08:59   3577:47:23  

 GGSCI (ZT1XPADRDB1) 5> start  GGSREP 

 Sending START request to MANAGER ...
 REPLICAT GGSREP starting

 GGSCI (ZT1XPADRDB1) 6>  info all

 Program     Status      Group       Lag           Time Since Chkpt

 MANAGER     RUNNING                                           
 REPLICAT    ABENDED     GGSREP      33665:56:32   00:03:08    
 GGSCI (ZT1XPADRDB1) 8> view report GGSREP 

 ***********************************************************************
                  Oracle GoldenGate Delivery for Oracle
      Version 11.1.1.1.2 OGGCORE_11.1.1.1.2_PLATFORMS_111004.2100 
   AIX 5L, ppc, 64bit (optimized), Oracle 11g on Oct  5 2011 02:03:36

 Copyright (C) 1995, 2011, Oracle and/or its affiliates. All rights reserved.

                     Starting at 2020-12-30 09:34:14
 ***********************************************************************

 Operating System Version:
 AIX
 Version 6, Release 1
 Node: ZT1XPADRDB1
 Machine: 00C08DF04C00
                          soft limit   hard limit
 Address Space Size   :    unlimited    unlimited
 Heap Size            :    unlimited    unlimited
 File Size            :    unlimited    unlimited
 CPU Time             :    unlimited    unlimited

 Process id: 35717358

 Description: 

 ***********************************************************************
 **            Running with the following parameters                  **
 ***********************************************************************
 REPLICAT ggsrep
 --SETENV (ORACLE_SID = ggstd)
 SETENV (NLS_LANG="AMERICAN_AMERICA.ZHS16GBK")
 Set environment variable (NLS_LANG=AMERICAN_AMERICA.ZHS16GBK)
 USERID goldengate, PASSWORD **********
 --DDL include MAPPED
 --DDLERROR DEFAULT IGNORE RETRYOP MAXRETRIES 3 RETRYDELAY 5
 --DDLOPTIONS report
 REPERROR DEFAULT, ABEND
 ASSUMETARGETDEFS
 DBOPTIONS DEFERREFCONST
 DBOPTIONS SUPPRESSTRIGGERS
 --BATCHSQL
 --GROUPTRANSOPS 2000
 DISCARDFILE ./dirrpt/ggsrep.dsc, APPEND, MEGABYTES 1000
 WILDCARDRESOLVE DYNAMIC
 Dynamicresolution
 ALLOWNOOPUPDATES
 --GETTRUNCATES
 MAP XPADB.*, TARGET XPADRPT.*;
 --OGG test table
 MAP t.*, TARGET tt.t*;

 CACHEMGR virtual memory values (may have been adjusted)
 CACHEBUFFERSIZE:                         64K
 CACHESIZE:                              512M
 CACHEBUFFERSIZE (soft max):               4M
 CACHEPAGEOUTSIZE (normal):                4M
 PROCESS VM AVAIL FROM OS (min):           1G
 CACHESIZEMAX (strict force to disk):    881M

 Database Version:
 Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
 PL/SQL Release 11.2.0.3.0 - Production
 CORE    11.2.0.3.0      Production
 TNS for IBM/AIX RISC System/6000: Version 11.2.0.3.0 - Production
 NLSRTL Version 11.2.0.3.0 - Production

 Database Language and Character Set:
 NLS_LANG = "AMERICAN_AMERICA.ZHS16GBK" 
 NLS_LANGUAGE     = "AMERICAN" 
 NLS_TERRITORY    = "AMERICA" 
 NLS_CHARACTERSET = "ZHS16GBK" 

 For further information on character set settings, please refer to user manual.

 ***********************************************************************
 **                     Run Time Messages                             **
 ***********************************************************************

 Opened trail file ./dirdat/tt002032 at 2020-12-30 09:34:14

 Wildcard MAP resolved (entry XPADB.*):
   MAP XPADB.RICH_CUSTPAYAMOUNTHIS, TARGET XPADRPT.RICH_CUSTPAYAMOUNTHIS;

 2020-12-30 09:34:14  WARNING OGG-00869  No unique key is defined for table RICH_CUSTPAYAMOUNTHIS. All viable columns will be used to represent the key, but may not guarantee uniqueness.  KEYCOLS may be used to define the key.
 Using following columns in default map by name:
   SERIALNO, TRADENO, ACCOUNT, CUSTACCOUNT, ISSUEID, FLAG, BANKID, 
   EXECDATE, CURRENCYID, AMOUNT, BANCSNO, CASHREMIT, STATE, MEMO, 
   PAYEXRATE, RESEND, EXECMODE, TRANCODE, RESPONSECODE, DATEEXPIRATION, 
   TRADECHANNEL, TSNUMBER, FREEZESRLNO, EXPDATE, AMTTYPE, RELTRANTYPE, 
   PROMOCODE, RELSERIALNO

 Using the following key columns for target table XPADRPT.RICH_CUSTPAYAMOUNTHIS: SERIALNO, TRADENO, ACCOUNT, CUSTACCOUNT, ISSUEID, FLAG, BANKID, EXECDATE, CURRENCYID, AMOUNT, BANCSNO, CASHREMIT, STATE, MEMO, PAYEXRATE, RESEND, EXECMODE, TRANCODE, RESPONSECODE, DATEEXPIRATION, TRADECHANNEL, TSNUMBER, FREEZESRLNO, EXPDATE, AMTTYPE, RELTRANTYPE, PROMOCODE, RELSERIALNO.

 2020-12-30 09:34:14  WARNING OGG-01004  Aborted grouped transaction on 'XPADRPT.RICH_CUSTPAYAMOUNTHIS', Database error 1403 ().

 2020-12-30 09:34:14  WARNING OGG-01003  Repositioning to rba 47143866 in seqno 2032.

 2020-12-30 09:34:14  WARNING OGG-01154  SQL error 1403 mapping XPADB.RICH_CUSTPAYAMOUNTHIS to XPADRPT.RICH_CUSTPAYAMOUNTHIS.

 2020-12-30 09:34:14  WARNING OGG-01003  Repositioning to rba 47143866 in seqno 2032.

 Source Context :
   SourceModule            : [er.main]
   SourceID                : [/scratch/aime2/adestore/views/aime2_staxk11/oggcore/OpenSys/src/app/er/rep.c]
   SourceFunction          : [take_rep_err_action(short, int32_t, const char *, extr_ptr_def *, std_rec_hdr_def *, char *, file_def *, bool)]
   SourceLine              : [16148]

 2020-12-30 09:34:14  ERROR   OGG-01296  Error mapping from XPADB.RICH_CUSTPAYAMOUNTHIS to XPADRPT.RICH_CUSTPAYAMOUNTHIS.

 ***********************************************************************
 *                   ** Run Time Statistics **                         *
 ***********************************************************************

 Last record for the last committed transaction is the following: 
 ___________________________________________________________________
 Trail name :  ./dirdat/tt002032
 Hdr-Ind    :     E  (x45)     Partition  :     .  (x04) 
 UndoFlag   :     .  (x00)     BeforeAfter:     A  (x41) 
 RecLength  :   854 (x0356)    IO Time    : 2017-02-26 15:37:42.331328  
 IOType     :   115  (x73)     OrigNode   :   255  (xff)
 TransInd   :     .  (x03)     FormatType :     R  (x52)
 SyskeyLen  :     0  (x00)     Incomplete :     .  (x00)
 AuditRBA   :       2301       AuditPos   : 568568848
 Continued  :     N  (x00)     RecCount   :     1  (x01)

 2017-02-26 15:37:42.331328 GGSKeyFieldComp    Len   854 RBA 47143866
 Name: XPADB.RICH_CUSTPAYAMOUNTHIS
 ___________________________________________________________________

 Reading ./dirdat/tt002032, current RBA 47143866, 0 records

 Report at 2020-12-30 09:34:14 (activity since 2020-12-30 09:34:14)

 From Table XPADB.RICH_CUSTPAYAMOUNTHIS to XPADRPT.RICH_CUSTPAYAMOUNTHIS:
        #                   inserts:         0
        #                   updates:         0
        #                   deletes:         0
        #                  discards:         1

 Last log location read:
      FILE:      ./dirdat/tt002032
      SEQNO:     2032
      RBA:       47143866
      TIMESTAMP: 2017-02-26 15:37:42.331328
      EOF:       NO
      READERR:   0

 2020-12-30 09:34:14  ERROR   OGG-01668  PROCESS ABENDING.

 CACHE OBJECT MANAGER statistics

 CACHE MANAGER VM USAGE
 vm current     =      0    vm anon queues =      0 
 vm anon in use =      0    vm file        =      0 
 vm used max    =      0    ==> CACHE BALANCED

 CACHE CONFIGURATION
 cache size       = 512M   cache force paging = 881M
 buffer min       =  64K   buffer highwater   =   4M
 pageout eligible size =   4M

 ================================================================================
 RUNTIME STATS FOR SUPERPOOL

 CACHE Transaction Stats
 trans active   =      0    max concurrent =      0 
 non-zero total =      0    trans total    =      0 

 CACHE File Caching
 disk current   =      0    disk total  =      0 
 disk caching   =      0    file cached =      0 
 file retrieves =      0 

 CACHE MANAGEMENT
 buffer links  =      0   anon gets   =      0 
 forced unmaps =      0   cnnbl try   =      0 
 cached out    =      0   force out   =      0 

 Allocation Request Distribution
 < 128B:       0 
  128B:        0         0     | 512B:        0         0 
    2K:        0         0     |   8K:        0         0 
   32K:        0         0     | 128K:        0         0 
  512K:        0         0     |   2M:        0         0 
    8M:        0         0     |  32M:        0         0 
  128M:        0         0     | 512M:        0         0 
    2G:        0         0     |   8G:        0 

 Cached Transaction Size Distribution
     0:        0 
  < 4K:        0 
    4K:        0         0     |  16K:        0         0 
   64K:        0         0     | 256K:        0         0 
    1M:        0         0     |   4M:        0         0 
   16M:        0         0     |  64M:        0         0 
  256M:        0         0     |   1G:        0         0 
    4G:        0         0     |  16G:        0         0 
   64G:        0         0     | 256G:        0         0 
    1T:        0         0     |   4T:        0         0 
   16T:        0         0     |  64T:        0         0 
  256T:        0         0     |1024T:        0         0 

 ================================================================================
 CUMULATIVE STATS FOR SUPERPOOL

 CACHE Transaction Stats
 trans active   =      0    max concurrent =      0 
 non-zero total =      0    trans total    =      0 

 CACHE File Caching
 disk current   =      0    disk total  =      0 
 disk caching   =      0    file cached =      0 
 file retrieves =      0 

 CACHE MANAGEMENT
 buffer links  =      0   anon gets   =      0 
 forced unmaps =      0   cnnbl try   =      0 
 cached out    =      0   force out   =      0 

 Allocation Request Distribution
 < 128B:       0 
  128B:        0         0     | 512B:        0         0 
    2K:        0         0     |   8K:        0         0 
   32K:        0         0     | 128K:        0         0 
  512K:        0         0     |   2M:        0         0 
    8M:        0         0     |  32M:        0         0 
  128M:        0         0     | 512M:        0         0 
    2G:        0         0     |   8G:        0 

 Cached Transaction Size Distribution
     0:        0 
  < 4K:        0 
    4K:        0         0     |  16K:        0         0 
   64K:        0         0     | 256K:        0         0 
    1M:        0         0     |   4M:        0         0 
   16M:        0         0     |  64M:        0         0 
  256M:        0         0     |   1G:        0         0 
    4G:        0         0     |  16G:        0         0 
   64G:        0         0     | 256G:        0         0 
    1T:        0         0     |   4T:        0         0 
   16T:        0         0     |  64T:        0         0 
  256T:        0         0     |1024T:        0         0 

 QUEUE Statistics:
 num queues    =     15     default index =      0 
 cur len       =      0     max len       =      0 
 q vm current  =      0     vm max        =      0 
 q hits        =      0     q misses      =      0 

 queue size  q hits  curlen  maxlen     cannibalized
   0   64K      0       0       0       0 
   1  128K      0       0       0       0 
   2  256K      0       0       0       0 
   3  512K      0       0       0       0 
   4    1M      0       0       0       0 
   5    2M      0       0       0       0 
   6    4M      0       0       0       0 
   7    8M      0       0       0       0 
   8   16M      0       0       0       0 
   9   32M      0       0       0       0 
  10   64M      0       0       0       0 
  11  128M      0       0       0       0 
  12  256M      0       0       0       0 
  13  512M      0       0       0       0 
  14    1G      0       0       0       0 

 ================================================================================
 RUNTIME STATS FOR CACHE POOL #0
 POOL INFO   group: ggsrep  id: p35717358_BLOB
 trans active  =       0   trans concurrent (max) =     0 
 trans total   =       0   (0 )
 flag          = 0x00000030
 last error    = (0=<none>)

 Allocation Request Distribution
 < 128B:       0 
  128B:        0         0     | 512B:        0         0 
    2K:        0         0     |   8K:        0         0 
   32K:        0         0     | 128K:        0         0 
  512K:        0         0     |   2M:        0         0 
    8M:        0         0     |  32M:        0         0 
  128M:        0         0     | 512M:        0         0 
    2G:        0         0     |   8G:        0 

 ================================================================================
 CUMULATIVE STATS FOR CACHE POOL #0
 POOL INFO   group: ggsrep  id: p35717358_BLOB
 trans active  =       0   trans concurrent (max) =     0 
 trans total   =       0   (0 )
 flag          = 0x00000030

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

target 端：

oracle@ZT1XPADRDB1:/gg/ogg$ ggsci

Oracle GoldenGate Command Interpreter for Oracle

Version 11.1.1.1.2 OGGCORE_11.1.1.1.2_PLATFORMS_111004.2100

AIX 5L, ppc, 64bit (optimized), Oracle 11g on Oct 5 2011 00:37:03

GGSCI (ZT1XPADRDB1) 1> info all

Program Status Group Lag Time Since Chkpt

MANAGER RUNNING

REPLICAT ABENDED GGSREP 30088:08:59 3577:00:35

GGSCI (ZT1XPADRDB1) 4> info all

Program Status Group Lag Time Since Chkpt

MANAGER RUNNING

REPLICAT ABENDED GGSREP 30088:08:59 3577:47:23

GGSCI (ZT1XPADRDB1) 5> start GGSREP

Sending START request to MANAGER ...

REPLICAT GGSREP starting

GGSCI (ZT1XPADRDB1) 6> info all

Program Status Group Lag Time Since Chkpt

MANAGER RUNNING

REPLICAT ABENDED GGSREP 33665:56:32 00:03:08

GGSCI (ZT1XPADRDB1) 8> view report GGSREP

***********************************************************************

Oracle GoldenGate Delivery for Oracle

Version 11.1.1.1.2 OGGCORE_11.1.1.1.2_PLATFORMS_111004.2100

AIX 5L, ppc, 64bit (optimized), Oracle 11g on Oct 5 2011 02:03:36

Starting at 2020-12-30 09:34:14

***********************************************************************

Operating System Version:

AIX

Version 6, Release 1

Node: ZT1XPADRDB1

Machine: 00C08DF04C00

soft limit hard limit

Address Space Size : unlimited unlimited

Heap Size : unlimited unlimited

File Size : unlimited unlimited

CPU Time : unlimited unlimited

Process id: 35717358

Description:

***********************************************************************

** Running with the following parameters **

***********************************************************************

REPLICAT ggsrep

--SETENV (ORACLE_SID = ggstd)

SETENV (NLS_LANG="AMERICAN_AMERICA.ZHS16GBK")

Set environment variable (NLS_LANG=AMERICAN_AMERICA.ZHS16GBK)

USERID goldengate, PASSWORD **********

--DDL include MAPPED

--DDLERROR DEFAULT IGNORE RETRYOP MAXRETRIES 3 RETRYDELAY 5

--DDLOPTIONS report

REPERROR DEFAULT, ABEND

ASSUMETARGETDEFS

DBOPTIONS DEFERREFCONST

DBOPTIONS SUPPRESSTRIGGERS

--BATCHSQL

--GROUPTRANSOPS 2000

DISCARDFILE ./dirrpt/ggsrep.dsc, APPEND, MEGABYTES 1000

WILDCARDRESOLVE DYNAMIC

Dynamicresolution

ALLOWNOOPUPDATES

--GETTRUNCATES

MAP XPADB.*, TARGET XPADRPT.*;

--OGG test table

MAP t.*, TARGET tt.t*;

CACHEMGR virtual memory values (may have been adjusted)

CACHEBUFFERSIZE: 64K

CACHESIZE: 512M

CACHEBUFFERSIZE (soft max): 4M

CACHEPAGEOUTSIZE (normal): 4M

PROCESS VM AVAIL FROM OS (min): 1G

CACHESIZEMAX (strict force to disk): 881M

Database Version:

Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production

PL/SQL Release 11.2.0.3.0 - Production

CORE 11.2.0.3.0 Production

TNS for IBM/AIX RISC System/6000: Version 11.2.0.3.0 - Production

NLSRTL Version 11.2.0.3.0 - Production

Database Language and Character Set:

NLS_LANG = "AMERICAN_AMERICA.ZHS16GBK"

NLS_LANGUAGE = "AMERICAN"

NLS_TERRITORY = "AMERICA"

NLS_CHARACTERSET = "ZHS16GBK"

For further information on character set settings, please refer to user manual.

***********************************************************************

** Run Time Messages **

***********************************************************************

Opened trail file ./dirdat/tt002032 at 2020-12-30 09:34:14

Wildcard MAP resolved (entry XPADB.*):

MAP XPADB.RICH_CUSTPAYAMOUNTHIS, TARGET XPADRPT.RICH_CUSTPAYAMOUNTHIS;

2020-12-30 09:34:14 WARNING OGG-00869 No unique key is defined for table RICH_CUSTPAYAMOUNTHIS. All viable columns will be used to represent the key, but may not guarantee uniqueness. KEYCOLS may be used to define the key.

Using following columns in default map by name:

SERIALNO, TRADENO, ACCOUNT, CUSTACCOUNT, ISSUEID, FLAG, BANKID,

EXECDATE, CURRENCYID, AMOUNT, BANCSNO, CASHREMIT, STATE, MEMO,

PAYEXRATE, RESEND, EXECMODE, TRANCODE, RESPONSECODE, DATEEXPIRATION,

TRADECHANNEL, TSNUMBER, FREEZESRLNO, EXPDATE, AMTTYPE, RELTRANTYPE,

PROMOCODE, RELSERIALNO

Using the following key columns for target table XPADRPT.RICH_CUSTPAYAMOUNTHIS: SERIALNO, TRADENO, ACCOUNT, CUSTACCOUNT, ISSUEID, FLAG, BANKID, EXECDATE, CURRENCYID, AMOUNT, BANCSNO, CASHREMIT, STATE, MEMO, PAYEXRATE, RESEND, EXECMODE, TRANCODE, RESPONSECODE, DATEEXPIRATION, TRADECHANNEL, TSNUMBER, FREEZESRLNO, EXPDATE, AMTTYPE, RELTRANTYPE, PROMOCODE, RELSERIALNO.

2020-12-30 09:34:14 WARNING OGG-01004 Aborted grouped transaction on 'XPADRPT.RICH_CUSTPAYAMOUNTHIS', Database error 1403 ().

2020-12-30 09:34:14 WARNING OGG-01003 Repositioning to rba 47143866 in seqno 2032.

2020-12-30 09:34:14 WARNING OGG-01154 SQL error 1403 mapping XPADB.RICH_CUSTPAYAMOUNTHIS to XPADRPT.RICH_CUSTPAYAMOUNTHIS.

2020-12-30 09:34:14 WARNING OGG-01003 Repositioning to rba 47143866 in seqno 2032.

Source Context :

SourceModule : [er.main]

SourceID : [/scratch/aime2/adestore/views/aime2_staxk11/oggcore/OpenSys/src/app/er/rep.c]

SourceFunction : [take_rep_err_action(short, int32_t, const char *, extr_ptr_def *, std_rec_hdr_def *, char *, file_def *, bool)]

SourceLine : [16148]

2020-12-30 09:34:14 ERROR OGG-01296 Error mapping from XPADB.RICH_CUSTPAYAMOUNTHIS to XPADRPT.RICH_CUSTPAYAMOUNTHIS.

***********************************************************************

* ** Run Time Statistics ** *

***********************************************************************

Last record for the last committed transaction is the following:

___________________________________________________________________

Trail name : ./dirdat/tt002032

Hdr-Ind : E (x45) Partition : . (x04)

UndoFlag : . (x00) BeforeAfter: A (x41)

RecLength : 854 (x0356) IO Time : 2017-02-26 15:37:42.331328

IOType : 115 (x73) OrigNode : 255 (xff)

TransInd : . (x03) FormatType : R (x52)

SyskeyLen : 0 (x00) Incomplete : . (x00)

AuditRBA : 2301 AuditPos : 568568848

Continued : N (x00) RecCount : 1 (x01)

2017-02-26 15:37:42.331328 GGSKeyFieldComp Len 854 RBA 47143866

Name: XPADB.RICH_CUSTPAYAMOUNTHIS

___________________________________________________________________

Reading ./dirdat/tt002032, current RBA 47143866, 0 records

Report at 2020-12-30 09:34:14 (activity since 2020-12-30 09:34:14)

From Table XPADB.RICH_CUSTPAYAMOUNTHIS to XPADRPT.RICH_CUSTPAYAMOUNTHIS:

# inserts: 0

# updates: 0

# deletes: 0

# discards: 1

Last log location read:

FILE: ./dirdat/tt002032

SEQNO: 2032

RBA: 47143866

TIMESTAMP: 2017-02-26 15:37:42.331328

EOF: NO

READERR: 0

2020-12-30 09:34:14 ERROR OGG-01668 PROCESS ABENDING.

CACHE OBJECT MANAGER statistics

CACHE MANAGER VM USAGE

vm current = 0 vm anon queues = 0

vm anon in use = 0 vm file = 0

vm used max = 0 ==> CACHE BALANCED

CACHE CONFIGURATION

cache size = 512M cache force paging = 881M

buffer min = 64K buffer highwater = 4M

pageout eligible size = 4M

================================================================================

RUNTIME STATS FOR SUPERPOOL

CACHE Transaction Stats

trans active = 0 max concurrent = 0

non-zero total = 0 trans total = 0

CACHE File Caching

disk current = 0 disk total = 0

disk caching = 0 file cached = 0

file retrieves = 0

CACHE MANAGEMENT

buffer links = 0 anon gets = 0

forced unmaps = 0 cnnbl try = 0

cached out = 0 force out = 0

Allocation Request Distribution

< 128B: 0

128B: 0 0 | 512B: 0 0

2K: 0 0 | 8K: 0 0

32K: 0 0 | 128K: 0 0

512K: 0 0 | 2M: 0 0

8M: 0 0 | 32M: 0 0

128M: 0 0 | 512M: 0 0

2G: 0 0 | 8G: 0

Cached Transaction Size Distribution

0: 0

< 4K: 0

4K: 0 0 | 16K: 0 0

64K: 0 0 | 256K: 0 0

1M: 0 0 | 4M: 0 0

16M: 0 0 | 64M: 0 0

256M: 0 0 | 1G: 0 0

4G: 0 0 | 16G: 0 0

64G: 0 0 | 256G: 0 0

1T: 0 0 | 4T: 0 0

16T: 0 0 | 64T: 0 0

256T: 0 0 |1024T: 0 0

================================================================================

CUMULATIVE STATS FOR SUPERPOOL

CACHE Transaction Stats

trans active = 0 max concurrent = 0

non-zero total = 0 trans total = 0

CACHE File Caching

disk current = 0 disk total = 0

disk caching = 0 file cached = 0

file retrieves = 0

CACHE MANAGEMENT

buffer links = 0 anon gets = 0

forced unmaps = 0 cnnbl try = 0

cached out = 0 force out = 0

Allocation Request Distribution

< 128B: 0

128B: 0 0 | 512B: 0 0

2K: 0 0 | 8K: 0 0

32K: 0 0 | 128K: 0 0

512K: 0 0 | 2M: 0 0

8M: 0 0 | 32M: 0 0

128M: 0 0 | 512M: 0 0

2G: 0 0 | 8G: 0

Cached Transaction Size Distribution

0: 0

< 4K: 0

4K: 0 0 | 16K: 0 0

64K: 0 0 | 256K: 0 0

1M: 0 0 | 4M: 0 0

16M: 0 0 | 64M: 0 0

256M: 0 0 | 1G: 0 0

4G: 0 0 | 16G: 0 0

64G: 0 0 | 256G: 0 0

1T: 0 0 | 4T: 0 0

16T: 0 0 | 64T: 0 0

256T: 0 0 |1024T: 0 0

QUEUE Statistics:

num queues = 15 default index = 0

cur len = 0 max len = 0

q vm current = 0 vm max = 0

q hits = 0 q misses = 0

queue size q hits curlen maxlen cannibalized

0 64K 0 0 0 0

1 128K 0 0 0 0

2 256K 0 0 0 0

3 512K 0 0 0 0

4 1M 0 0 0 0

5 2M 0 0 0 0

6 4M 0 0 0 0

7 8M 0 0 0 0

8 16M 0 0 0 0

9 32M 0 0 0 0

10 64M 0 0 0 0

11 128M 0 0 0 0

12 256M 0 0 0 0

13 512M 0 0 0 0

14 1G 0 0 0 0

================================================================================

RUNTIME STATS FOR CACHE POOL #0

POOL INFO group: ggsrep id: p35717358_BLOB

trans active = 0 trans concurrent (max) = 0

trans total = 0 (0 )

flag = 0x00000030

last error = (0=<none>)

Allocation Request Distribution

< 128B: 0

128B: 0 0 | 512B: 0 0

2K: 0 0 | 8K: 0 0

32K: 0 0 | 128K: 0 0

512K: 0 0 | 2M: 0 0

8M: 0 0 | 32M: 0 0

128M: 0 0 | 512M: 0 0

2G: 0 0 | 8G: 0

================================================================================

CUMULATIVE STATS FOR CACHE POOL #0

POOL INFO group: ggsrep id: p35717358_BLOB

trans active = 0 trans concurrent (max) = 0

trans total = 0 (0 )

flag = 0x00000030

target端告警日志：

oracle@ZT1XPADRDB1:/gg/ogg$ tail -f ggserr.log

2020-12-30 09:27:54 WARNING OGG-00956 Oracle GoldenGate Manager for Oracle, mgr.prm: hours_since_modified calculated as -39947, modtime is 212619862981000000 for /gg/ogg/dirdat/tt001891.

2020-12-30 09:27:54 WARNING OGG-00956 Oracle GoldenGate Manager for Oracle, mgr.prm: hours_since_modified calculated as -39951, modtime is 212619877906000000 for /gg/ogg/dirdat/tt001892.

2020-12-30 09:27:54 WARNING OGG-00956 Oracle GoldenGate Manager for Oracle, mgr.prm: hours_since_modified calculated as -40038, modtime is 212620189201000000 for /gg/ogg/dirdat/tt001893.

2020-12-30 09:27:54 WARNING OGG-00950 Oracle GoldenGate Manager for Oracle, mgr.prm: Purge of old extract file /gg/ogg/dirdat/tt001894 failed because the prev seqno exists. Purge rule was applying UseCheckPoints purge rule: Oldest Chkpt Seqno 2032 > 1894.

2020-12-30 09:34:14 INFO OGG-00987 Oracle GoldenGate Command Interpreter for Oracle: GGSCI command (oracle): start GGSREP.

2020-12-30 09:34:14 INFO OGG-00963 Oracle GoldenGate Manager for Oracle, mgr.prm: Command received from GGSCI on host 22.188.132.82 (START REPLICAT GGSREP ).

2020-12-30 09:34:14 INFO OGG-00975 Oracle GoldenGate Manager for Oracle, mgr.prm: REPLICAT GGSREP starting.

2020-12-30 09:34:14 INFO OGG-00995 Oracle GoldenGate Delivery for Oracle, ggsrep.prm: REPLICAT GGSREP starting.

2020-12-30 09:34:14 INFO OGG-00996 Oracle GoldenGate Delivery for Oracle, ggsrep.prm: REPLICAT GGSREP started.

2020-12-30 09:34:14 WARNING OGG-00869 Oracle GoldenGate Delivery for Oracle, ggsrep.prm: No unique key is defined for table RICH_CUSTPAYAMOUNTHIS. All viable columns will be used to represent the key, but may not guarantee uniqueness. KEYCOLS may be used to define the key.

2020-12-30 09:34:14 WARNING OGG-01004 Oracle GoldenGate Delivery for Oracle, ggsrep.prm: Aborted grouped transaction on 'XPADRPT.RICH_CUSTPAYAMOUNTHIS', Database error 1403 ().

2020-12-30 09:34:14 WARNING OGG-01003 Oracle GoldenGate Delivery for Oracle, ggsrep.prm: Repositioning to rba 47143866 in seqno 2032.

2020-12-30 09:34:14 WARNING OGG-01154 Oracle GoldenGate Delivery for Oracle, ggsrep.prm: SQL error 1403 mapping XPADB.RICH_CUSTPAYAMOUNTHIS to XPADRPT.RICH_CUSTPAYAMOUNTHIS.

2020-12-30 09:34:14 WARNING OGG-01003 Oracle GoldenGate Delivery for Oracle, ggsrep.prm: Repositioning to rba 47143866 in seqno 2032.

2020-12-30 09:34:14 ERROR OGG-01296 Oracle GoldenGate Delivery for Oracle, ggsrep.prm: Error mapping from XPADB.RICH_CUSTPAYAMOUNTHIS to XPADRPT.RICH_CUSTPAYAMOUNTHIS.

2020-12-30 09:34:14 ERROR OGG-01668 Oracle GoldenGate Delivery for Oracle, ggsrep.prm: PROCESS ABENDING.

2020-12-30 09:37:54 WARNING OGG-00956 Oracle GoldenGate Manager for Oracle, mgr.prm: hours_since_modified calculated as -39947, modtime is 212619862981000000 for /gg/ogg/dirdat/tt001891.

2020-12-30 09:37:54 WARNING OGG-00956 Oracle GoldenGate Manager for Oracle, mgr.prm: hours_since_modified calculated as -39951, modtime is 212619877906000000 for /gg/ogg/dirdat/tt001892.

2020-12-30 09:37:54 WARNING OGG-00956 Oracle GoldenGate Manager for Oracle, mgr.prm: hours_since_modified calculated as -40038, modtime is 212620189201000000 for /gg/ogg/dirdat/tt001893.

2020-12-30 09:37:54 WARNING OGG-00950 Oracle GoldenGate Manager for Oracle, mgr.prm: Purge of old extract file /gg/ogg/dirdat/tt001894 failed because the prev seqno exists. Purge rule was applying UseCheckPoints purge rule: Oldest Chkpt Seqno 2032 > 1894.

故障分析及解决过程

确认归档文件是否存在

根据报错信息，可以知道source端的thread 1 2598归档日志不存在了，如果备份存在的话可以先把备份的归档还原回来就可以了，我们看看归档是否存在？

oracle@ZTHXPADDB2:/oracle$ sqlplus / as sysdba

SQL*Plus: Release 10.2.0.5.0 - Production on Wed Dec 30 08:50:48 2020

Connected to:

Oracle Database 10g Enterprise Edition Release 10.2.0.5.0 - 64bit Production

With the Partitioning, Real Application Clusters, Data Mining and Real Application Testing options

SQL> set line 9999

SQL> select * from v$log;

GROUP# THREAD# SEQUENCE# BYTES MEMBERS ARC STATUS FIRST_CHANGE# FIRST_TIME

---------- ---------- ---------- ---------- ---------- --- ---------------- ------------- ------------

1 1 2907 1072693248 1 NO CURRENT 1.2242E+13 30-DEC-20

2 1 2905 1072693248 1 YES INACTIVE 1.2242E+13 28-DEC-20

3 1 2906 1072693248 1 YES INACTIVE 1.2242E+13 29-DEC-20

4 2 2718 1072693248 1 YES INACTIVE 1.2242E+13 29-DEC-20

5 2 2719 1072693248 1 NO CURRENT 1.2242E+13 29-DEC-20

6 2 2717 1072693248 1 YES INACTIVE 1.2242E+13 28-DEC-20

6 rows selected.

SQL> archive log list;

Database log mode Archive Mode

Automatic archival Enabled

Archive destination /gg/sarch2

Oldest online log sequence 2717

Next log sequence to archive 2719

Current log sequence 2719

SQL> exit

Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.5.0 - 64bit Production

With the Partitioning, Real Application Clusters, Data Mining and Real Application Testing options

oracle@ZTHXPADDB2:/oracle$ cd /gg/sarch2

oracle@ZTHXPADDB2:/gg/sarch2$ l

total 193332080

-rw-rw---- 1 oracle dba 1065581568 Mar 05 2020 2_2625_704996932.arc

-rw-rw---- 1 oracle dba 1065574400 Mar 06 2020 2_2626_704996932.arc

-rw-rw---- 1 oracle dba 1065574400 Mar 06 2020 2_2627_704996932.arc

-rw-rw---- 1 oracle dba 1065577472 Mar 07 2020 2_2628_704996932.arc

-rw-rw---- 1 oracle dba 1065568768 Mar 20 2020 2_2629_704996932.arc

-rw-rw---- 1 oracle dba 1065582592 Mar 21 2020 2_2630_704996932.arc

《《《《。。。。。。。。篇幅原因，有省略。。。。。。。。》》》》

-rw-rw---- 1 oracle dba 1065574400 Dec 23 13:06 2_2707_704996932.arc

-rw-rw---- 1 oracle dba 1065574400 Dec 24 00:26 2_2708_704996932.arc

-rw-rw---- 1 oracle dba 1065574912 Dec 24 13:15 2_2709_704996932.arc

-rw-rw---- 1 oracle dba 1065744384 Dec 25 02:23 2_2710_704996932.arc

-rw-rw---- 1 oracle dba 1065566208 Dec 25 19:23 2_2711_704996932.arc

-rw-rw---- 1 oracle dba 1065576960 Dec 26 09:08 2_2712_704996932.arc

-rw-rw---- 1 oracle dba 1065627648 Dec 26 23:21 2_2713_704996932.arc

-rw-rw---- 1 oracle dba 1065574400 Dec 27 12:25 2_2714_704996932.arc

-rw-rw---- 1 oracle dba 1065691648 Dec 28 04:32 2_2715_704996932.arc

-rw-rw---- 1 oracle dba 1065658368 Dec 28 21:02 2_2716_704996932.arc

-rw-rw---- 1 oracle dba 1065575936 Dec 29 10:37 2_2717_704996932.arc

-rw-rw---- 1 oracle dba 1067223552 Dec 29 23:22 2_2718_704996932.arc

-rw-r--r-- 1 oracle dba 79 Jun 27 2013 afiedt.buf

oracle@ZTHXPADDB2:/gg/sarch2$ l *1_2598*

ls: 0653-341 The file *1_2598* does not exist.

oracle@ZTHXPADDB2:/gg/sarch2$ cd /gg/sarch1

oracle@ZTHXPADDB2:/gg/sarch1$ l

total 166951440

-rw-rw---- 1 oracle dba 1065574400 Mar 05 2020 1_2825_704996932.arc

-rw-rw---- 1 oracle dba 1065580032 Mar 06 2020 1_2826_704996932.arc

-rw-rw---- 1 oracle dba 1065574400 Mar 06 2020 1_2827_704996932.arc

-rw-rw---- 1 oracle dba 1065574400 Mar 07 2020 1_2828_704996932.arc

《《《《。。。。。。。。篇幅原因，有省略。。。。。。。。》》》》

-rw-rw---- 1 oracle dba 1065574400 Dec 27 10:52 1_2902_704996932.arc

-rw-rw---- 1 oracle dba 1065574400 Dec 28 00:59 1_2903_704996932.arc

-rw-rw---- 1 oracle dba 1065687040 Dec 28 15:16 1_2904_704996932.arc

-rw-rw---- 1 oracle dba 1065607680 Dec 29 08:40 1_2905_704996932.arc

-rw-rw---- 1 oracle dba 1065577984 Dec 30 01:32 1_2906_704996932.arc

-rw-rw---- 1 oracle dba 278522368 Aug 06 07:15 2_2689_704996932.arc

-rw-r--r-- 1 oracle dba 90 Dec 22 2013 afiedt.buf

oracle@ZTHXPADDB2:/gg/sarch1$

RMAN> list archivelog all;

List of Archived Log Copies

Key Thrd Seq S Low Time Name

------- ---- ------- - --------- ----

4651 1 2825 A 05-MAR-20 /gg/sarch1/1_2825_704996932.arc

4653 1 2826 A 05-MAR-20 /gg/sarch1/1_2826_704996932.arc

4655 1 2827 A 06-MAR-20 /gg/sarch1/1_2827_704996932.arc

4656 1 2828 A 06-MAR-20 /gg/sarch1/1_2828_704996932.arc

4658 1 2829 A 07-MAR-20 /gg/sarch1/1_2829_704996932.arc

4660 1 2830 A 20-MAR-20 /gg/sarch1/1_2830_704996932.arc

4662 1 2831 A 20-MAR-20 /gg/sarch1/1_2831_704996932.arc

4665 1 2832 A 21-MAR-20 /gg/sarch1/1_2832_704996932.arc

4667 1 2833 A 30-MAR-20 /gg/sarch1/1_2833_704996932.arc

4670 1 2834 A 30-MAR-20 /gg/sarch1/1_2834_704996932.arc

《《《《。。。。。。。。篇幅原因，有省略。。。。。。。。》》》》

4793 2 2702 A 02-OCT-20 /gg/sarch2/2_2702_704996932.arc

4795 2 2703 A 20-DEC-20 /gg/sarch2/2_2703_704996932.arc

4796 2 2704 A 21-DEC-20 /gg/sarch2/2_2704_704996932.arc

4798 2 2705 A 21-DEC-20 /gg/sarch2/2_2705_704996932.arc

4801 2 2706 A 22-DEC-20 /gg/sarch2/2_2706_704996932.arc

4803 2 2707 A 22-DEC-20 /gg/sarch2/2_2707_704996932.arc

4804 2 2708 A 23-DEC-20 /gg/sarch2/2_2708_704996932.arc

4806 2 2709 A 24-DEC-20 /gg/sarch2/2_2709_704996932.arc

4808 2 2710 A 24-DEC-20 /gg/sarch2/2_2710_704996932.arc

4811 2 2711 A 25-DEC-20 /gg/sarch2/2_2711_704996932.arc

4813 2 2712 A 25-DEC-20 /gg/sarch2/2_2712_704996932.arc

4815 2 2713 A 26-DEC-20 /gg/sarch2/2_2713_704996932.arc

4817 2 2714 A 26-DEC-20 /gg/sarch2/2_2714_704996932.arc

4819 2 2715 A 27-DEC-20 /gg/sarch2/2_2715_704996932.arc

4821 2 2716 A 28-DEC-20 /gg/sarch2/2_2716_704996932.arc

4823 2 2717 A 28-DEC-20 /gg/sarch2/2_2717_704996932.arc

4824 2 2718 A 29-DEC-20 /gg/sarch2/2_2718_704996932.arc

可以看到 2598号的归档日志已经不存在了。

第一种办法：改变抽取进程的时间

改变抽取进程的时间，但是会导致数据不一致，若是测试库，可以执行：alter extract extl,begin now

GGSCI (ZTHXPADDB2) 11> alter extract GGSEXT,begin now

EXTRACT altered.

GGSCI (ZTHXPADDB2) 12> info all

Program Status Group Lag Time Since Chkpt

MANAGER RUNNING

EXTRACT STOPPED GGSEXT 00:00:00 00:00:05

EXTRACT RUNNING GGSPUMP 00:00:00 00:00:00

GGSCI (ZTHXPADDB2) 13> start GGSEXT

Sending START request to MANAGER ...

EXTRACT GGSEXT starting

GGSCI (ZTHXPADDB2) 14> info all

Program Status Group Lag Time Since Chkpt

MANAGER RUNNING

EXTRACT RUNNING GGSEXT 00:00:00 00:00:28

EXTRACT RUNNING GGSPUMP 00:00:00 unknown

GGSCI (ZTHXPADDB2) 17> info all

Program Status Group Lag Time Since Chkpt

MANAGER RUNNING

EXTRACT RUNNING GGSEXT 00:00:00 unknown

EXTRACT RUNNING GGSPUMP 00:00:00 00:00:00

GGSCI (ZTHXPADDB2) 18> info all

Program Status Group Lag Time Since Chkpt

MANAGER RUNNING

EXTRACT RUNNING GGSEXT 00:00:00 unknown

EXTRACT RUNNING GGSPUMP 00:00:00 unknown

GGSCI (ZTHXPADDB2) 19> info all

Program Status Group Lag Time Since Chkpt

MANAGER RUNNING

EXTRACT RUNNING GGSEXT 00:00:00 unknown

EXTRACT RUNNING GGSPUMP 00:00:00 00:00:01

可以看到虽然处于RUNNING的状态，但是Time Since Chkpt列是不对的，而这是target端也不能正常启动。