Troubleshooting
Problem
This technote explains why attempts to import an IBM® Rational® ClearCase MultiSite® packet results in the error, "multitool: Error: Replica incarnation for "
Symptom
A replica's incarnation time is set to the last time the replica was restored using the restorereplica procedure. When a replica is initially created this value is set to 0 with a default date of 1970-01-01T00:00:00 and it remains at 0 until a restore is done. The incarnation is used to prevent stale packets (packets that were created before the restore operation) from being replayed and possibly causing divergence. During packet creation, the exporting site's belief of the incarnation for each target replica is placed in the header of the packet.
During the import, the incarnation in the packets header <old-timestamp> is checked against the incarnation of the importing replica <new-timestamp>. If the importing replica's incarnation is later, the Replica incarnation is old error will appear indicating that there has been a restorereplica done that the exporter is unaware of.
Cause
There are a few possible reasons for this which include the following:
====================================
Cause 1
The packet was created before the restorereplica procedure was started. This packet is stale and should be discarded to prevent possible divergence.
By default packets are created with the date and time of creation in the name. After adjusting for time differences between the two sites, compare that time on the importing packet to the expected time in the incarnation error. If the packet creation time is earlier than the incarnation time, this is the most likely cause.
Cause 2
The restorereplica oplog gets removed when the restorereplica is completed. If one or more replicas are unable to import the packet containing the restorereplica oplog, the incarnation will not be updated at those replicas and all future packets generated from them will produce the Replica incarnation is old error upon import at replicas that do have the correct incarnation date.
This can be observed under the following circumstances:
- The syncing pattern does not update certain replicas regularly.
For example, replica A only sends packets to replica B regularly and B sends packets to replica C but only once a week so C never gets the restorereplica oplog.
- The packet was lost or discarded before the remote replica had a chance to import it and the override switch was used to finish the restore procedure without an update from that replica.
- An optimized restorereplica procedure was used and only the replicas included in the request for updates were sent an update packet containing the restorereplica oplog.
Example:
Replica A goes to backup and begins a restorereplica sequence. Since its most recent sync was to replica B it optimizes the restore process by specifying only replica B be sent the restore replica oplog.
When replica A receives the return update from B it sets the state of A to restore complete. This in turn removes the restorereplica oplog at replica A and the next time replica B receives an update from A it too removes the restorereplica oplog.
If B did not sync to C while it held A's restorereplica oplog, C will never be notified that A has been restored. The next time C syncs to A, A's target header in the packet will contain the incarnation time of A before it was restored.
Replica A will refuse to import this packet because the incarnation time will be old.
Optimization and the override switch should only be used when replicas are unable to send a reply back and those replicas do not contain any oplogs required to complete the restoration procedure.
Even if the optimized procedure was used make sure you export a sync packet to all replicas when you are doing the export for first time after starting the restorereplica command to prevent this error. It is not required to get the sync packets back from all the replicas to complete restoration. Since the restorereplica was initiated for one or only a few replicas, the restored replica will expect the packets from only those replicas it sent requests to in order to complete the restoration process. It is only required to send export packets to all replicas so that the incarnation date for the restoring replica is communicated to all sites.
Cause 3
A stale restorereplica oplog can cause the incarnation date to repeatedly return to the old incarnation date. Review technote 1131690 for more information on and causes of stale restorereplica oplogs.
Resolving The Problem
Solution 1
Remove the packet, and regenerate it from the sending site, after ensuring that site is up-to-date.
Solution 2
If the problem persists after you are sure that the sending site is up-to-date, follow the procedure below:
INSTRUCTIONS:
- Verify the incarnation time for <replica-name>..
At the importing site run cleartool dump of the replica object (cleartool dump replica:<replica_name>) to display the current replica's record of the incarnation time.
Example:
C:\>cleartool dump replica:home@\multisite
home
oid=7898a7d8.6bda4b32.a638.28:9c:e7:61:b0:6e dbid=3 (0x80eb5d)
mtype=replica name="home" type=2
usid=NT:S-1-5-21-4179696171-2760821386-689065543-500
sid=NT:S-1-5-21-4179696171-2760821386-689065543-1011
master replica dbid=3
flags: predefined, preserve_both
host name="host"
incarnation=0
epoch row:
replica= 10071 oplog id= 0
replica= 10149 oplog id= 0
replica= 3 oplog id= 82
attributes:
FeatureLevel = 5
The line beginning with "incarnation=" will contain a time stamp if a restorereplica has been run. The time will be in either UTC format (Universal Time Coordinated) or local time depending both on the version of MultiSite in use and or any variables in use. In either case, this should match the <new-timestamp> in the error message.
- Determine the mastership of the replica and whether it is identity-preserving.
At any site describe the replica to display the mastership and identity of the replica:
Example:
C:\>cleartool describe replica:home@\multisite
replica "home"
created 07-Jul-06.08:06:12 by user.group@HOST
replica type: unfiltered
master replica: home@\multisite
request for mastership: disabled
owner: DOM\user
group: DOM\group
host: "host"
identities: preserved
permissions: preserved
feature level: 5
connectivity: connected
Attributes:
FeatureLevel = 5
- At the master replica site (as seen from the describe output above) change the replica incarnation time.
Use the following command to change the replica incarnation date:
multitool chreplica -incarnation <new-timestamp> replica:<replica-name>
Example:
multitool chreplica -incarnation 10-Oct-2004.11:20:23UTC replica:home@\multisite
Note: Even if the master of the replica shows the correct incarnation date, and all the replicas appear to be up to date, issue the chreplica -incarnation command at the replica master anyway and export a sync packet to the other sites immediately. Import that sync packet at the other sites immediately and verify that the incarnation date has been updated.
- Export the packet to all replicas in the family.
- After the original sending site has imported this packet, it can generate a packet that the original receiver can import.
If the errors persist after following the above procedures, contact Rational Client Support supplying the following information:
- Full text of the import error.
- Output of the multisiteinfo.pl script from the replica importer, exporter and the recovery site identifying the replica that generated each output file sent in.
Solution 3
Review technote 1131690 for information on finding and removing stale restorereplica oplogs. Once the stale restorereplica oplog has been removed the incarnation date should revert back to the old date
If the incorrect incarnation date is coming from a deleted replica, special tools and instructions will be needed. Contact Rational Client Support for further assistance and supply the following information.
- The full error message from the import.
- "multisiteinfo" - see technote 1125539
Related Information
Was this topic helpful?
Document Information
Modified date:
16 June 2018
UID
swg21151039