Troubleshooting media database corruption

 This post is about how to repair simple media database corruption, and media database compression

Symptoms & Error Messages

• Media database corruption
• Error: ‘media database must be scavenged’
• Error: ‘nsrmmdbd: WISS error, database was not shutdown cleanly’
• Index corruption
• Error: ‘error on open of recfile for index [index-name]’
• Error: ‘Cannot mount index [index-name]’
• Error: ‘nothing in index at this time’
• Unable to perform backup due to index corruption
• Unable to perform recovery due to index corruption
• Slow writing to tape
• Error: ‘nsrmmdbd: WISS error: Unable to mount /nsr/mm/mmvolume6: bad database header’
• Error: ‘nsrmmdbd: WARNING: clients file missing from /nsr/mm/mmvolume6’
• Error: ‘nsrmmdbd: WISS error, database was not shutdown cleanly’
• Error: ‘nsrd: nsrmmdbd has exited on signal 11’
• Error: ‘nsrmmdbd: WISS error: invalid slot number’
• Error: ‘nsrmon #(number): query resdb failed, err: Cancelled.’
• Error: ‘nsrmmdbd: media db must be scavenged’
• Error: ‘media database must be scavenged’
• Error: ‘bad database header’
• Error: ‘WISS Error’
• Error: ‘nsrmmdbd: Error on close of volume index file #x (invalid file number)’
• mmlocate loses locations of volumes when NetWorker daemons are restarted
• Error: ‘save: error, media index problem’

Note: Not all of the Symptoms listed above will be found in all Cases. However, each is indicative of index corruption; therefore the fixes listed below are relevant to each situation.

The above noted error messages are typically the result of problems in the media database. Corruption in the media database could be caused by any number of events, such as hardware failure, or improperly shutting down NetWorker daemons. The general approach to resolving any fatal media database problem is to use the nsrim, and nsrck tools. If those tools did not solve the issue, so you have to proceed with scavenging the Media Database.

Follow the steps listed below to perform media database compression and consistency checks to correct possible media database corruption. You may also follow this procedure to simply compress the backup server’s indexes (Client File Index, and Media Database) as part of periodic system maintenance.

Media Database Consistency Check / Compression:

1. Ensure that there are no running jobs (backup, recovery, cloning, etc…)
2. Stop NetWorker daemons/services (NetWorker Backup and Recovery Server and NetWorker Remote Exec Service)
3. Empty the files in the /nsr/tmp directory
4. Delete /nsr/mm/cmprssd file  ( which will force NetWorker to compress the media db)
5. Start NetWorker Services again (first start the NetWorker Remote Exec Service and then NetWorker Backup and Recovery Server)
6. Run the following command line: nsrim -X
7. Run the following command line: nsrck -L6

If the problem has not been solved with the above steps, proceed with the following procedures for Media Database Scavenging:

Media Database Scavenging:

1. Make sure you have a bootstrap backup before commencing.
2. Ensure that there are no running jobs (backup, recovery, cloning, etc…)
3. Stop NetWorker daemons/services
4. Cause a “controlled corruption” by removing:  /nsr/mm/mmvolume6/{*_i*,ss,vol}
5. BE VERY CAREFUL WITH THIS STEP, Please read it all before proceeding. In /nsr/mm/mmvolume6 delete all files EXCEPT the 4 files (ss.0, vol.0, clients.0, ss.1, vol.0 , clients.0,ss.1, clients.1, vol.2, ss.2, clients.2, VolHdr ) You MUST keep those files.

NOTE:  Be careful when deleting the files and make sure not to delete the (vol.0, ss.0, clients.0, vol.1, ss.1, clients.1, vol.2, ss.2, clients.2, vol.3, ss.3, clients.3, etc.) These files are NEEDED to rebuild the media database. If you do (# rm ss*), this will delete ss.0 also and basically the media db is gone.

Note: this step is simply a way to force NetWorker to rebuild the media database. The files that are deleted are all rebuild-able from the media db file. The files mentioned above and cautioned to not delete them are not rebuild-able, NetWorker will not be able to rebuild your media db without these files (ss.0 and vol.0, volhdr, clients.0 , etc…) so take care well in this step.

6. Empty the files in the /nsr/tmp directory

7. Restart NetWorker daemons/service

In the situation where the corruption is severe and the above procedure did not resolve the media database corruption, the media database can be recovered from the bootstrap using the mmrecov command to get a good copy of the media database. NetWorker software attempts to generate a bootstrap every day, so no more than one day of data should be lost.