EMC Technical Advisory
ETA esg112952: NetWorker: Possible Performance Degradation. Hang or Crash of Oracle Databases after Extended Period of Backups on Windows.
Products
NetWorker Module for Oracle 5.0 (Windows only)
NetWorker Module for Oracle 4.5 (Windows only)
NetWorker Module for Databases and Applications 1.0 – only if protecting Oracle (Windows only)
NetWorker Module for SAP with Oracle 4.0 – only if using the BRTools RMAN feature (Windows only)
NetWorker Module for SAP with Oracle 3.5 – only if using the BRTools RMAN feature (Windows only)
Symptoms
A gradual increase in the number of handles used by the NetWorker module in the Oracle database service leads to degradation of Oracle database performance on Windows.
After extended periods of time between Oracle database restarts, the Oracle database server might hang or crash.
Issue
On Windows systems, every time a manual or scheduled backup runs, the NetWorker module opens handles, but the module does not close all the open handles when the backup is completed. As a result, a handle leak occurs in the Oracle database service. Over a period of months of recurring backups of an Oracle database, if the database is never restarted, the performance of the Oracle database might degrade due to the handle leak. As time passes, the handle leak might also cause a crash or hang in the Oracle database server.
Resolution
All customers using any of the following products are advised to apply the hotfix immediately:
NetWorker Module for Oracle (NMO) 4.5 and 5.0 on Windows
NetWorker Module for Databases and Applications (NMDA) 1.0 on Windows (if protecting Oracle)
NetWorker Module for SAP with Oracle (NMSAP) 3.5 and 4.0 on Windows (if using the BRTools RMAN feature)
1. Access the hotfixes for download at ftp.legato.com
:
* For NMO 4.5, go to
ftp://ftp.legato.com/pub/NetWorker/Updates/NW112917/nmo45
* For NMO 5.0, go to
ftp://ftp.legato.com/pub/NetWorker/Updates/NW112917/nmo50
* For NMDA 1.0, go to
ftp://ftp.legato.com/pub/NetWorker/Updates/NW112917/nmda10/oracle * For NMSAP 3.5, go to
ftp://ftp.legato.com/pub/NetWorker/Updates/NW112917/nmsap35
* For NMSAP 4.0, go to
ftp://ftp.legato.com/pub/NetWorker/Updates/NW112917/nmsap40
2. Download the hotfix for the appropriate platform of the Oracle client to a tmp directory on the client machine:
Hotfix directory Operating system
win_x86 Windows 32-bit Intel
win_x64 Windows 64-bit Intel
win_ia64 Windows 64-bit Itanium
3. Rename the file \bin\orasbt.dll to \bin\orasbt.orig on the client.
By default, is C:\Program Files\Legato\nsr.
4. Move the downloaded file orasbt.dll from the tmp directory to \bin.
5. Restart the Oracle service.
Note: This hotfix will be included in the future NMDA and NMSAP releases, starting with NMDA 1.1 and NMSAP 4.1.
EMC issues Technical Advisories (ETAs) to communicate information to customers.
ETAs are articles from our Knowledgebase that address:
* Critical problems that cannot be prevented or controlled through our proactive tools and processes.
* Product defects that might cause unexpected behavior or any kind of data loss.
Source : http://solutions.emc.com/emcsolutionview.asp?id=esg112952
EMC has announcement the General Availability of NetWorker Module for Microsoft Applications 2.2 SP1 which delivers a unified solution for protection of Microsoft Server applications. It enables EMC NetWorker to leverage Microsoft’s Volume Shadow Copy Service (VSS) feature to create and manage consistent point-in-time snapshot data copies for quick recovery and off-host backup. NetWorker users can capture more frequent recovery points on disk and recover information in a fraction of the time it takes to restore from a tape, lessening the impact of data protection operations on production servers.
EMC NetWorker Module for Microsoft Applications Features & Benefits:
Low-impact backups:
Leverage instant low-impact snaps that can be used for recovery and off-host backups.
Business application protection:
Improve backup and recovery for Microsoft Windows, Exchange, SQL Server, SharePoint, Data Protection Manager, and Active Directory.
Backup and operational recovery:
Complete backup and recovery in minutes using NetWorker to manage VSS snapshot scheduling, creation, and restore.
Snap management:
Perform easy shadow copy scheduling, management, and restoration within EMC NetWorker Management Console.
Unified recovery:
Leverage a single common recovery user interface for all supported Microsoft server applications.
EMC VSS hardware provider support:
Integrate with the EMC VSS provider to enable creation and transport of snapshots on EMC CLARiiON, EMC Symmetrix, and EMC Celerra hardware.
Proxy-host backup:
Enable any Microsoft Windows SAN-based host to access back up and recover VSS snapshots.
For full details on fixes and enhancements in this release, please see the release notes on Powerlink.
Here i collect All the services exists on the NetWorker Server, Each by name and its own role and duties.
ansrd:
Monitors an active recover session. It is called by nsrd in response to a recover session.
gstd:
Known as the Generic Services Toolkit (GST), controls other services provided by the Console server.
hagentd:
Manages the NetWorker host agent modules and maintains the NetWorker host agent resource database.
jobsd:
Manages monitoring and other communication between the NetWorker server and the NetWorker console.
nsrd:
Monitors an active save or recover program session. This is an agent process spawned by nsrd in response to a save or recover session. It (nsrd) is also the master service that controls other services on the NetWorker server, clients, and storage nodes.
nsrck:
Checks the consistency of the client file index. It is invoked by nsrd whenever the consistency of the client file index needs to be confirmed.
nsrim:
Automatically manages the server media database. It is invoked by nsrmmdbd when it starts up at the end of the savegrp program. It removes both aborted and expired save sets once every 24 hours after a savegroup is completed (if Volume Recycle is set to Auto). It (nsrim) is also called by nsrd when a user removes the oldest backup cycle.
nsrindexd:
Provides a method for inserting entries into the client file index that is based on information passed by the save program.
nsrlogd:
Allows logging of security audit messages from any NetWorker machine.
nsrmmd:
Provides device support, generates mount requests, and multiplexes save set data during a multi-client backup. It writes the data sent by save to storage media. An additional nsrmmd service is started while mount requests are pending. The nsrd service starts one nsrmmd service for each enabled device. Forwards storage information to nsrmmdbd for recording in the NetWorker server media database.
nsrmmdbd:
Provides media database management services to the local nsrd and nsrmmd services and records entries in the media database. This is the media management database service.
nsrmmgd:
Manages media library operations. It provides an RPC-based service that manages all jukebox operations on behalf of the nsrd service. The nsrd service starts one and only one instance of nsrmmgd on the NetWorker server as needed.
nsrlcpd:
Provides nsrmmgd with a uniform interface to supported libraries. One nsrlcpd service is started by the nsrmmgd service for each virtual library instance. The nsrlcpd services run on the storage node that controls the library that the service manages.
nsrpm:
Windows OS ONLY service that Works with the energy-saving features of Windows to keep NetWorker components running properly during power-saving modes. If it is not required, it can be disabled.
savegrp:
Runs a group of NetWorker clients through the save process
In NetWorker release 7.0 and later, all devices created on storage nodes (except servers) include the Dedicated Storage Node attribute. A dedicated storage node can back up only its own, local data.
So Dedicated Storage Node (DSN) License is required, You can authorize a license from EMC), But the question here is How to activate working with your DSN license ?
Here is the steps:
1- Navigate from the NMC GUI to the Devices Tab.
2- Choose Devices in the left pane.
3- Choose one device ( any device ) of the desired storage node which you want to convert it to Dedicated Storage Node and right click on that device and choose Properties.
4- Navigate to the Configuration tab.
5- Set the Dedicated Storage Node attribute to YES.
6- This action will convert all the devices associated to this storage node to Dedicated Storage Node attribute because a storage node host cannot mix storage node types. Either all devices on a storage node must be set up for a dedicated storage node, or all must be set up for a standard storage node. Means that you don’t need to change every device properties, Just change one device and all the other devices associated to this storage node will be converted automatically.
Note: Dedicated Storage Node attribute default value is set to No which require a standard storage node license.
Note: The Dedicated Storage Node License also can be used for backing up virtual clients in a cluster.
I would like to share with you this small tricky hint that allows you to direct the output of nsradmin query to a file to be as a report file, here you will find the procedures:
If we say as an example we want to query all the save sets of all the clients associated to the NetWorker server:
1- create two txt files on the C:\ partition (or anywhere you plan to) . one with name Input.txt (That will hold the query commands) and the other is output.txt ( which will handle the output results).
2- Edit the input.txt file with the nsradmin command , In our example it will be the following:
show save set: ; name:
print type: NSR client
(That will shows the client name and its save sets)
Note: the trick here is to add one or two blank lines at
the bottom of this file.
3- Save the input.txt
4- Then from the command prompt navigate to the C:\ prompt and type nsradmin -i input.txt > output.txt ( Just provide the output file path)
5- This will take some while, but after it finishes you will find all the outputs in output.txt file.
Another nsradmin command example can be querying the last successful backup time of each group, that can be done by using the same previous steps buy with the following commands in the input.txt.
show time of the last successful backup: ; name:
print type: NSR group
Hope that this information helped you.
Install the NetWorker software on the Windows 2008 Server Core
When installing the NetWorker software on the Windows 2008 Server Core, do not use autorun.exe as Windows Explorer is not available. Instead, from the command prompt use setup.exe located in the networkr directory:
Note: NetWorker Software is not supported yet on Windows 2008 R2 server core.
Note: In Windows 2008, the vcredist_x64.exe and vcredist_x86.exe files are required for the NetWorker software. Do not uninstall these files.
1.Run the following command to begin the NetWorker installation: c:\installdir\networkr>setup.exe
where installdir is the location of the NetWorker software.
2.Complete the installation wizard.
Uninstalling NetWorker software from the Windows 2008 Server Core, Perform one of the following:
1- Run the setup.exe file from the NetWorker installation disk and select to uninstall the software in maintenance mode.
2- Use the Windows Management Instrumentation command-line utility (wmic.exe)to access information about NetWorker and to uninstall it:
c:\>wmic product where name=”NetWorker” uninstall
Executing(\\ANGUR-W2K8-CORE\ROOT\CIMV2:Win32_Product.IdentifyingNumber=” {37AD0879-5B35-4A5C-9739-13302230CD8B}”,Name=”NetWorker”,Version=”7.6.0″ )->Uninstall()
Method execution successful.
Out Parameters:
instance of __PARAMETERS
{
ReturnValue = 0;
};
Note: For uninstalling NMC, name=”NetWorker Management Console Server” must be used.
Note: The Microsoft Knowledgebase article 290216 provides more information on the wmic.exe utility.
3- Use wmic.exe with msiexec to discover the installation product name for Networker, and the location of the msi package on the server to uninstall the software:
i. Run the wmic.exe utility:
c:\>wmic product NetWorker get /value
AssignmentType=1
Caption=NetWorker
Description=NetWorker
HelpLink=http://customernet.emc.com
HelpTelephone=1-877-534-2867
IdentifyingNumber={37AD0879-5B35-4A5C-9739-13302230CD8B}
InstallDate=20091001
InstallDate2=
InstallLocation=C:\Program Files\Legato\nsr\
InstallSource=C:\Build 102\nw76_win_x86\win_x86\networkr\
InstallState=5
Language=0
LocalPackage=C:\Windows\Installer\48b213b.msiEMC NetWorker Release 7.6 Installation Guide 144
Microsoft Windows Installation
Name=NetWorker
PackageCache=C:\Windows\Installer\48b213b.msi
PackageCode={76673145-1A42-4354-98F6-E56374F89004}
PackageName=NetWorker.msi
ProductID=none
RegCompany=
RegOwner=Windows User
SKUNumber=
Transforms=@1033.MST
URLInfoAbout=http://software.emc.com
URLUpdateInfo=http://softwareforms.emc.com/resources/downloads/ Vendor=EMC Corporation
Version=7.6.0
WordCount=0
ii. Use the msiexec utility to uninstall the product with the IdentifyingNumber:
c:\>wmic>msiexec /I {37AD0879-5B35-4A5C-9739-13302230CD8B}
iii. In NetWorker maintenance mode, use remove to uninstall the software.
Dell EMC NetWorker Troubleshooting 