20
November

NetWorker Exchange failure – The group or resource is not in the correct state to perform the requested operation.

As with any windows system backup, we sadly have to deal with VSS. VSS is the bane of any backup administrators existence. Add in Exchange, and you have another layer of services that require the correct orchestration to ensure protection.

In today’s post, we revisit our old friend VSS and provide some additional insight into NetWorker and the associated services that help coordinate backups.

I had some new Exchange systems to add to backup rotation. These had Exchange 2013 on W2k12.

We started to see the following errors:

Networker error messages:
APPLICATIONS:\Microsoft Exchange 2013: Backup of [APPLICATIONS:\Microsoft Exchange 2013] failed
rsnap_vss_save:NMM .. error caught calling RM to commit the replica.
49931:nsrsnap_vss_save:RM .. 027114 ERROR:Exchange Replication Service is stable. The error is VSS_E_WRITERERROR_RETRYABLE. The code is: 0x800423f3. Check the application event log for more information.
49931:nsrsnap_vss_save:RM .. 027114 ERROR:Exchange Replication Service is stable. The error is VSS_E_WRITERERROR_RETRYABLE. The code is: 0x800423f3. Check the application event log for more information.
49931:nsrsnap_vss_save:RM .. 027114 ERROR:Exchange Replication Service is stable. The error is VSS_E_WRITERERROR_RETRYABLE. The code is: 0x800423f3. Check the application event log for more information.
49931:nsrsnap_vss_save:RM .. 024168 ERROR:The VSS shadow copy for Exchange Replication Service was not successful because of an error that occurred earlier.
83394:nsrsnap_vss_save:NMM .. cluster failover or application role change after a replica created may have caused snapshot creation failure, try to restart the backup.
Microsoft DiskPart version 6.3.9600
Copyright (C) 1999-2013 Microsoft Corporation.
On computer: exchange-node
Automatic mounting of new volumes enabled.
37959:nsrsnap_vss_save:The group or resource is not in the correct state to perform the requested operation.
.
63335:nsrsnap_vss_save:NMM backup failed to complete successfully.
Internal error.
102333 1447959921 3 0 0 15084 14984 0 server nsrsnap_vss_save NSR error 21 Exiting with failure. 0

Corrective actions taken:
Rebooted Exchange servers
Removed networker client , NMM,

And some of the following from the event viewer:
[RpcHttp] Marking ClientAccess 2010 server server(https://server/rpc/rpcproxy.dll) as unhealthy due to exception: System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. —> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. —> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
**************************************************************************************************************************
[Autodiscover] Marking ClientAccess 2010 server server (https://server) as unhealthy due to exception: System.Net.WebException: The operation has timed out
at System.Net.HttpWebRequest.GetResponse()
at Microsoft.Exchange.HttpProxy.ProtocolPingStrategyBase.Ping(Uri url)
****************************************************************************************************************************

The description for Event ID 5068 from source Replication API Service cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event:

2015 11 19 00:00:01
exchange-node

The specified resource type cannot be found in the image file
***********************************************************************************************************************

********************************************************************************************
PS: (CPSImportService::GetOperationStatus) ERR: Error for operation 1
*********************************************************************************************
NMM .. Error during import of snapshot: PS .. Error completing snapshot import
**********************************************************************************************.
NMM .. Registration of snapshot set failed — PS .. Error completing snapshot import
****************************************************************************************************
Microsoft Exchange VSS Writer backup failed. No log files were truncated. Instance d93e725e-486d-440f-8397-93061460ee02. Database 8bee9b3c-8fff-405e-a42d-b739d05caadd.
********************************************************************************************************************
The Microsoft Exchange Replication service VSS Writer (Instance d93e725e-486d-440f-8397-93061460ee02) failed with error FFFFFFFC when processing the backup completion event.
********************************************************************************************************************
An internal transport certificate will expire soon. Thumbprint:EE063967EB62D17668AD392286EFF96622F84BFC, hours remaining: 1594
******************************************************************************************************************

What we did?

We rebooted the system, attempted to restart some of the associated VSS services. No Joy. Opened a ticket with EMC, Anthony gave me a call back. I had worked with him before, so I was glad to hear from him.

On your Exchange Server run the following commands and give their output:

 

  1. Vssadmin list writers
  2. Vssadmin list shadows
  3. If the above command is listing any shadow copies, perform following steps:
  4. type diskshadow
  5. type list shadows all
  6. type delete shadows all
  7. Stop all networker services on client, verify all services are stopped aside from nsrpm.
  8. While NetWorker services are stopped restart Replication Manager RMAgentPS and make sure the other Replication Manager service is stopped as well.
  9. Restart NetWorker services
  10. In cmd prompt type tasklist | findstr nsr

We retried the backup and it worked.

A little insight into some of these services and the roles they play. We are familiar with nsrexecd and Powersnap. We also had to restart the RMAgentPS service. This service executes operations for Replication Manager Client for RMAgentPS. In addition we had to stop the Replication Manager Exchange Interface. This service executes Exchange commands for replication manager. In the end there were some stale shadows and we needed to restart the required service in the correct order. Of course also remember to make sure the Exchange VSS writer is present.

 

 

 

no comments

11
April

Configuring NMM Exchange Backups

stephen-colbert

First I know what you are all wondering, Dan how do you feel about Stephen Colbert being named as David Letterman’s replacement in 2015? I find Stephen Colbert both charming and handsome and look forward to meeting the real Colbert when he debuts. It’s a good thing Mr. Colbert can relay on his many talents to help fulfill a role as a lat night talk show host. Without those skills he may have ended up a lowly Data recovery specialist in Calgary faced with an NMM configuration.

That my friends, is what we call a segue in the biz.

 

 

Recently I rejoined the professional services team at my company. I had been embedded for sometime, but lets not talk about that. Suffice to say I’m excited to return to PS as a trusted adviser on all things NetWorker. Those words are a complete fabrication of reality and were proven when I attempted to configure NMM backups for my clients Exchange systems.

Given that this was my first project upon returning to PS I was eager to put a “W” up on the board. I had thought I was prepared having reviewed the NMM documentation. I found it slightly convoluted and confusing in some parts, but this is not an unusual circumstance for my brain. Perhaps I was delusional in assuming that the documentation contained all the knowledge needed to be successful? Yes, yes I was.

I don’t have the time to re-write the NMM documentation here. What I can do is fill you in on some glaring omissions form the documentation that EMC support was able to inform me of.

After adding the client check the client resources.

Not sure how critical this was but EMC support advised that a pool should not be specified.

4-11-2014 12-43-22 PM

 

 

Ensure the DAG entry is correct

4-11-2014 12-43-35 PM

 

 

 

 

 

Ensure all DNS aliases are entered

4-11-2014 12-43-51 PM

The remote access are should contain entries for all the exchange nodes.

4-11-2014 12-44-09 PM

 

 

The administration field should contain the following. An entry for the NMMBackupUser, this user should have been created with the required permissions and a mailbox assigned. There should also be entries for the exchange nodes and the Exchange DAG name.

4-11-2014 12-55-31 PM

 

 

Again, not sure how important, but the EMC support zeroed out these entries on the group properties

 

4-11-2014 12-44-49 PM

 

 

The snapshot policy and pool should be identified under the group properties.

4-11-2014 12-44-32 PM

 

Finally, ensure you have the following entries in administration  under setup in the NetWorker Server properties.

4-11-2014 12-55-31 PM

 

 

EMC documentation is pretty good, but I don’t know how these very critical items could be missed. I searched the NMM documentation portfolio for the following items.

I hope this post helps other users, VAR’s and EMC staff in any future NMM implementations.  All said, it seems to work well when you get it going.

no comments