EMC Avamar Error 10007 is an interesting error with a little history behind it. After opening up a support case with EMC, they indicated that error 10007, vmx file is suspiciously small, is a legacy error message from the VMware vCenter 4.x version of vCenter. From what the support technician said, is that the VMX file would be backed up through web services. This all changed in vCenter 5.x. Our error 10007, was resolved by restarting the MCS services on the Avamar grid. What happened is that the cached credentials and session state became stale/invalid. The EMC Avamar grid would try to reuse that session information that was now invalid thus causing the Avmar 10007 error.
This may seem like a strange fix, but restarting MCS services on the Avamar grid resolved our issue. This resolution was the recommended fix from EMC support. The article, Backup job fails for all virtual machines after vSphere Data Protection 5.1 deployment (2038597), is published by VMware regarding a similar error but this was not the particular error which we were encountering.
The EMC Avamar Error message would look like:
2013-07-08 08:31:41 avvcbimage Error : vmx file is suspiciously small (under 30 bytes), please examine the log on the Avamar Administrator for root cause analysis (Log #2) 2013-07-08 08:31:41 avvcbimage Error : Backup of VM metadata failed. (Log #2) 2013-07-08 08:31:45 avvcbimage Error : Avtar exited with 'code 163: externally cancelled' (Log #2)
The EMC Avamar proxy log would display the following messages:
avmproxy1:/usr/local/avamarclient/var-proxy-1 # egrep -i "sdk|reused" Default_Domain-1372926600107-e55a99675c5f5a260028ec81d88bf620d9678824-1016-vmimagel.log 2013-07-04 08:32:32 avvcbimage Info : Login(https://vcenter.domain.com:443/sdk) problem with reused sessionID='52f3ba9e-ff44-fa6d-ed6f-61b44fe35ec5' contacting data center 'Cleveland'. 2013-07-04 08:32:32 avvcbimage Warning : Problem logging into URL 'https://vcenter.domain.com:443/sdk' with session cookie. 2013-07-04 08:32:32 avvcbimage Info : Logging into URL 'https://vcenter.domain.com:443/sdk' with user 'DOMAINavamar' credentials. 2013-07-04 08:32:32 avvcbimage Info : Login(https://vcenter.domain.com:443/sdk) problem with reused sessionID='524408a5-443b-1d22-9878-6f5fe4de2816' contacting data center 'Cleveland'.
Resolution: Restart MCS Services on the EMC Avamar Grid.