Recently we had a case where an offsite backup in Altaro VM Backup was supposed to be written to Western Digital external drive. Before that is done we usually encrypt drive with Veracrypt (replaced TrueCrypt). After the drive was connected and mounted backups started failing.
Problem Description
This is our standard setup for most of our Clients. While Altaro VM Backup offers AES encryption by itself (Unlimited edition only) we usually encrypt offsite drives with Veracrypt for double protection, so that when the drive is lost data stays safe. While at the time of writing we already know that external drive was the cause of our problem it wasn't that obvious in the beginning.
First signs started to show up something is wrong when the backup failed and we got Altaro to send us daily email with failed backups.
This message doesn't say much except that there's a problem with Windows Shadow Copy:
Altaro Hyper-V Backup cannot create a Windows Shadow Copy at this time. The backup will be retried in a few minutes, since this problem is sometimes caused by the system being too busy to complete a Shadow Copy or other temporary issues. However, if the error persists, please contact support@altaro.com so that we can identify what is required to configure this host correctly and avoid Shadow Copy problems. (4103)
Since it's not uncommon for backup to fail with VSS errors we went ahead and verified Altaro configuration and retried manual backup of Hyper-V machines. Unfortunately, it still failed.
At this point, it was clear we need to investigate VSS. First steps involved checking if VSS writers are in good condition.
Microsoft Windows [Version 6.2.9200]
(c) 2012 Microsoft Corporation. All rights reserved.
C:\Users\Administrator>vssadmin list writers
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2012 Microsoft Corp.
Writer name: 'Task Scheduler Writer'
Writer Id: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}
Writer Instance Id: {1bddd48e-5052-49db-9b07-b96f96727e6b}
State: [1] Stable
Last error: No error
Writer name: 'VSS Metadata Store Writer'
Writer Id: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}
Writer Instance Id: {088e7a7d-09a8-4cc6-a609-ad90e75ddc93}
State: [1] Stable
Last error: No error
Writer name: 'Performance Counters Writer'
Writer Id: {0bada1de-01a9-4625-8278-69e735f39dd2}
Writer Instance Id: {f0086dda-9efc-47c5-8eb6-a944c3d09381}
State: [1] Stable
Last error: No error
Writer name: 'System Writer'
Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
Writer Instance Id: {0babd47f-6c5d-43c9-aa30-9196298ea324}
State: [7] Failed
Last error: Timed out
Writer name: 'Microsoft Hyper-V VSS Writer'
Writer Id: {66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}
Writer Instance Id: {57db7997-3c1c-4f1f-8e77-a8ae9e6c07f2}
State: [7] Failed
Last error: Timed out
Writer name: 'ASR Writer'
Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}
Writer Instance Id: {db7730e3-19ef-4238-b945-1301d9aec6cc}
State: [1] Stable
Last error: No error
Writer name: 'Shadow Copy Optimization Writer'
Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
Writer Instance Id: {0061d653-9efa-4009-9471-c2fef191442c}
State: [1] Stable
Last error: No error
Writer name: 'WMI Writer'
Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
Writer Instance Id: {7aa68501-6e3c-4161-bb69-efb2f80f0566}
State: [7] Failed
Last error: Timed out
Writer name: 'Registry Writer'
Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
Writer Instance Id: {7f5f2bb5-22e8-4f3b-98b8-e540acced9c6}
State: [1] Stable
Last error: No error
Writer name: 'COM+ REGDB Writer'
Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
Writer Instance Id: {231dea70-ba88-4c0c-8edb-3f8b96b13ede}
State: [1] Stable
Last error: No error
Few timeouts confirm the problem with VSS. Another check involved verification of volumes on which VSS is expected to work.
C:\Users\Administrator>vssadmin list volumes vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool (C) Copyright 2001-2012 Microsoft Corp. Error: The shadow copy provider had an unexpected error while trying to process the specified command.
This can also be verified by right clicking any drive in Explorer, going into Properties and under Shadow Copies there should be a nice table with all VSS-enabled drives. Unfortunately again it confirms there's a problem with VSS.
While rebooting machine used to fix such issues it's always worth to know where it comes from and what caused the problem, especially if it happens often.
Solution
So what was the problem? It seems the problem is related to VeraCrypt/Truecrypt way of mounting devices. If you encrypt the drive and mount it standard way VSS isn't able to provide shadow copy of that drive and blocks backup ability on Windows. After dismounting VeraCrypt volume vssadmin commands starts working fine.
C:\Users\Administrator>vssadmin list volumes
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2012 Microsoft Corp.
Volume path: \\?\Volume{ef372b16-96cc-11e2-93e7-806e6f6e6963}\
Volume name: \\?\Volume{ef372b16-96cc-11e2-93e7-806e6f6e6963}\
Volume path: D:\
Volume name: \\?\Volume{d615943a-ed3f-4c85-8298-d76c6b7ece96}\
Volume path: F:\
Volume name: \\?\Volume{080c4ea2-4ded-4d3c-a89e-4b1a91bc4438}\
Volume path: C:\
Volume name: \\?\Volume{ef372b17-96cc-11e2-93e7-806e6f6e6963}\
Quick verification under Windows Explorer also confirmed that VeraCrypt is at fault.
And backups starts working again… but still, we needed to have VeraCrypt working. To get VeraCrypt working properly and not block VSS based backups one has to make sure that mounted external drives are treated as a removable medium.
This makes sure Shadow Copies work as designed and backups can be made properly, including offsite backups to external drives.
