Arcserve CEO: StorageCraft Backup Data Loss ’Not Acceptable’

’In terms of magnitude, it’s the only thing I’m thinking about waking up and going to bed every night. It’s orders of magnitude more important than anything else that I have going on in terms of operating this business,’ says Arcserve CEO Brannon Lacey of the move by his company to bounce back from a human error that caused customers of some StorageCraft MSPs to lose their backup data.

A Data Protection Company’s Nightmare

Data protection software and appliance technology developer Arcserve is in the midst of a long-term recovery from a human error which resulted in an as-of-yet unknown number of end-user clients from the company’s StorageCraft business losing their backup data. The error, which happened when the company was moving the StorageCraft backend operations from an internal data center to the Google Cloud Platform as a way to add resilience to the system, is now practically the only thing Arcserve CEO Brannon Lacey thinks about, Lacey told CRN.

For Lacey, the loss of some backup data has become the top issue of his so-far five-month tenure as CEO of Arcserve. “I guess you could say the honeymoon might be over,” he told CRN.

The data loss primarily affected clients of MSPs working with StorageCraft, which early last year merged with Arcserve. While the data loss does not impact primary data, it does leave clients’ primary data at risk should it be lost or attacked because it now may not be backed up. Arcserve is in the process of remediating the issue with its StorageCraft MSPs, including looking at how to reseed the backups and ensure previously protected data is protected again.

To help partners impacted by the backup loss, Arcserve is proactively providing partners with specific information on how to re-seed to its Google Cloud Platform (GCP). The company is also offering options to restore backups, including a no-cost expedited direct delivery of seed drives for machines over 3 TBs of capacity, Arcserve-initiated re-seeding over the wire for partners with ShadowProtect machines, and re-seeding over the wire for partners with ShadowXafe and Solo machines. Arcserve is also recommending that partners create an additional offsite backup.

For details about how the backup data was lost and on the impact of the loss on Arcserve and its partners, read on.

What is the situation with the data loss?

We’re a partner-first organization. We’ve combined two entities [Arcserve and StorageCraft], and both entities sell through partner channels. We’ve got a small direct business, but the primary go-to-market motion for us is through partners. And on the MSP side, we put our partners in a difficult situation, as you’ve come to find out. The first message I want to give you is that we are in a period of accelerated remediation. We’re knee-deep in an RCA (root cause analysis) process. And I don’t have all the answers yet because that process isn’t done, and I don’t have full data. So I’m not going to give you anything that could be incorrect ... but I’ll share what I can share. And I’ll start by outlining what’s happened here.

We are a business in transformation. I’m a CEO that comes from a heritage of managed services at Rackspace and other service providers. And I’ve come from a heritage of providing world-class service on other people’s technology and building my own technology. And I’ve had success doing that in the past. And that’s why Marlin Equity Partners, our private equity owners, brought me in. And it’s unfortunate we’ve hit this issue right in the middle of what is a very exciting transformation for us. What happened is, as part of a planned move of our StorageCraft cloud services to GCP (Google Cloud Platform), which we’re doing intentionally for redundancy and reliability purposes in our system, in that process, we had a human error that during the migration caused servers containing critical metadata to be decommissioned prematurely. So it’s not a technology issue, but an operational issue, a people issue.

So what happened?

[As Arcserve is] moving massive amounts of data from our own data centers into GCP in this planned migration, this type of stuff can happen. And unfortunately it did. And it’s very unfortunate. It’s certainly not the first time a company has lost data. Won’t be the last time. But we have our arms around it. And we’re focused on RCA and remediation at this point. When we look at the affected partners, at this point I don’t have the exact number, but I can tell you that it’s limited mainly to North American partners.

And so what we’re doing to remediate here is, we’ve gone and identified several recovery scenarios (see above) depending on the type of machines that partners are running. [This has] all have been communicated to our affected partners. So we’ve gone to all affected partners with the scenarios that they can follow based on the use cases to start remediating. And we’re in the process of reseeding machines and communicating continuously daily to all the partners. ... The majority of our partners are in the process of reseeding as of today.

It goes without saying that we’re doing everything that should be done. I’ve been through these types of crisis at much larger scale with prior organizations. And we know how to handle a crisis of this magnitude. And make no mistake about it, it’s a big magnitude. But we have our arms around it, and we’re doing everything we can to get our partners out of this situation. It’s a 24x7 job for us remediating the data situation as quickly as possible. And it’s clear we put some partners in a difficult situation. And that personally really, really affects me as I’ve spent the first five months of my time [at Arcserve] equally inside our business and outside of our business ... meeting one-on-one with partners and our advisory councils. And I’ve gotten to know a large swath of our partners, particularly here in North America, and it’s not acceptable for us to put them in this situation.

I think they, to a man and woman, as I sit on the front lines, communicating with them daily since the start of this crisis, all of them appreciate our openness around the issue and our commitment to correct it. And I feel like even though the situation is still to be fully remediated, we’re building goodwill for the future with these partners. I’ll close by saying that we completely know at Arcserve that it’s not acceptable, and we’re moving as quickly as possible so we can get back to building Arcserve into the world-class service organization that we’re building it to be with innovation and reliable technology that in the past we’ve been known for. And I believe that the steps we’re taking in parallel to remediation, which are focused on what the operational failing was and how we never get into this situation again, means that we’re going to exit this issue stronger and better.

You said you're not sure how many partners were impacted. How many customers were impacted?

I don’t know that exact number either. I want to reiterate this is coming from the StorageCraft technology side. So when I say I don’t know how many customers, what I mean, and I just want to be unambiguously clear, is that there are certainly customers affected, customers of our partners, so therefore our customers. And I can’t quantify that number for you. But we also have direct customers on the historical Arcserve side, and those customers are unaffected because this is a StorageCraft operational issue.

Why the move to Google?

I think it’s the same reason a lot of organizations are adopting public cloud. You’re looking for someone to manage the infrastructure stack or more of the infrastructure stack so that you can focus on your primary business. We will always be in the data center business. We’re not moving out of the data center business. But what we’re doing is building redundancy and resiliency into our data center operations and infrastructure operations by bringing GCP into the fold as an infrastructure offering for many of our products, and gain additional flexibility and scale.

Do you know how much customer data was impacted?

No, not yet.

Was the data actually lost and totally unrecoverable? Or is the data going to be able to be recovered?

We’re still working through that. I don’t have a definitive answer for you yet. So I don’t want to mislead unintentionally by giving you an answer at this stage. It’s still too early for us to tell.

What kind of compensation is Arcserve providing partners and/or customers for lost backup data?

Our primary focus is to remediate as quickly as possible, and we are making rapid headway. We are holding impacted partners’ hand through the entire process to minimize any impact. We will do right by our partners and, of course, they will not be invoiced throughout this process.

Because this hit customers’ backup data, customers should be able to continue operations, albeit with risk if they lose primary data that is now not backed up. Is Arcserve providing an alternative way for them to back up their data going forward as this problem is being resolved?

Yes, there are multiple scenarios that we’re providing our partners for recovery and reseeding. But we also have a continual amount of unaffected partners uploading new data every single day.

Okay, but that's uploading data for customers that previously were not impacted by this, correct?

Yes. Unaffected and unimpacted partners are operating in the same way that they have before the incident.

But for those customers whose backup data is no longer available, are you at least providing a way for them to backup new data while the situation's being taken care of? In other words, can they still do backups now, even though their data might be missing?

Yes. So the short answer is yes. And you’ll see that as part of the scenario [we have laid out].

What has the impact on Arcserve sales been? Are you pulling back on sales, or going forward? Are partners pulling back?

We haven’t seen an impact yet, but it’s been in play for only a short period of time. I think time will tell. We’re hopeful that the high-touch approach, the transparency and optionality that we’re providing partners, frankly being partner-like as much as you can in a scenario like this, will carry us through this and help generate goodwill with partners so that we come out on the other side with trust still being in place and an ability to transact and a willingness to transact on the partner side.

Do you expect to lose any customers because of this?

Again, the way that our business is split, we have both direct customers from the Arcserve side, and customers on the MSP side are partners. Our Arcserve customers are unaffected by this. It’s only a subset of our partners on the MSP side, and I am anticipating a lot of different reactions, and we’re seeing a lot of different reactions today. But again, the way that we’re handling this, I think, has been positively received by partners based both anecdotally for me and my conversations, and I’m having a lot of them every day, and in the feedback that my teams are surfacing as we engage in a high-touch way. But I’ve been in the technology game for 20-plus years. We’re in the reliability and trust business, and with these types of events you never leave them without some sort of impact with partners because our partner community is in the business of providing a service to their own customers and a lot of what they do depends on the relationships they have with their customers. So, do we anticipate some impact to our P&L (profit and loss)? Of course, I think that goes without saying. But I feel like we’re controlling everything that we can control in order to limit the impact.

We expect to avoid this type of situation in the future in, in no small part due to the fact that we’ll be moving and migrating large portions of our data into GCP. And that’s one of the drivers behind the strategy of moving to Google as an infrastructure provider.

Have your competitors started making hay from the situation?

I would expect them to.

How do you respond when a competitor points at this backup data loss and says, 'Maybe you should try us instead?'

My philosophy with partners is to think about what I would want from a vendor like an Arcserve. And I would look at the sum total of the relationship. Many of our MSP partners have been with us for double-digit years. And we’ve generated a lot of goodwill with them over the years with world-class customer service and technology. And I’d ask them to take the long view and consider the cost of migrating over to another competitor’s platform and realize that when dealing with technology, these types of things will happen, and will happen on other competitors’ platforms. This is something that isn’t relegated to Arcserve. And I would respect their decision to make a decision that’s best for their company.

And I would say that we are intentionally moving into a different infrastructure in the future because we want to avoid this type of situation. And then I’m confident in the team that I’m putting in place, the operational maturity that we’re building in this business, and the efficacy of our product set that in the future. We will look back on this and say, ’Yeah, this was an inflection point for the relationship, and Arcserve came out of it a better company.’