Recently, I received a frantic call from a client, the CEO of a global consultancy working in all parts of the world with some of the largest brands in every industry. They are an agency of record client, and also take advantage of the Google Apps for Work platform.
The first thing the client said to me was “I believe our Google Drive has a virus”.
I thought to myself, “No way, that can’t happen. It has to be just a few corrupted files.”
I took the call and began to investigate the issue with the help of a couple of team members. It appeared that a significant amount of the Google Drive had been corrupted in some way, but we weren’t sure how – yet.
As you can see in the screenshot here, the files have a binary string attached and a new prefix… “firstname.lastname@example.org”. The files were unusable and they wouldn’t even open in the browser. It was obvious that something bad was happening. While we opened a support ticket with Google Apps for Work, we also dove into the Google Apps Admin Console to investigate this issue a bit further on our own.
In the Admin console, there is a feature called “Reports”, which provides Administrators the ability to pull logs of all user actions that take place within the entire Google Apps organization. Administrators are able to filter by user, date range, file type and more, to narrow down a specific set of issues. We used this to search for any abnormalities that took place within the last 72 hours, that could hint at why these files were appearing corrupted.
We discovered that a specific user’s account had made a mass amount of edits (10’s of thousands), within about a 24 hour period, ending just hours before the first issues were reported. What it looked liked, was all of the files that this user had permissions to were renamed to this new filetype.
Our immediate recourse was to contact this user and notify them of the known issues (they had been one of the original complainants of a problem earlier that day). We also asked that he disconnect the Google Drive app from his PC so that it would no longer sync, and also shut down his browser access. We wanted to ensure that whatever was causing this was halted immediately, at this point we still weren’t sure if it was necessarily happening on that device.
Due to the urgent, and high-risk factors of this support case, we contacted our trusted technology partners at VineIT for additional support and resources. One of the key benefits of our agency, and how we remain so nimble and efficient, is our trusted core-competency partners. VineIT is the first, full managed service provider to offer their services to individuals as well as businesses. Their SimplicITy Suite brings together the #1 rated antivirus and malware protection software, a 24/7 USA based help desk, automated data backup and 24/7/365 monitoring of crucial process and security of hardware and networks.
“We were happy to have the assist with Social Faucet’s clients, and always are. It’s a great pleasure working with their team and they definitely have returned the favor with their world-class services.” – Nathan Ginter, Chief Technology Officer of VineIT
Their team was able to surmise that this user’s PC was affected by a known Cryptolocker virus, that attacked the /GoogleDrive/ file path on his local machine which then re-synced the locked files back to the shared Google Drive, using the native app function in the background. Without knowing, the user had just locked down 33.6GB of shared data across the entire team. At this point, there were two main questions:
- Is the data still there?
- Can we restore it to a previous version?
There were several different thoughts for action. Maybe if there was a device that hadn’t resynced the Google Drive in a few days we would have a backup… No such luck. Or maybe Google will be able to revert the file versions to restore the data as it was before the attack, nope – they don’t backup data like that. For a short time, it seemed as if we were striking out. Google Apps for Work Support was not able to assist, as they had no answers and the issue was quickly growing out of scope for them. They raised the ticket as a priority to some Engineers at Google that suggested we look into the “Drive API” and see if there was a way we could use a function for the restore. But this was not a supported feature, and we would have to rely on the Google Developers team to respond to questions/issues.
And then VineIT figured it out. They discovered that the data was in fact still there but had just been encrypted and renamed by this virus. They were able to successfully restore the last version available prior to the ill-fated sync by deleting the current version and resorting back to the original file within the Google Admin console. However, this presented a new set of problems. How do we manually restore 10’s of thousands of files, individually, within 24 days (the version history window for Google servers), without expending obscene amount resources towards the project.
Our solution sounded simple, but in reality it’s uncharted territory. Now that VineIT had identified the problem and provided use-cases to troubleshoot a resolve. It was up to our internal development team to make that happen. The plan was utilize the Google Drive API, to build an application that will retrieve revision feed of the affected files, fetch the content/data inside of those files and replace them with a new unencrypted version in the same location. Remember the part where I said we only had 24 days? Now we only have 22, and the time is ticking.
To resolve this issue in a way that was efficient as possible, for both cost and time, we took to our Engineers and Developers. After less than one day of research and several hours of planning, we determined we would be able to build an application that utilizes the Drive API to rollback file versions in mass, based on any unique ID or name. After returning the files to their uncorrupted version, we would have to run a script to also rename every file – to remove the “…@mail.bg” from the filename. This process needed to be repeated for each affected user account in the Google Apps Organization (3 in total).
After nine days of development and testing, we were able to deploy our application, written in Python, a widely used general-purpose, high-level programming language. The app was designed to restore 1,000 files at a time, and then provide a .csv print-out of the result. As an Agency of Record, accountability and client reporting is a priority. Our application successfully restored 16,314 files, which were then renamed to their original filename using a separate script.
Every day our role as an Agency of Record is changing and expanding. Remaining nimble and efficient is the #1 priority of our leadership during our rapid growth. We look forward to being able to provide this level of service and attention to every client, no matter how big we are or how complicated the situation.
In the coming weeks we will be releasing this app to the public, making this solution to a potentially disastrous situation available for the standard Google Apps user that is not a Programmer or have a team behind them. Complete this form, here, to be notified when the app becomes available.
Are you using the cloud today? Have you implemented a cloud-to-cloud backup solution for your cloud files? Contact Social Faucet today to get started.
Elliott Ramsdell is a serial entrepreneur, with a passion for technology and social media. His experience in start-ups, investor relations, franchise, licensing, and business development allows him to maintain close relationships with partners and clients of Social Faucet. Elliott’s core competencies are in Brand Design, Marketing and Development. In his free time, he enjoys rearranging alphabetical discrepancies in local record stores, coding, and designing new brands or breathing new life into old ones.