Case Study: Server Storage and Data Management for Video Editors

Data management is an ongoing issue for most businesses, especially those working with video. Our cache of digital data grows endlessly every day. Emails, files, and multimedia continue to be created, sent and received daily in an endless tide of 0s and 1s. You’re working with HD, UHD, 4K and 3D and every minute of footage is getting exponentially bigger. So, what to do about it?

Video production companies often suffer the most from the challenges of data management. The latest 4K and Ultra-HD formats result in colossal raw footage files which must be carefully protected as quickly as possible after they’re shot.

The producers’ clients often expect the edit to be retained for years, in case they wish to make minor alterations in the future. And all the while the production company has to keep adding servers and disks to keep up. Even cloud storage is a challenge due to the upload time and the cost of storage.

The Starting Point

Like many similar clients, our video production client’s starting point was a large Synology NAS RAID storage server with a secondary unit held offsite, to act as a backup / mirror of the first. Data backups were meant to happen over the internet, overnight.

The primary server was nearing capacity, but more pertinently the amount of data being generated and stored every day was too large for the offsite backup task to ever complete in the time allocated overnight. As such, we were faced with an imminent data protection issue that needed to be resolved before anything else could happen.

Beyond that, it was clear that we couldn’t hold every project ever completed indefinitely on the same server. There was just too much stuff. The RAID volume could be expanded with more disks, but it felt like putting all the eggs in one basket. The majority of key intellectual property of the business was on one server, the backups could not complete, and any rebuild would have taken weeks to complete in the event of a disk failure – leaving the data at risk from drive failure as well, despite being on a RAID.

Solution: Part 1 – Streamlining

Immediately it became apparent that we needed to reducing the burden of the backup process.

Most businesses require offsite backup to protect from fire, flood, theft or some other localised disaster. But too much data cannot be sent offsite reliably in the time available, before new data starts getting created the next day.

So, we started by identifying which data was being changed regularly and which data was largely static. It quickly became apparent, as is often the case, that large swathes of data on the server were finished projects. The offsite backup was having to scan these projects for changes every day, even though they were never changed. So the backup task could not ‘get going’ until this verification check had completed, delaying the backup process with already limited time at our disposal.

As such, we began by spitting the finished projects from the active ones. Once we were comfortable that all finished projects were stored on the offsite server, they were locked to read-only, preventing changes and discrepancies occurring between the live and offsite (backup) servers. This reduced the time the backup task needed to scan for changes so it could get on with backing up the data.

Solution: Part 2 – Offsite Archiving

We then deployed a BackBlaze B2 account to serve as a new, permanent offsite home for the finished projects.

For a small monthly fee of just $5 per Terabyte per month. This saved the client from having to maintain vast amounts of server and storage hardware in the long term, by delegating the responsibility for protecting the data to BackBlaze.

To do this in-house would cost vastly more money, by the time you account for replacement hardware in the long-term, as well as proactive cycling of hard disks to reduce the risk of failure. BackBlaze have far more redundancy than most businesses can justifiably put in place though maintaining their own hardware.

Solution: Part 3 – Intelligent Storage

The final piece of the puzzle was how to protect the most valuable of content, the irreplaceable raw footage (re-doing a shoot due to lost footage is never going to be good news).

However, we could not simply disregard the ‘working files’ in Final Cut Pro X where the editors had been spending countless working hours in putting together the edit.

It occurred to us that raw footage never changes once it’s been produced. So we split our backup tasks by splitting our data on the server; we advised the client to stop importing raw footage into the Final Cut project, and instead reference the raw footage in a separate shared folder.

In this way, we could have two backup tasks: the raw footage, which only ever needs to be sent offsite once, and the ‘working files’ which need to be scanned and backed up offsite every day.

By doing this, we made the backup routine vastly more efficient and able to once again complete in the time available between close of business and the next working day.