Data management is an ongoing issue for most businesses, especially those working with video. Our cache of digital data grows endlessly every day. Emails, files, and multimedia continue to be created, sent and received daily in an endless tide of 0s and 1s. You’re working with HD, UHD, 4K and 3D and every minute of footage is getting exponentially bigger. So, what to do about it?
Video production companies often suffer the most from the challenges of data management. The latest 4K and Ultra-HD formats result in colossal raw footage files which must be carefully protected as quickly as possible after they’re shot.
The producers’ clients often expect the edit to be retained for years, in case they wish to make minor alterations in the future. And all the while the production company has to keep adding servers and disks to keep up. Even cloud storage is a challenge due to the upload time and the cost of storage.
We would love to hear from you about your changing data storage needs and collaboration challenges. No obligation, no hard sell. Just good honest and friendly advice.
You can also read on for a case study about how we helped one client with their data growth and management.
Like many similar clients, our video production client’s starting point was a large Synology NAS RAID storage server with a secondary unit held offsite, to act as a backup / mirror of the first. Data backups were meant to happen over the internet, overnight.
The primary server was nearing capacity, but more pertinently the amount of data being generated and stored every day was too large for the offsite backup task to ever complete in the time allocated overnight. As such, we were faced with an imminent data protection issue that needed to be resolved before anything else could happen.
Beyond that, it was clear that we couldn’t hold every project ever completed indefinitely on the same server. There was just too much stuff. The RAID volume could be expanded with more disks, but it felt like putting all the eggs in one basket. The majority of key intellectual property of the business was on one server, the backups could not complete, and any rebuild would have taken weeks to complete in the event of a disk failure – leaving the data at risk from drive failure as well, despite being on a RAID.
Immediately it became apparent that we needed to reducing the burden of the backup process.
Most businesses require offsite backup to protect from fire, flood, theft or some other localised disaster. But too much data cannot be sent offsite reliably in the time available, before new data starts getting created the next day.
So, we started by identifying which data was being changed regularly and which data was largely static. It quickly became apparent, as is often the case, that large swathes of data on the server were finished projects. The offsite backup was having to scan these projects for changes every day, even though they were never changed. So the backup task could not ‘get going’ until this verification check had completed, delaying the backup process with already limited time at our disposal.
As such, we began by spitting the finished projects from the active ones. Once we were comfortable that all finished projects were stored on the offsite server, they were locked to read-only, preventing changes and discrepancies occurring between the live and offsite (backup) servers. This reduced the time the backup task needed to scan for changes so it could get on with backing up the data.
We then deployed a BackBlaze B2 account to serve as a new, permanent offsite home for the finished projects.
For a small monthly fee of just $5 per Terabyte per month. This saved the client from having to maintain vast amounts of server and storage hardware in the long term, by delegating the responsibility for protecting the data to BackBlaze.
To do this in-house would cost vastly more money, by the time you account for replacement hardware in the long-term, as well as proactive cycling of hard disks to reduce the risk of failure. BackBlaze have far more redundancy than most businesses can justifiably put in place though maintaining their own hardware.
The final piece of the puzzle was how to protect the most valuable of content, the irreplaceable raw footage (re-doing a shoot due to lost footage is never going to be good news).
However, we could not simply disregard the ‘working files’ in Final Cut Pro X where the editors had been spending countless working hours in putting together the edit.
It occurred to us that raw footage never changes once it’s been produced. So we split our backup tasks by splitting our data on the server; we advised the client to stop importing raw footage into the Final Cut project, and instead reference the raw footage in a separate shared folder.
In this way, we could have two backup tasks: the raw footage, which only ever needs to be sent offsite once, and the ‘working files’ which need to be scanned and backed up offsite every day.
By doing this, we made the backup routine vastly more efficient and able to once again complete in the time available between close of business and the next working day.
We would love to hear from you about your changing data storage needs and collaboration challenges. There’s no obligation, no hard sell. Tell us what you’re struggling with and we will gladly offer our advice and guidance.
When we solve a support ticket, clients are given the choice of leaving good or bad feedback along with an optional comment. We post the 10 most recent comments here automatically and in real-time. You can view even more on our reviews page.
|Sep 24th||Kerry L||Super fast service, issue resolved in moments by Lochie. Excellent as always, thank you|
|Sep 14th||Sue H||All good response received :-)|
|Sep 14th||Kerry||Prompt and efficient service and problem resolved, fantastic service as always, thanks Lochie. I would recommend everyone to try Purple Computing, we'd never look elsewhere!|
|Sep 8th||Lynn H||Really rapid response as always. Excellent service. Hi Martyn & Lochie I have spoken to Quickbooks. I was wrong and we can use the online system so does not need to be on the server. Sorry to have wasted your time but I do appreciate your support. Not sure how long it will take to get up and running but at least it looks like Tas Books will be a thing of the past in the near future. I will keep you posted when I have transferred all the financial info and Tas Books is NR. Probably about 4-6 weeks , hopefully. Many thanks again. Kind regards Lynn Hemmings MEP Accounts Dept|
|Sep 8th||Lynn H||Really rapid response as always. Excellent service|
|Sep 8th||Simon D||You guys have always provided a remarkably responsive service. I'd like to thank you for all the help and professional support you've given to the business.|
|Sep 1st||Sally T||Great. Thank you again for your support - apologies for being such a luddite!! Thank you Sally|
|Aug 26th||Chris S||Very fast and efficient response - thank you!|
|Aug 23rd||Amanda H||Very helpful general advice including sorting out the email problem.|
|Aug 12th||Caroline W||Very quick response and all sorted. Thank you|