Sharing Information

During any kind of production, a lot of data are created. A sizeable amount of that data is just intermediary—the result of research and development, placeholders for work that is completed later, or just iterative versions on ideas, by-products of the creative process. But the rest of it can represent valuable data, pieces that will ultimately become part of a completed production. There’s likely to be a lot of it, it’s likely to come from many different places, and there will be a lot of people who will need parts of it. Completed storyboards might be needed by the director or cinematographer, final edited footage would be needed by a colorist, a completed music video would be needed by a broadcaster.

As such, at a certain point the creative process gives way to logistics. With everything in the Cloud, the process can be simplified. People working on stereoscopic conversion of a feature film, for example, might need a lot of information about how each finished shot in the film was composed, which lens was used, and so on. Traditionally this type of information exchange would happen through complicated data dumps or, even worse, constant correspondence between different departments. With the Cloud, the stereo team could just be given access to the tracking systems or databases used by the respective teams to store this information in the first place. With some enforced consistency across different projects as to how this information is stored and displayed, the stereo team wouldn’t even necessarily need to learn to navigate a new system each time.

Likewise, with people who need access to files and media rather than meta-data, a similar approach can be taken with Cloud Storage. Visual effects facilities, for example, might need access to a lot of media generated by the production crew during a shoot, from principal camera footage to reference images, LIDAR scans, or “witness” camera footage. For productions that already store this information in the Cloud, granting the facilities access to specific files and folders should be easier and faster than the alternative of making copies of the data from a server and then sending it to the vendor by putting it on an external drive and sending it via courier or uploading it via FTP or Aspera.

Figure 13.1 Data Stored in the Cloud

There are other situations where distributing media via the Cloud is an appealing prospect. In 2015, a major leak of DVD “screener” movies, intended for viewing by people voting on Academy Awards, meant that many movies ended up in the hands of pirates, including one that had not yet seen a general theatrical release. This re-ignited the debate to use digital screeners instead of physical DVDs, as they are thought to be more secure and less susceptible to being leaked in this way (as a bonus, they are also much cheaper to produce).

The technology to do this already exists and is available through the Cloud. GoScreening (goscreening.com) provides a way to send DVDs to the service, which will then encode and make them available for viewing in the Cloud. Prime Focus Technologies’ SecureScreener (primefocustechnologies.com/securescreener) takes this further by providing more options and security measures. That said, many creators feel this approach is anathema to their intended vision, as watching via a browser or mobile device is an inferior experience to watching on a big screen.

Box 13.1 The Nuts & Bolts of Cloud Distribution

Part 1—Internet Transport Layer Protocols

The Transport layer of the Internet protocol suite (see chapter 2), defines how different computers communicate with each other. Within this layer, several different approaches (or “protocols”) are defined, the most widely-used being the Transmission Control Protocol (TCP), followed by the User Datagram Protocol (UDP). Each of these is fundamentally different, and as such suit different purposes, but it is useful to know the characteristics of each.

TCP operates on the principle that the most important factor is reliability—that when data are sent from one device to another, that data get there intact. It works by making a connection (secure or otherwise) between the two devices and then allowing data to be sent via that connection, in much the same way as making a phone call. TCP provides both flow control (ensuring you don’t send too much data in one go) and error-checking (resending data that did not get through). However, it can be relatively slow to send data, especially over large distances, where latency (the delay between data being sent by one device and receiving an acknowledgment from the other) can get very large, like having the long pauses on a long-distance phone call while you wait for the other person to reply. As a result, it’s practically useless for anything that needs real-time data transfer.

UDP on the other hand works by blindly sending out data without needing an established connection and assuming that the data will just reach the destination. There are no checks to make sure this happens (and as a result, no delays). If TCP is two people making a phone call, then UDP is two people standing at opposite ends of a busy street with megaphones. This results in faster data transmission than is possible with TCP but without any controls to verify the data are intact or to limit the transmission to the available bandwidth. In practice this means you can stream live data using UDP (the most common example being Voice over IP), but occasionally there’ll be “packet loss” (such as the person you’re talking to freezing for a split second or other video artifacts). Though many of the limitations of UDP can actually be resolved by designing the applications sending and receiving UDP-transmitted data to work around them, there are other limitations which are harder to solve, such as the fact that many network firewalls block UDP transmissions, and typically must be specifically configured to work with a particular application.

There have been some attempts to address the shortcomings of TCP compared to UDP. Most notably is the concept of buffering, whereby the application delays presenting data received to the user until enough of it is queued up so that any future intermittent transmission delays may be compensated for by grabbing data out of the buffer, giving the user an effectively seamless stream of data. There are several downsides to this, the most immediate being the initial delay between a user requesting data (like clicking play on a video) and it actually being presented (the video starts to play), and there might be situations where there’s enough of a drop in the flow of data that the buffer gets completely used up and has to refill before continuing (the video stops part-way, and there’s a delay before it resumes); but on the whole this provides a good work-around (at least for anything that doesn’t need to be real-time, like a web application that responds directly to a user’s input).

An alternative to either of these protocols (and a popular one in the film and television industry) is IBM’s Aspera “FASP”, a proprietary protocol which is functionally similar to TCP but with different mechanisms for error-correction and flow control. The result is the ability to send data reliably but much faster than is possible over equivalent TCP connections. In practice, while transfers using Aspera technology tend to be both fast and reliable, there’s a tendency for them to monopolise all of the available network bandwidth, meaning that other network traffic (such as opening web pages or receiving emails) will slow dramatically. There’s also a substantial cost implication to setting up an Aspera connection, as it requires the purchase and administration of proprietary client and server software.

Other iterations of UDP attempt to eliminate its shortcomings. Google’s Quick UDP Internet Connections (QUIC) is designed to bring the benefits of UDP’s faster data transfer to applications that currently use TCP, though in practice the only significant benefits are seen by users with slow connections. It’s currently in an experimental form and is yet to see widespread adoption outside of Google.

Broadcasting in the Cloud

There’s a surprising lack of commercially-available options for people who want to be able to broadcast a finished product to the masses, considering how well the Cloud is suited for doing just that. If you then want to monetise it, well there’s even fewer options.

To take a finished digital video and make it publically viewable, there are a few Cloud-based services, most significantly YouTube (youtube.com) or Vimeo (vimeo.com). YouTube has the benefit of a much larger audience, but the picture quality is somewhat lacking compared to Vimeo. With both services, audiences can comment on the video (if the option is enabled), share the link, or even “embed” (post a playable copy) on other sites.

If you want to actually make money from your finished product in the Cloud, there are three options. The first, and most preferable option, is to have a distribution agreement with a service like Netflix (netflix.com) or iTunes (apple.com/itunes). This works in a similar way as having a distribution deal with a television broadcaster or theatre chain, in that they’re extremely selective about what they’ll include on their service and will likely only deal with distribution companies directly.

Alternatively, you could turn to one of YouTube’s monetisation options— which is to either allow YouTube to show adverts alongside your video, in which case they’ll give you a (reportedly low) percentage of the profits, or you can gate access to the video behind a paywall, whereby you set a price and YouTube will charge viewers for access, either on a temporary “rental” basis, or for permanent access (the actual price for viewers will be different from your “suggested retail price” and will be set by Google). Vimeo has similar options for setting up pay-walls, but there’s no possibility to monetise through advertisements.

The third option is to use something like Distribber (distribber.com), an “aggregator” service that acts as a gateway to services such as iTunes and Netflix. These aggregator services charge a processing fee per video and then submit it to digital stores. They make no guarantees as to whether a video will be accepted by the services or not (and either way they keep part of the fee). As such it may be of questionable value to different types of productions.

When it comes to broadcasting live events, there are more options. Most of the popular services in this area such as Twitch (twitch.tv) cater to live coverage of televised gaming, but there are some that are less specific. Ustream (ustream.tv) will stream live videos to viewers across the Internet, though doing this for more than a few videos will cost money, with nothing built into the service to allow you to monetise your broadcasts, meaning you’d have to include your own advertisements as part of the stream if you plan to earn anything, something made easier by services such as Livestream (livestream.com), which allows for pre-roll advertisements, or advertisements inserted into the stream.

Box 13.2 The Nuts & Bolts of the Cloud

Part 2—Peer-to-Peer Networking

Though (undeservedly) synonymous with piracy, peer-to-peer networks are a technology used to make file transmission faster. With a “server-client” model, one server is responsible for transmitting copies of a file to several “clients”. If there are a lot of clients, all requesting the same file, the process can become very inefficient, as all the data transfers end up in a bottleneck at the server.

Figure 13.2 Transferring Data in a Server-Client Environment

With a peer-to-peer network, someone starts to download a file from a server as before. However, if additional clients request the same file, they instead download pieces of it from other clients instead. As more people download a specific file, there become more sources available for each person to download the file from, and so the process becomes more efficient with more popular files—in direct contrast to the server-client model.

There are some drawbacks to this approach, however. For one thing it can put undesirable strain on the clients’ systems (and Internet connections), as they’re effectively becoming servers in their own right. Moreover, there’s additional overhead in tracking and synchronising which pieces of each file are available from which client. It’s also difficult to allow “streaming” of media within this model, so files need to be downloaded in their entirety before they can be viewed. Even so, there are definite gains to be had with the technology in the right situations, with companies like Netflix reportedly interested in potential applications.

Figure 13.3 Transferring Data in a Peer-to-Peer Environment

Saving for the Future

The question of what to keep after a production is complete could just as easily be “what do you need to keep?” This in turn will depend on a variety of factors, such as how likely there is to be a sequel and the overall budget for the production. Though it may seem like there’s little point keeping anything once a film is done, there’s still the potential for some of it to be reused in the future for marketing materials (especially if the production becomes unexpectedly successful), and future technologies may also see your production getting new life.

History has shown how, for example, the proliferation of household DVD players ushered in a wave of “re-mastering” of old films and long-forgotten television series, with a great many forgotten titles finding a new audience, even for some that were not particularly successful the first time around. The re-mastering process was exactly without associated costs, but even so represented a relatively risk-free opportunity for the rights-holders to try and make some easy money on properties that were otherwise gathering dust.

In the fully digital, Cloud-enabled era we currently enjoy, such opportunities could come even easier with the proper planning. Consider the DVD “extras” that made for a great selling point for older titles, which could include out-takes, behind-the-scenes footage, concept art, and the like which previously had served little purpose once the production had wrapped. How fortunate for those productions that had archived some of those materials, to find that 20 or so years later, they had a new audience with a huge appetite for all of it.

More recently, we’ve seen the resurgence of stereoscopic 3D. Unlike the DVD re-mastering process, stereoscopic conversion (the process of taking something shot in conventional 2D and making it look as if it had been shot in 3D) is a costly and complex process, meaning it’s not something many older productions will likely go through. Even so, those that are regarded highly enough to make the process seem financially worthwhile almost certainly benefit from having many of the original assets readily available. 2013’s re-release of Jurassic Park in 3D was undoubtedly worth the cost of making it happen, but even so, that cost could have been greatly reduced had all the original visual effects elements been available (as it was, the majority of the film was treated in the same way as other in-camera footage; with elements painstakingly rotoscoped and 3D elements re-created after the fact).

That said, it’s not easy to predict exactly what would be useful to keep for the future whilst a production is in motion. From a purely idealistic perspective, it seems reasonable to just attempt to keep absolutely everything. From a practical perspective, though, doing so is often impossible, as many elements are lost or otherwise discarded even before the cameras finish rolling. Furthermore, there’s the other complicating factor: how to store whatever it is you want to keep.

Although the industry has been desperately seeking a standard for archiving different types of assets for the last decade, seemingly nothing has stuck. Still images are at least starting to seem to have some persistence—thanks to the web, it seems likely that even in 50 years’ time there’ll be a way to view a JPEG file, but once you start trying to store original RAW files that tend to be in a proprietary format, things become a little more complicated. Adobe’s Digital Negative format has seemed a likely candidate, as have the visual effects and digital film mastering formats of DPX and EXR, but these are all yet to reach the critical mass that means absolutely everyone everywhere is using them.

Move away from simple still images, and things become even more complicated. Currently, the best way to archive a finished film is to encode each individual frame as a still image, as there’s no established uncompressed movie format, and anything that is compressed as, say, an MP4 file simply suffers too much in terms of quality degradation (the MP4 file format itself lacks any sort of rigid definition, so the ability to play one MP4 file doesn’t imply the ability to play all MP4 files). Then there are things like 3D assets, which are almost always tied to the software used to create them, and so there are no guarantees that it will be possible to read the data at some unspecified point in the future (indeed, there are so many disparate elements working in tandem on a typical visual effect shot that it can be challenging to exactly re-create them even a few months after they are archived).

Still, not archiving something due to worry that it won’t be useful in the future doesn’t make for a good archiving strategy, so if we take the attitude that action is preferable to inaction, then a sensible answer to the question of what should be kept might as well be “how much did it cost to make in the first place?”, in conjunction with “how much will it cost to keep?”

The Cloud, of course, makes it easy to calculate the cost of keeping something. Whereas traditionally you’d have to consider a number of factors, the cost of Cloud storage depends only on two things: how much data do you want to store and how long you want to store it?

With this in mind, you might find that it becomes viable to keep things that you might otherwise discard. Storyboards, saved as JPEG images, for example, would probably amount to a few hundred megabytes at most, which is a lot less than what many Cloud storage providers give you for free. There’s practically no reason not to keep data that’s that small.

Digital Compression

Digital compression is a very involved and complicated topic. However, the basic concept is easy to grasp: you have some data in the form of files or metadata that occupy a specific amount of digital storage. Compression just reduces the amount of storage needed. There are a number of ways that can be done, but they fall into two categories: “lossless” compression and “lossy” compression.

Lossless compression uses mathematical principles to make the way data are organised more efficient and use less storage as a result. A common example of this is if you zip a file before emailing it. The zip file contains a copy of the original file, completely intact, but the size is smaller. Depending upon the type of data being compressed in this way, this can produce large reductions in the overall file sizes. For example, text files tend to compress extremely well compared to images or video files. The important thing about lossless compression is that no data are ever lost as a result of doing it (if you’re wondering at this point why everything isn’t just always compressed using lossless compression all the time, the reason is because there’s significant overhead required to decompress files compressed in this way so that they can be viewed).

Figure 13.4 Lossless Compression

Figure 13.5 Lossy Compression

With lossy compression, some of the data are discarded in order to reduce file size. Though this can result in much, much smaller file sizes as a result, the process can’t later be reversed to get the original file back. With lossy compression, the important factor is in deciding exactly which information is lost. Perhaps the most common example of this is the JPEG image format. With JPEG images, an algorithm removes detail in the image that are (in theory at least) imperceptible to the human eye. What this means is that, to the average person, there’s no visible difference between the original version of an image and the JPEG-compressed one, but only at around 10% of the original file size (at higher levels of compression, this breaks down and visual compression artifacts are noticeable).

There are similar options for lossy compression for digital video, notably H.264 which is used throughout the web, as well as the new H.265 (or “High Efficiency Video Coder”), which affords greater reductions of file size at the same perceptual level of quality (H.265 is protected by a number of patents and stricter licensing than H.264, so its future adoption is uncertain at this point). As with JPEG compression, even though there might be no significant difference in quality to the average viewer, the loss of data means that any attempts to modify the content can cause severe distortion or artifacts. This is particularly the case when trying to colour-correct compressed video, for example.

Two conclusions can be drawn from this with regards to archiving. First, it almost always makes sense to use some form of lossless compression when archiving data, as this will reduce the amount of storage space needed and lower costs as a result (it should be noted, though, that such compression should be set up in a way that means you still are able to access files on an individual basis, as opposed to having to retrieve an entire archive and then extracting it to get to a single file).

Second, lossy compression should be used only if the alternative would be to just not archive a file at all. If there are several terabytes of out-takes that you don’t consider being worth the expense to keep, it might be worth converting them to a smaller, lossy format and archiving that instead. Similarly, where original files are already lossy-compressed there’s no benefit to converting them to a lossless format instead (recompressing or converting them to another lossy format can also lead to further visible degradation, so this should be avoided too).

Box 13.3 Data Deduplication

One other way to reduce data storage needs is to “deduplicate” it. This process assumes that with a large enough set of files to archive, a significant proportion of them will be duplicates of each other. For example, if you need to archive everyone’s email account, there’s a good chance that many emails will have been sent to multiple people. With a deduplication system in this case, only one entire copy of the email needs to be archived, and the system just records a reference to the archived copy in place of making a new, complete copy.

Whether such a system will provide worthwhile benefits will largely depend on the type of data being archived. If most of the data are video or audio footage, and there’s known to be only a single copy of each camera take, then there’s unlikely to be any gains from such an approach, compared to, say, archiving terabytes of metadata, which would likely have significant repetition.

Long-Term Cloud Storage

Though it’s certainly possible to use any of the Cloud Storage services covered in chapter 4 to archive (or back up) files and data, there are better options available if all the following points apply to your archival data:

• ▸ It will not need to be accessed frequently;
• ▸ When it does need to be accessed, it does not need to be available immediately.

With those points in mind, there are a few options for Cloud Storage of data that can be significantly less expensive than regular Cloud Storage.

Amazon Glacier (aws.amazon.com/glacier), for example, offers extremely cheap storage (less than one cent per gigabyte per month) with no limits on how much data can be stored. However, the service is designed so that withdrawals of data are kept to a minimum. Uploading data to the service is free, as is listing what’s currently stored, but only a limited amount of data (5%) can be retrieved per month (at which point transfer fees will be applied), and you have to be prepared to wait several hours from the time you request it until it becomes available. If this approach is too inflexible for your requirements, you can try a more traditional service such as InfiniDisc (infinidisc.com), a tape-based system that allows you to retrieve data as and when you need it via the web or FTP, though it is far less cost-effective than Amazon’s offering.

GoScreening (goscreening.com)

GoScreening’s main feature is that it will digitise and host private screenings from a DVD source. The service charges per viewing (although these have to be purchased in bulk beforehand). For an undisclosed fee there’s also the option to enable watermarking and provide Blu-Ray or ProRes video instead of a DVD.

SecureScreener (primefocustechnologies.com/securescreener)

SecureScreener provides an enterprise-level service for getting screeners to viewers. A whole host of security features are included to prevent leaks or piracy, along with analytics capabilities and support for playback on mobile devices.

YouTube (youtube.com)

YouTube is the largest video hosting site in the world. Primarily intended as a way to allow anyone to publish a video online (either publically or privately), it does offer some features to content creators to allow them to monetise videos, through advertisements or video purchase and video rental options. Though the service supports up to 4k video, it has been criticised for poor visual quality, likely due to the high compression rates used. Its popularity does mean it has the added benefit of being available on almost every device with an Internet connection.

Vimeo (vimeo.com)

Though not even close to the size of YouTube, Vimeo does several things to differentiate itself from the larger service. First of all, there are no advertisements anywhere on the site (for viewers or publishers). Second, the compression quality of hosted videos is thought to be much better than YouTube’s. There are monetisation options for publishers in the form of paywalls (and the service does a much better job of promoting the videos using them than YouTube does). However, there is a limit on how much video even the paid accounts can upload per week.

Ustream (ustream.tv)

Ustream offers a convenient way to broadcast live video across the Internet to browsers and mobile devices. The paid accounts have limits on “viewer hours” or the length of a video multiplied by the number of viewers. With longer broadcasts or larger audiences, the service can be very expensive.

Livestream (livestream.com)

Livestream differentiates itself from other live broadcast services in a number of ways. First of all there are no limits on the length of broadcasts or the size of your audience. In addition, there’s a selection of software and hardware products available to buy to help control the broadcasts.

Amazon Glacier (aws.amazon.com/glacier)

Amazon Glacier is something of an alternative to tape backup systems used by IT departments across the world. Essentially you upload as much data as you want into the service, with the ultimate aim that probably won’t need to access most of the data ever again. If you do need to retrieve data from the service in the future, be prepared for a wait time of several hours, along with retrieval fees from $0.01 per gigabyte. There is also a charge incurred for deleting data that’s less than 90 days old.

InfiniDisc (infinidisc.com)

InfiniDisc offers a Cloud-based tape backup system. You can store tera-bytes of data in the system, which then behaves like your own personal tape backup system, with you able to send and retrieve files as needed. The pricing structure is much more complicated than Glacier, but whichever way you calculate the costs of the service they will likely be much more expensive.