Storing Video: Resolution

The original video resolution used before HD became popular was called SD, or Standard Definition digital video. In the United States, SD video originally used 480 lines of resolution in height, so 4:3 aspect ratio was VGA resolution or 640 by 480 and wide-screen aspect ratio SD video was 720 by 480. In Europe, SD video uses 576 lines of resolution in height, so 4:3 SD video in the EU would be 768 by 576 and wide-screen aspect ratio SD video in Europe would be 1024 by 576.

Recently High Definition (HD) digital video has become popular and uses the wide-screen 16:9 aspect ratio. There are two HD resolutions, the first was 1280 by 720 pixels, which I call Pseudo HD resolution, and the second, and now more common HD resolution, is 1920 by 1080 pixels, which is called True HD in the video industry.

Interestingly, all these resolutions are very close to common screen sizes found on Android consumer electronics devices. There are entry-level Android phones with 640 by 480 VGA screens, and mainstream Android phones with 800 by 480 WVGA screens, which are close to the 720 by 480 wide SD standard resolution. There are also 1024 by 600 entry-level (a smaller form factor) Android tablets, which are close to the European wide SD resolution of 1024 by 576.

The newer Android HD phones are 1280 by 720, or Pseudo HD, and the newest Android tablets are True HD 1920 by 1080. This is pretty convenient, as we can use the broadcast resolution standards for our video content and still hit most of the popular Android screen resolutions pixel for pixel.

Accessing Digital Video Data: Captive and Streaming

So, how do Android devices access digital video data in the first place? This can be done in one of two ways. The digital video data (file) can be captive within your application itself, in which case, it is a new media asset in your resource folder, just like your images and animation data.

The second way that digital video can be accessed is via a concept called streaming, where a digital video file is decoded (played) from an external video server outside your Android devices and applications.

The upside to streaming digital video data is that it can greatly reduce the data footprint of your application. This is because you do not have to include all that heavy new media digital video data in your .APK file.

The downside to streaming digital video is that if your user’s connection (or the video server) goes down, your video file may not always be present for your end-users to play and enjoy; thus reliability and availability of video content is a factor to be considered on the other side of this coin.

One of the central concepts in streaming digital video is the bit-rate of the digital video data. Bit-rate is defined during the compression process and thus we will be getting into that in more detail in future sections of this chapter, but let’s define it here as it’s an important video concept.

Bit-rate essentially defines how much compression your digital video data is going to have applied to it. The bit-rate is defined during the digital video compression process, which is why I am going to cover it in greater detail in the section of this chapter on data footprint optimization.

Digital video files that feature lower (small number) bit-rates are going to have more compression applied to the data, which will result in a lower level of quality, but which will play back more smoothly, across a greater number of consumer electronics devices.

This is because the bit-rate is a measure of the bits per second, or BPS, that can be processed or transmitted effectively. As a computer processor gets faster it can process more bits per second, and similarly, as a data bandwidth connection gets faster it can comfortably send or receive more bits per second as well.

So as you can see, bit/s is important not only for streaming digital video content, due to what will fit through the bandwidth, but also once it gets to the Android device it also affects what can be processed (decoded) fast enough to allow smooth playback by processors inside an Android device. Thus, the bit-rate is important for two reasons with regards to streaming video, and important only in one regard (the decoder processing speed) to captive (imbedded) video files inside your Android application.

Thus, with a captive or imbedded video inside an Android app, the lower the bit-rate that a video asset has, the more Android devices that can decode that video asset smoothly (without dropping frames).

The reason for this is fairly obvious, as fewer bits per second of digital video data to process will obviously lead to a reduced processing load on your processor. This results in superior performance, not only for the video playback, but for the Android application as a whole, and everything else that is going on inside that Android device, for that matter.

For this reason, our data footprint optimization for digital video is very important, and getting good video image quality at lower bit-rates becomes our ultimate goal, which is why we want to use the best codec available.

A bit per second is written in the video industry as: bit/s and it usually includes a size modifier, like kbit/s in front of it. This k would signify kilobits per second, which means thousands of bits per second. Most video compression bit-rates are set between 256 kbit/s (256kbps) and 768 kbit/s, although we are going to try and optimize our digital video data for our Android app to a lower range, between 192 kbit/s and 384 kbit/s, and still obtain great image quality. If we can do this, our digital video will play smoothly, and will look great across all the different types of Android consumer electronics devices. That is why this chapter is very important.

Generating Uncompressed Frames

In the next chapter, where we will get into using some more advanced Media Player controls via Java code, we can adjust our playback speed. For now, we will generate the uncompressed frames that we need inside an .AVI file format Squeeze can read to practice video compression.

1. Once Bryce has launched, use the File Open menu sequence, and locate the book resources folder and the Bryce sub-folder, and then open the Mars.br7 file, as shown in Figure 11-2. Notice the 3D data is only 77 KB.

   

   Figure 11-2. Using the File Open menu sequence in Bryce 7.1 to open Mars file

2. Once this file is open, you will see a flat land terrain mesh and horizon line; you won’t see the image this will generate until you render this 3D scene. If you want to render the first frame of the 3D Mars scene, you can click the larger (lowest centered) green sphere, in the left UI panel.
3. Next, we use the File Animation Setup . . . menu sequence to set-up our 3D animation parameters, including the Current start time or frame number, Duration, or ending time or frame number, our frame rate using FPS, and our playback parameters and type of frame display (Frame Count or SMPTE Time).
4. Set the Current time to 0:00:00.00 Frame Number 0 and the Duration time to 0:01:00.00 Frame Number 900 and set the FPS to 15 as shown in Figure 11-3.

   

   Figure 11-3. Setting animation parameters

   Note   Notice that Bryce describes Pong animation using the term: Pendulum animation!

5. I guess Pendulum animation is a clever improvement on the Pong animation description, but we are going to set our animation to Play Once and control any other playback parameter in Android. Also, set the Display parameter to Frame Count, so you will be able to see which frame is currently rendering in Bryce, once you actually render this Mars animation. First, we must configure our rendering engine.

Configuring the Render Engine

Next let’s set up our Render Options. These control 3D features, determining how long each frame of video takes to render, as shown in Figure 11-4.

1. Access the Render Options dialog by clicking the down-arrow next to the Render Sphere, opening a menu, and selecting Render Options.
2. The main determination of rendering time (or speed of frame rendering) for our Mars animation at each resolution level is set using Quality Mode settings. The more Anti-Aliasing passes we do on each frame, the longer it takes to render. To get top-quality video and a reasonable render time use the Regular setting shown in the upper-left corner of the dialog.
3. We do a Full Render on each frame, and Render With Textures so that our Mars planet surface has its reddish-rock look. We also Optimize for Uniform Scenes and Minimal Optimizations to save on per-frame render time.
4. We turn all the Post-Processing options off, because we do not need Gamma Correction, and we do not want the extra noise introduced into each frame via 48-bit Dithering, as that option makes our file size larger, as learned previously in this book.
5. We are using Perspective Projection (a normal camera lens) and No Masking as we are not using any alpha channels. Finally, we will turn off all the “expensive” calculation intensive options under Optics and Light Settings, as these are not needed for this simple planet surface fly-over, and mostly involve scenes with water, and there’s not too much water on Mars.
6. The final section, on the lower-right section of the dialog, controls the Anti-Aliasing settings, as well as the type of algorithms used to perform the Anti-Aliasing function on each frame of the 3D animation video we are creating. The algorithms get more complex (and time consuming) from top to bottom, but provide a better result, at the expense of processing cycles.
7. Finally, the AA Radius defines the number of pixels surrounding each pixel that need to be anti-aliased. Leave this set for: 1.0. AA Rays determines the number of rays or the number of ray-tracing calculations if you will that the rendering engine uses to do its anti-aliasing. Higher AA Rays values yield longer rendering times. Finally, leave AA Tolerance set to 15, and the algorithm set to the Box algorithm, as the other algorithms use far more processing time and increase your render times.

Rendering the Animation

Now, we are ready to render our Mars planet surface fly-over 3D animation, so drop-down the File Render Animation menu sequence and enter the duration of the animation and specify output parameters and file location. Refer to Figure 11-5 for the sequence we are about to go through.

1. Select the Entire Duration option, which shows the information that we entered in the Animation Setup dialog, shown in Figure 11-3.
2. Next click the Edit button, and select the Full Frames (Uncompressed) setting for our video data, and store it in an AVI (Audio Video Interleaved) container. This means we don’t have to use numbered files, which can get a bit tedious with 900 frames, as you might well imagine.
3. Next click the Set button, and specify a file location for our Mars.avi file, in your User folder, under Documents. Create a folder called Android if you don’t have one already, and then a Hello_World sub-folder inside that for this particular application’s work-in-process data files.
4. If you have a network set up and want to render across all the machines on your network, then you will need to install Bryce 7.1 Pro across all your workstations. Once this is done, you can use the Configure button at the bottom of the dialog to configure your render farm. Then, all you have to do is select the Render on Network radio button to enable this feature.
5. Once everything in this dialog is configured, as shown in Figure 11-5, you can click the check mark at the bottom right of the dialog, and the rendering of the 900 frame Mars surface animation video will commence. On a Hexa-Core 64-bit workstation at 1080×1920 resolution it took 8 hours.

Creating the Resolution Files

The next step in the work process is to create the other three resolution source files for the 270 by 480 (MDPI), 480 by 854 (HDPI), and 720 by 1280 (XHDPI) Android device resolution densities. This is done in the Document Setup dialog, which as you already know is accessed by double-clicking the green Bryce Render Sphere, as shown in the left side of Figure 11-4.

1. For each of our target DPI resolutions, we need to enter our desired target resolution into a Document Resolution field in this Document Setup (see Figure 11-6).

   

   Figure 11-6. Setting Document Resolution, Aspect Ratio, Anti-Aliasing, and Report Render Time options in Bryce

2. Make sure that your Document Aspect Ratio is 9:16 widescreen (in this case our app is utilized in portrait mode, or using a vertical orientation) and that the exact Android device screen resolutions shown above are utilized.
3. If your resolution is off by one pixel, uncheck your Constrain Proportions option (this keeps your aspect ratio locked), and change the pixel by one, and then re-check that same Constrain Proportions radio button to turn it back on again once you have made the change. As you know, we are trying to hit these mainstream (common) Android device screen resolutions pixel-for-pixel when we play our video full-screen to obtain the best quality by not needing Android to scale the data for us, which can introduce artifacts. The one thing Android is not currently really good at is scaling images and video.
4. When you click the check mark at the bottom of the dialog, shown in Figure 11-6 each time you set one of the target resolutions, you will notice that Bryce 7 scales your viewport to that number of pixels in resolution.

Next, repeat the work process shown in Figures 11-3 through 11-5 to render the three smaller resolutions, and name the files Mars270.avi, Mars480.avi, and Mars720.avi, respectively.

Notice that in Figure 11-5, the Mars.avi High Definition HD digital video file (1080 by 1920 resolution) has already been rendered, and that we are setting the filename for our Mars270.avi file in the dialog on the right-side of this screen shot. Also notice that we selected an option to Report Render Time, so that we can see what the rendering engine is doing, from a mathematical standpoint. This is shown in Figure 11-7 and I included it so we can see why our 3D rendering process is taking so much time on each of the 900 frames of digital video that we are generating.

I generated this screenshot during the 480 by 854 rendering, and since we know that 480 times 854 is 409,920, we already know how this first number was calculated. The second number is the number of pixels that were anti-aliased, and we already know that means that there are 149,699 pixels in the image where there are drastic color changes between two pixels (which simply equates to edges between different objects or colors in the image).

The rendering engine cast out 1.46 million rays to be able to define the color values for each pixel in this image, or an average of 3.55 rays per pixel. If you want the exact number, 3.55 times 409,920 yields 1,455,216.

Because we turned shadows off there are no shadow rays cast by the rendering engine; as you can imagine, that saved us some rendering time! Of the 1.46 million rays cast, 1.25 million of them hit something, and 206,376 did not hit anything, and thus were not used to create pixel data values with.

Importing the Video

The first thing that we need to do is to import our digital video AVI full frames uncompressed format file.

1. This is done by clicking the icon on the upper right labeled Import File. This opens a file navigation dialog window, shown in Figure 11-9, which allows us to navigate through the hard disk drive, and find the 3D Mars.avi source digital video files that we created earlier in Bryce 7.1 Pro.

   

   Figure 11-9. Using the Import File dialog to locate our Mars.avi source file

2. Find your My Documents folder (it should be in your Users folder) and the Android folder underneath it, which contains your Android related working assets. Find the sub-folder that you created earlier for your Hello_World application new media resources, and select the Mars.avi file and click the Open button. Once you do this you will see the first frame of the Mars video in the Video Editing portion of Sorenson Squeeze (see Figure 11-11).
3. Locate the + icon on the left at the bottom of the Presets Pane, and click it to open a new presets dialog, so that we can develop a codec that is Android specific, as shown in Figure 11-10. Type Android 1080x1920 15 FPS in the Name: and Desc: fields, to label our custom settings, which will be saved for future use with other digital video. The Format Constraints are set to None, and the Stream Type is set to Non-Streaming as we are optimizing to a .MP4 file. The best Codec to use for H.264 is MainConcept.

   

   Figure 11-10. Creating a 1080×1920 Android video compression settings preset for 15 FPS and 1 Mbps data rate

4. The reason we are using the MainConcept H.264 codec is because it is the most advanced, and features a Multi-Pass Method, which makes several passes over the video data to achieve the best quality to file size ratio possible. This takes more time, but I assume here that you have at least a quad-core workstation (if not an octa-core), and that quality is your ultimate concern. Once you select Multi-Pass set the Frame Rate: to 15 FPS and set a Target Data Rate of 1024 Kbps (which is also 1 Mbps).
5. Select Constrain Maximum Data Rate and set the Max Data Rate to 150% of 1024 Kbps, which is 1536 Kbps. This allows some wiggle room for our data to burst, if it needs a little more headroom on any given frame.
6. Set the Frame Size to 1080 by 1920 to prevent scaling and Maintain Aspect Ratio and let’s make sure the codec looks at KeyFrames every 40 frames. Finally, select Auto KeyFrame on Scene Change, and select an average keyframe frequency of 50 to start with, for this compression setting preset.
7. Once everything is set click the OK button shown in Figure 11-10 and you will be returned to the main Squeeze Pro program screen where you can now click the Apply button in the top left panel to apply these settings (see Figure 11-11).

   

   Figure 11-11. Applying the Android 1080×1920 15 FPS video compression preset to the imported Mars.avi file

8. Finally, we can click the Squeeze It! button, on the lower-right of the software screen, and start our video compression process.

While your video data is compressing, we will talk a little bit about what keyframes are, and what exactly they do in the overall video compression process.

A keyframe is a frame of your animation data that the codec looks at to store an entire image of your video at that exact frame in time, which is why, of course, it is called a “keyframe!”

The way that a codec saves space (reduces data footprint) for your video is to not have to save every single frame in the digital video, in this case, it’s a 3D animation of a fly-over of the surface of Mars.

This is done via some of the most complex mathematics in new media algorithms today, and beyond the scope of a Learn Android book to be sure. It is essentially going into the fourth dimension (time) and looking at frames after each keyframe that it encodes to see what has changed on the next frame, relative to the keyframe that it just sampled.

The codec then encodes only the changed data from frame to frame, and this can save a significant amount of encoding data in many video scenarios. A great example of this would be talking head videos, for example a teacher or a politician. If the person remains calm, still, and fixed in place, then only their mouth (speech) movements would change from frame to frame.

In this case, because that area of pixels that contains the mouth represents only a very small percentage of the entire video frame’s pixels, the codec could end up encoding only those pixels from frame to frame instead of the entire image. Much of the video frame’s pixels are frozen over time in this type of scenario, and that is something that a codec can turn into reduced data footprint, which is the name of the game where codecs are concerned.

Thus, a very still (zero head movements) talking person encodes very well, as long as the background behind them is not too noisy or does not have a lot of fast-paced movement that the codec would need to address. So a talking head in a busy newsroom would thus not encode as well as a talking head on a special set with a solid color, evenly lit background.

The things that codecs cannot transcode (code across frames) very well are noise, just like with static image codecs, and full-frame movement. Full frame movement is where every pixel in a video frame changes its location on each frame of video. Examples of this would include a very rapid camera panning, such as is used in the filming of car racing; camera zooming in or out, such as is used in the filming of nature documentaries; and camera fly-throughs, such as are used in action films and in our Mars fly-overs.

By now your digital video should be finished encoding and you will see a playable video icon in the bottom of the Squeeze Pro working area on the bottom right of the software screen, as shown in Figure 11-12.

Compressing the Video

Next, we need to compress our MDPI resolution Android screen’s video asset:

1. Use File Save to save your Squeeze Pro environment as it sits now. Next use File New to set out a new blank canvas, so we can use the Import File button to bring in our Mars270.avi raw video data file to compress into H.264 MPEG-4 data, as shown in Figure 11-13.

   

   Figure 11-13. Compressing 270×480 digital video source at 15FPS in Squeeze using 192 Kbps Multi-Pass H.264

2. Follow the same work process that we did for the 1080 version and click the + button in the Presets Panel and set up a preset for our 270 by 480 resolution video that uses a Target Data Rate of 192 Kbps and a Max Data Rate of 256 Kbps, or a 133% burst data rate ceiling. Keep KeyFrames every 30 frames (30 total per 900 frames), and be sure to lock in the resolution by specifying it, and then using Maintain Aspect Ratio, or Same as Source, as shown in Figure 11-14.

   

   Figure 11-14. Setting the codec settings for our 270×480 video file output

3. Be sure to label your new MDPI codec settings using the Name and Desc fields at the top; I used the label Android 270×480 15 FPS as it was short and descriptive and would fit well in the Presets Pane. Once everything is configured click the OK button and then select the new codec definition in the Presets Pane and click the Apply button to apply it to your Mars project settings window. Now, you are ready to click the Squeeze It! button, and compress your full frames uncompressed .AVI into an MP4 file.

The MPEG-4 file that was created was only 1.3 megabytes, so we obtained some amazing compression with these settings, given that our source (raw) data in the .AVI file was close to 342 megabytes. As we learned earlier, this gives us a 263:1 compression, and it takes the amount of data that an Android device has to deal with processing from a couple hundred megabytes in one minute, to not much more than one megabyte processed over the span of one minute, which an Android device processor should be able to handle.

It is important to note that optimizing video is never a one-time process, and if you find the video plays back smoothly across all devices, you can add in quality, by specifying a higher data rate. For example 256 Kbps with a 384 Kbps ceiling would be the next setting you would try, and more keyframes, say, use keyframes every 20 frames, instead of 30 (which would equate to 45 keyframes sampled and stored, instead of 30 over the total 900 frames). The file size goes up, as will the visual quality of the video data, and there will be more video data for the processor to decode and display.

Next, we need to compress our HDPI resolution Android screens video asset:

1. Use File Save to save your Squeeze Pro environment as it sits now. Next use File New to set out a new blank canvas, so we can use the Import File button to bring in our Mars480.avi raw video data file to compress into H.264 MPEG-4 data, as shown in Figure 11-15.

   

   Figure 11-15. Compressing 480×854 digital video source at 15FPS in Squeeze using 384 Kbps Multi-Pass H.264

2. Follow the same work process that we did for the 1080 version, and click the + button in the Presets Panel, and set up a preset for our 480 by 854 resolution video that uses the Target Data Rate of 384 Kbps and a Max Data Rate of 512 Kbps, or a 133% burst data rate ceiling. Keep KeyFrames every 30 frames (a total of 30 per 900 frames), and be sure you lock in the resolution, by either specifying it and using Maintain Aspect Ratio, or by using the Same as Source option instead, as shown in Figure 11-16.

   

   Figure 11-16. Setting the codec settings for our 480×854 video file output

3. Finally, we need to compress our XHDPI target resolution Android screen’s video asset, so again, use File Save to save your previous Squeeze Pro environment. Next use File New to set out a new blank canvas, so we can use the Import File button to bring in our Mars720.avi raw video data file to compress into an H.264 MPEG-4 MP4 data file, as shown in Figure 11-17.

   

   Figure 11-17. Compressing 720×1280 digital video source at 15 FPS in Squeeze via 768 Kbps Multi-Pass H.264

Follow the same work process that we did for the 1080 version and click the + button in the Presets Panel and set up a preset for your 720 by 1280 resolution video, which uses the Target Data Rate of 192 Kbps and Max Data Rate of 256 Kbps, or a 133% burst data rate ceiling. Keep KeyFrames every 30 frames (30 total per 900 frames), and be sure to lock in the resolution by either specifying it in the dialog UI, and using Maintain Aspect Ratio, or by using the Same as Source option instead, as shown in Figure 11-18.

Next we need to put the MPEG-4 H.264 video assets we have created into the proper Hello_World project resource folder, or Hello_World/res/raw for raw video data files that have already been optimized, and do not need to be further optimized by Android. This is a concept we covered earlier in the book, and that we will now go into in detail in the next section of the book. Once this is done, our video data will be ready to utilize.