4. Video and Audio
The introduction of YouTube was a quantum leap for displaying videos online. Before the video platform came along, it was practically impossible for computer novices to make a video file available to others via the Internet: The files were usually too big to send via e-mail, and if they did arrive, the likelihood was great that they could not be played on the recipient’s computer.
YouTube on the other hand offers online storage, allowing you to save the video files. It also converts the different video formats, so they can be played with the Adobe Flash Player.
Adobe supports Flash on many operating systems, offering plug-ins for all common browsers. Browser plug-ins are generally a great idea, but the communication between plug-in and browser can sometimes be difficult if not impossible. Also, closed-source plug-ins, such as the Adobe Flash Player, are not very popular with the browser manufacturers, because they make it much harder to find the error in case of a crash.
HTML5 wanted to remedy this situation. The necessary new HTML element was easily found: video. But that was not enough to solve the problem.
4.1 A First Example
We will give you a short example to demonstrate how easy the new HTML5 video element is to use:
<!DOCTYPE html>
<title>Simple Video</title>
<video controls autoplay>
<source src='videos/mvi_2170.webm' type='video/webm'>
<source src='videos/mvi_2170.ogv' type='video/ogg'>
Sorry, your browser is unable to play this video.
</video>
You can play a video in the browser with remarkably little effort. In Figure 4.1 you can see the result in Mozilla Firefox. The HTML syntax is almost self-explanatory, but we will investigate it in more detail in the next section.
Figure 4.1 A video in WebM format in Mozilla Firefox
4.2 The “video” Element and Its Attributes
In the preceding example, two attributes are assigned to the video element: controls and autoplay. The attribute controls tells the browser to display control elements for the video (see Figure 4.1), and autoplay tells the browser to start playing the video as soon as this is possible.
Like the canvas element (see Chapter 5, Canvas), the video element belongs to the category embedded content; in other words it is one of the contents that is not directly connected to HTML. Within the embedded content, you can include an alternative solution (fallback) in case the browser does not support the video element. If this happens in our example in section 4.1, A First Example, the text Sorry, your browser is unable to play this video appears. Additionally, you could display a still image from the video. But let’s look at the possible attributes of the video element in more detail (see Table 4.1).
Table 4.1 Attributes of the “video” element
If the video element does not have a src attribute, the browser processes one or more of the source elements contained within the video element. The attributes src, type, and media are intended for this purpose (see Table 4.2). In turn, if there is a source element, you must not specify a src attribute for video.
Table 4.2 Attributes of the “source” element
The browser uses two criteria to decide which of the existing source elements will be displayed: the video’s MIME type and, if present, the media attribute in which you can specify additional limitations in the form of a CSS media query.
For CSS3, media queries were significantly expanded, so you can now have more complex instructions in addition to familiar keywords like print, screen, handheld, or projection. Here is an example:
media="screen and (min-width: 800px)"
This is where it gets interesting for video output, because depending on the browser size, the video can then be offered at different resolutions. Thanks to this trick, even mobile devices with smaller display screens and slower Internet connections can manage perfectly. A complete example for displaying a video in reduced size based on media queries looks like this:
<!DOCTYPE html>
<title>Simple Video</title>
<video controls autoplay>
<source src='videos/mvi_2170.webm' type='video/webm'
media="screen and (min-width: 500px)" >
<source src='videos/mvi_2170_qvga.webm'
type='video/webm' media="screen" >
Sorry, your browser is unable to play this video.
</video>
Browsers with less than 500 pixels in width for displaying the video will automatically display the smaller video format mvi_2170_qvga.webm.
Note
The specification of CSS3 Media Queries is currently in the Editors Draft stage. Some details are therefore likely to change. You can look up the current stage of the specification on the W3C website at http://dev.w3.org/csswg/css3-mediaqueries.
The second criterion for determining which video will be displayed is the MIME type. The optional addition of the codecs used lets the browser recognize, even before loading, whether the video can be decoded. But what are these codecs about? The following section attempts to shed light on the codec jungle.
4.3 Video Codecs
Modern video formats use a container file where audio and video contents can be saved separately. This flexible approach has several advantages. For example, several audio tracks can be saved in one file, allowing the user to switch between languages (as you would on a video DVD). Figure 4.2 shows the schematic representation of a video container file. The way in which audio and video are compressed within this container file is referred to as codec.
Figure 4.2 Schematic representation of a video container format
One bone of contention during the creation of the HTML5 specification was the definition of allowed audio and video codecs. These debates were caused on one hand by commercial interests of companies holding patents for certain coding processes and on the other by the desire to choose a capable and high-quality format. More precisely, the camp was divided into a group that supported the patent-protected video codec H.264 and another group (led by the Mozilla team) calling for the open-source format Ogg Theora. When Ian Hickson realized that this conflict could endanger the important video element, he decided to take the definition of the format out of the specification. It is now up to the browser manufacturers to decide which formats they support and for which formats they are willing to pay license fees.
Although Mozilla fought vehemently to avoid repeating the same mistake that was made in the case of the image format GIF for which CompuServe later demanded license fees, H.264 seemed to be the favorite in the race for the new, online video format. But Google did not want to passively await the misery of potential patent infringements and decided to take care of the problem. By purchasing the video specialist On2 Technologies, which had already developed important video formats, Google came to own the as yet unpublished codec VP8. During the Google developer conference, Google-IO 2010, the software giant finally let the cat out of the bag: The new project WebM, based on the video codec VP8 and the audio format Ogg Vorbis, was published as an open-source project on the Internet at http://www.webmproject.org, and was soon after implemented in Firefox and Opera.
In early 2011, Google even went one step further, announcing that support for the H.264 codec would be removed from future versions of its Chrome browser. The justification for this surprising step was that Google wants to enable open innovation, believing that the core techniques of the World Wide Web need to be based on open standards, which H.264 is not.
After this brief history we will now explain a little more about the individual formats. Don’t worry; we will not discuss the technical details of video compression at great length. We will just introduce the common formats for the Internet.
4.3.1 Ogg: Theora and Vorbis
When the Fraunhofer society began to demand license fees for the popular MP3 format at the end of the last millennium, the Xiph.Org Foundation developed the free audio codec Vorbis. Based on the video codec VP3.2, which was released in 2002 (developed by the aforementioned company On2), Xiph also created the video format Theora. Video and audio are combined in a container format, Ogg, and the container can contain one or more audio and video tracks. The MIME type for Ogg video files is video/ogg, and the corresponding filename extension is .ogv. (The file extension .ogg also works, but according to Xiph.org, we should avoid it and instead use the more explicit file extension .ogv for Ogg video and .oga for Ogg audio.)
Do not confuse the Ogg Media container format (file extension .ogm) with the Ogg container discussed here. The Ogg Media (OGM) container is an extension that supports a large number of additional video codecs. Initially, this sounds very useful, but it does lead to some problems: Xiph insists that Ogg should be mentioned only in the context of free formats, but this is not the case with Ogg Media, which can also use patented formats.
4.3.2 MPEG-4: H.264 and AAC
The MPEG-4 (MP4) container is a derivation of the multimedia format QuickTime commonly used in Apple operating systems. Like the Ogg container, MP4 can have audio and video tracks; it even goes one step further and can embed images and text. The most common codecs in MP4 are the patented video codec H.264 and the audio codec Advanced Audio Coding (AAC). The file extension is .mp4, and common media types are video/mp4, audio/mp4, and application/mp4.
Note
Apple created some confusion when files with the extension .m4a started to appear on iPods and other Apple devices. These are MP4 files, but Apple wanted the file extension to indicate that it is a pure audio file. Other file extensions used are .m4b for audio books and .m4r for iPhone ringtones.
It was mostly the huge success of Apple’s mobile devices (iPod, iPhone, iPad) that contributed to the rapid spreading of the MP4 file format. To achieve an acceptable performance when playing back videos on devices with weak processors (such as cell phones), the computer-intensive process is transferred to a separate chip. This hardware acceleration saves energy and prolongs battery life.
The patent problem regarding the H.264 codec should not be underestimated. The type of encoding is patent protected until at least 2028—a veritable sword of Damocles hanging over the software manufacturers who could be required at any time to pay fees for the encoding process.
4.3.3 WebM: VP8 and Vorbis
As mentioned at the beginning of this section, Google caused some excitement and euphoria by founding the WebM project. The video codec VP8 received very good feedback in general, and the audio codec Vorbis had already proven successful. Google decided to use the open-source format Matroska as a container, which was already tried and tested as well. But although the Matroska format supports a number of different codecs, the WebM container only allows for the video codec VP8 and the audio codec Vorbis.
The standard file extension for WebM videos is .webm, and the corresponding MIME type is video/webm.
Immediately after Google’s announcement, the browser manufacturers of Mozilla Firefox, Opera, and even Microsoft for Internet Explorer announced that they would support the WebM format. It goes without saying that Google’s browser Chrome offers support for WebM, so there is only one browser without support for the new codec (at least at the time of this writing): Apple’s Safari.
4.4 Tools for Video Conversion
Because most peoples’ digital cameras usually do not produce videos in WebM or Ogg format, the next section introduces different tools for converting videos. They are all open-source products that run on Windows, Mac OS, and Linux, except for the Miro Video Converter.
4.4.1 FFmpeg
FFmpeg is sometimes referred to as the Swiss army knife of video conversion. And rightly so, because the list of audio and video formats that FFmpeg can read and write is remarkably long. It can also split multimedia files into their components; for example, it can strip out only the audio track of a film and then convert it. If you are thinking of adding converted YouTube videos to your MP3 collection, be warned: The quality of the audio track on YouTube is usually rather disappointing.
Because the developers of FFmpeg did not bother with such trivialities as the programming of a graphic user interface, the user is expected to be none too shy about using the command line. If you do not change the FFmpeg default settings, you only need the following function call to convert an existing Flash Video (FLV) to the WebM format:
$> ffmpeg -i myflashvideo.flv myflashvideo.webm
FFmpeg is also excellent for finding out the format of a video:
$> ffmpeg -i myflashvideo.flv
...
Input #0, flv, from '/tmp/myflashvideo.flv':
Duration: 00:05:12.19, start: 24.8450, bitrate: 716 kb/s
Stream #0.0: Video: h264, yuv420p, 480x360 [PAR 1:1
DAR 4:3], 601 kb/s, 25 tbr, 1k tbn, 49.99 tbc
Stream #0.1: Audio: aac, 44100 Hz, stereo, s16,
115 kb/s
In this example, we are dealing with an approximately five-minute long video in a Flash container in which the video track is saved using the H.264 codec, and the audio track is in AAC.
Since version 0.6, FFmpeg has supported WebM videos. But the developers were not satisfied with using the libvpx library available through Google: They reimplemented VP8 based on the existing FFmpeg code, hoping to achieve considerably better performance in converting videos.
A significant part of the FFmpeg project is the libavcodec library where supported audio and video formats are saved. Players like vlc, mplayer, or xine use this library to play or re-encode videos.
Note
The list of parameters for FFmpeg is practically endless and cannot be reproduced in detail within the scope of this book. If you are interested in finding out more, please refer to the excellent FFmpeg Documentation available online at http://www.ffmpeg.org/ffmpeg-doc.html.
Table 4.3 shows some important parameters for encoding with FFmpeg.
Table 4.3 Some important FFmpeg parameters
Thanks to the option of letting FFmpeg work without user interaction, it is particularly suitable for automatic video conversion.
4.4.2 VLC
For many years, the VideoLan project has been developing the popular media player VLC, available for various operating systems (Windows, Mac OS, Linux, and other UNIX variations) with a simple graphic interface. The media player uses, among others, the libavcodec library of the FFmpeg project and therefore also supports the WebM format.
VLC does not just play videos of different formats and sources; you also have the option to convert multimedia content via the menu item Convert / Save. As you can see in Figure 4.3, you can use predefined profiles for converting to common formats—a very useful feature.
Figure 4.3 Dialog for converting videos in VLC
If you want to set quality and size of the video more precisely, you can open another dialog via Tools.
If you examine VLC more closely, you will discover further interesting functions. For example, you have the option of capturing the screen as a video (screencast) so you can record your current work or the option of streaming videos to the Net via different protocols. Of course, FFmpeg can do that, too, but VLC even has a GUI on top of that. You can download VLC at http://www.videolan.org for all common platforms.
4.4.3 Firefogg
If you are not completely comfortable using the command line and you do not want to install VLC, you can use the Firefox extension Firefogg. After installation, you can go to http://firefogg.org/make to easily select a video on your computer and convert it to the Ogg or WebM video format. Firefogg.org only offers the GUI buttons in this case; the conversion takes place on the local computer. An adapted version of FFmpeg, downloaded during the Firefogg installation, is working in the background.
In the menu item Preset you will find defaults for Ogg and WebM video in high and low quality (see Figure 4.4). You can also conveniently set metadata, such as title, author, recording date, and copyright, via the user interface.
Figure 4.4 Settings for video conversion in Firefogg
But Firefogg is more than just a graphic interface for FFmpeg. The extension comes with a JavaScript library, which makes it very easy for web developers to implement video uploads for users. The advantage is obvious: Instead of uploading a video format in low compression and then converting it on the server, the conversion takes place on the client side before the upload. This saves bandwidth and computing power on the web server’s side. Wikipedia is also betting on this concept, so we can hope that the development of Firefogg will continue.
Note
The website http://firefogg.org/dev/chunk_post_example.html shows in a few lines of source code how the Firefogg JavaScript library works. Firefogg divides the upload into 1MB chunks, which means that if the Internet connection fails, you do not need to upload the entire video again.
4.4.4 Miro Video Converter
The Miro Video Converter was developed as an offshoot of the Miro Media Player (http://www.getmiro.com), an innovative open-source audio and video player available for all common operating systems. The Miro Video Converter is only available for Windows and Mac OS. Figure 4.5 shows the simple user interface, which offers selection not only by video codec but also by device (iPad, iPhone, PlayStation, and Android phones).
Figure 4.5 Video conversion with Miro Video Converter
Load the video file via drag and drop, and FFmpeg starts the conversion. If FFmpeg should fail for any reason (which can occasionally happen), the FFmpeg Output button can help: Apart from the exact commands, it also shows all conversion status messages (see Figure 4.6). A quick Google search for the relevant error message will usually help you.
Figure 4.6 Troubleshooting during conversion with Miro
4.5 Which Format for Which Browser?
If you want to make videos available online for as many different browsers as possible, you will currently have to resort to a fallback solution for the video element. As shown in Table 4.4, there is no single video format at the moment that can be displayed by all common browsers. For the correlation of browser versions and release dates, please refer to the end of the Introduction chapter or look at the timeline shown on the website at
http://html5.komplett.cc/code/chap_intro/timeline.html?lang=en
Table 4.4 Codec support in current browsers
4.6 Interim Solutions for Older Browsers
Fortunately, not every web developer who wants to cater to different platforms or browsers has to completely reinvent the wheel. There are several free libraries online focusing on this problem. Currently, Kaltura’s JavaScript library mwEmbed has reached a very good stage of development. Wikipedia uses it to make video and audio elements available for most platforms. The main focus of this library is on the Ogg format. If you want to offer WebM and MP4 as well, use of the html5media library is a good solution.
4.6.1 mwEmbed
The mwEmbed library gained wider recognition mainly through the integration in Wikipedia. Kaltura, the company behind mwEmbed, offers integration not only for MediaWiki, the free encyclopedia’s wiki software, but also for ready-made plug-ins for common CMS and blog software like Drupal or WordPress.
To ensure that even older browsers do not choke on the new HTML5 syntax, this example adds the elements head and body:
<!DOCTYPE html>
<html>
<head>
<title>mwEmbed fallback</title>
<script type="text/javascript"
src="http://html5.kaltura.org/js" > </script>
</head>
<body>
<h1>mwEmbed fallback</h1>
<video controls autoplay>
<source src='videos/mvi_2170.mp4' type='video/mp4'>
<source src='videos/mvi_2170.webm' type='video/webm'>
<source src='videos/mvi_2170.ogv' type='video/ogg'>
Sorry, your browser is unable to play this video.
</video>
</body>
</html>
The JavaScript library mwEmbed is downloaded directly from the project website (http://html5.kaltura.org/js) and then sorts out how the video can be played. In any case, a small control bar appears at the bottom edge of the video. Figure 4.7 shows the reaction of Internet Explorer 8, which does not yet know the HTML5 video element: To play the Ogg video, it loads the Java applet Cortado.
Figure 4.7 Internet Explorer 8 with Kaltura’s fallback library mwEmbed
If you are not happy with Java applets as replacements for native video in the browser, you can use the html5media library instead.
4.6.2 html5media
The JavaScript library html5media works even more reservedly than mwEmbed and only takes action if the browser cannot play any of the specified video formats. In that case, it loads the open-source Flash Video Player Flowplayer and expects an MP4 (H.264) video as input. Unfortunately, the library contains a bug in the current version, which means that older browsers return a JavaScript error and output nothing if several source elements are specified:
<!DOCTYPE html>
<html>
<head>
<title>html5media fallback</title>
<script type="text/javascript"
src="libs/html5media.min.js" > </script>
</head>
<body>
<h1>html5media fallback</h1>
<video src="videos/mvi_2170.mp4" width=640 height=480
controls>
</video>
</body>
</html>
In this case it is important to specify the width and height; otherwise, the Flowplayer will be displayed with a height of only a few pixels. Figure 4.8 provides an example.
Figure 4.8 A video in Internet Explorer 8 playing on the free Flowplayer (Flash fallback)
4.7 Video and Scripting—A Simple Video Player
Not only can you display videos in the browser, you can also control them directly with JavaScript via the HTMLMediaElement interface. This section shows you how this works. We will implement a simple JavaScript HTML5 video player with the following features:
• Start and stop the video
• Display and set the playback position on a control bar
• Fast forward and backward
• Select specific scenes in the movie
• Switch volume between high, low, and mute
A suitable video for our video player is easily found: Big Buck Bunny—a roughly ten-minute long computer-animated cartoon, which is the result of a free film project, as its URL http://bigbuckbunny.org indicates. The project was initiated by the Blender Foundation. From October 2007 to April 2008, seven 3D animation specialists used free software, like Blender, Gimp, Inkscape, or Python, all running on Ubuntu, to create this film and made it available online under an open license. A summary of the action, based on the motto funny and furry, can be found on Wikipedia at http://en.wikipedia.org/wiki/Big_Buck_Bunny. But our main concern is the video player. Figure 4.9 shows what it will look like.
Figure 4.9 Screen shot of the JavaScript HTML5 video player
The video player’s HTML page with JavaScript library and CSS styles can be found on this book’s companion website at the following links:
• http://html5.komplett.cc/code/chap_video/js_videoPlayer_en.html
• http://html5.komplett.cc/code/chap_video/js_videoPlayer.js
• http://html5.komplett.cc/code/chap_video/js_videoPlayer.css
4.7.1 Integrating the Video
Most likely, you are already familiar with the HTML code for integrating video. Apart from the two event handler attributes oncanplay and ontimeupdate, which will play an important role later on, there is not much new here:
<video preload=metadata
poster=videos/bbb_poster.jpg
width=854 height=480
oncanplay="initControls()"
ontimeupdate="updateProgress()">
<source src='videos/bbb_480p_stereo.ogv'
type='video/ogg;codecs="theora, vorbis"'>
<!-- further source elements as alternatives -->
Sorry, your browser is unable to play this video.
</video>
With preload=metadata, we first load only so much of the film that the film duration and at least the first frame are available. During loading, we display the picture specified in the poster attribute and then the first frame, which, unfortunately, is completely black in our case.
The width and height is specified for demo purposes to reenlarge the original video—reduced from 854 × 480 to 428 × 240 after downloading—back to 854 × 480 pixels. Why? Well, the reduced version is 39MB and is easier to test than the original video at 160MB. Also, explicitly specifying the attributes width and height can help explain 80% of the short HTMLVideoElement interface. This interface consists of only four attributes for the video dimensions; an attribute for the poster frame’s URL, if there is one; and the audio attribute that reflects whether the audio track is muted or not.
Provided that the variable video contains a reference to our video element, we have the following attribute values:
• video.width = 854 (specified width)
• video.height = 480 (specified height)
• video.videoWidth = 428 (original width)
• video.videoHeight = 240 (original height)
• video.poster = URL for bbb_poster.jpg (poster frame)
These few attributes are of course not enough to implement our video player. And indeed they are only additional elements of the HTMLVideoElement, which also represents an HTMLMediaElement—the object that contains all the necessary methods and attributes. If you are curious, you can look it up in the specification at http://www.w3.org/TR/html5/video.html#htmlmediaelement.
The real work starts with oncanplay, because it refers to the JavaScript function to be executed as soon as the browser can play the video. In our example this function is initControls() where a reference to the video is created and saved in the global variable video. In the course of implementing our video player, we will have to add entries to initControls() a few more times, but for now we only need the following code:
var video;
var initControls = function() {
video = document.querySelector("VIDEO");
};
The method document.querySelector() is part of the CSS Selectors API. In the video variable it provides a reference to the first video element in the document. This gives us access to the HTMLMediaElement interface, and we can now start implementing our first feature—starting and stopping playback.
4.7.2 Starting and Stopping the Video
To start and stop playback, we first need a button in the HTML document that can react to a user clicking it:
<input type=button
value="▶"
onclick="playPause(this);">
id="playButton"
▶ is a character reference to the Unicode symbol BLACK RIGHT-POINTING TRIANGLE, which we can conveniently use as Play button. The function of starting and stopping playback is contained in playPause(), a callback function called with every click, which gets passed the button object in the argument this:
var playPause = function(ctrl) {
if (video.paused) {
video.play();
ctrl.value = String.fromCharCode('0x25AE','0x25AE'),
}
else {
video.pause();
ctrl.value = String.fromCharCode('0x25B6'),
}
};
The attribute video.paused tells us if the film is playing or not. It returns true if the film is paused and false if it is playing. This makes starting and stopping playback easy. video.start() and video.pause() are the suitable methods that in turn set video.paused to false or true accordingly.
The button object passed in the argument ctrl is used to change the button to a Pause or Play button via ctrl.value, depending on the current state. If we were to assign ▶ directly, this would not have the desired result; instead, the character string ▶ would be displayed literally as text written on the button. The correct method of creating Unicode symbols in JavaScript is via String.fromCharCode(). To this, we pass the desired UTF 16 hexadecimal codes as strings, separated by commas. Incidentally, the label text on the Pause button is made up of two BLACK VERTICAL RECTANGLE symbols (▮).
We will need the playButton ID again later on.
4.7.3 Displaying and Setting the Playback Position
To display the current playback position, we use the new input type range, previously mentioned in Chapter 3, Intelligent Forms:
<input type="range"
min=0 max=1 step=1 value=0
onchange="updateProgress(this)"
id="currentPosition">
The attributes min and max set the permitted value range, and step determines the interval by which the value will be changed when the user drags the slider. Applied to our video, min specifies the start and max the end of our film, which means that we have to set the value max to the total length of the video in seconds. The right place to do this is initControls(), the right attribute to do it with is video.duration. So we add the following lines to our initControls() function:
curPos = document.getElementById("currentPosition");
curPos.max = video.duration;
This now gives max the value 596.468017578125, which means the video is about ten-minutes long. Setting the playback position directly is done in the onchange event handler callback updateProgress() when the slider is dragged or clicked:
var updateProgress = function(ctrl) {
video.currentTime = ctrl.value;
};
A single instruction is sufficient here; the attribute video.currentTime not only reflects the current playback position, but can also be set directly. We get the suitable value from the slider’s value attribute. To implement the display of the current playback position in the format MM:SS, we still need the following steps:
1. Add a span element in connection with the slider:
<span id="timePlayed"> </span>
2. Save a reference to the span in the initControls() function and initialize this variable curTime with the value 0:00:
curTime = document.getElementById("timePlayed");
curTime.innerHTML = '0:00';
3. Update the timestamp curTime at each call of updateProgress():
mm = Math.floor(video.currentTime / 60.0);
ss = parseInt(video.currentTime) % 60;
ss = (ss < 10) ? '0'+ss : ss;
curTime.innerHTML = mm+':'+ss;
We are nearly finished. Only one essential slider feature is still missing: While the video is playing, it has to stay synchronized with the running time. The solution lies in the HTML code for integrating the video: ontimeupdate. The specification states that a timeupdate event should be triggered at intervals of at least 15 and up to 250 milliseconds during media stream playback. The event handler attribute ontimeupdate determines which callback function is called. If we set it to updateProgress(), we have found the perfect timer for synchronizing our slider.
Compared to setting the position manually by clicking or dragging the slider, we now must not change the playback position but instead set the slider and the time display to the value of video.currentTime. The slightly adapted and thus final version of our updateProgress() function is shown in Listing 4.1:
Listing 4.1 Change and update playback position
var updateProgress = function(ctrl) {
if (ctrl) {
video.currentTime = ctrl.value;
}
else {
curPos.value = video.currentTime;
}
// Setting the time in format MM:SS
mm = Math.floor(video.currentTime / 60.0);
ss = parseInt(video.currentTime) % 60;
ss = (ss < 10) ? '0'+ss : ss;
curTime.innerHTML = mm+':'+ss;
};
The purpose of the if/else block is to find out if updateProgress() was called with the slider or with ontimeupdate. In the former case, the passed slider object is assigned to ctrl, and we need to set the playback position to the slider value. In the latter case, a timeupdate event is present, and we need to set the slider to the current playback time in the variable curPos.
Now that the playback and controlling the playback position are sorted out, you have some time to sit back and relax. Take ten minutes off and go explore Big Buck Bunny with your very own, homemade, and almost finished video player!
4.7.4 Fast Forward and Backward
For these two features, we first need buttons in the HTML document. Their labels will again be Unicode symbols, this time guillemets—angle quotation marks. The Unicode name describes what they look like: LEFT-POINTING DOUBLE ANGLE QUOTATION MARK («) and RIGHT-POINTING DOUBLE ANGLE QUOTATION MARK (»). Two event listener attributes start and stop the quick search, which starts onmousedown and ends onmouseup:
<input type="button"
value="«"
onmousedown="fastFwdBwd(-1)"
onmouseup="fastFwdBwd()">
<input type="button"
value="»"
onmousedown="fastFwdBwd(1)"
onmouseup="fastFwdBwd()">
The JavaScript callback fastFwdBwb() is rather short and looks like this:
var fastFwdBwd = function(direct) {
_pause();
_play();
if (direct) {
video.playbackRate = 5.0 * direct;
}
};
Two attributes play an important role in speeding up a video. One of them we can see in our callback function with video.playbackRate. It represents the current playback rate. The second one is video.defaultPlaybackRate, a default value that determines the film’s normal speed as 1.0. For faster playback, we need to change the playback rate; for example, 2.0 would mean twice as fast, 4.0 would be four times as fast, and so on. The number and where applicable the minus sign determines the direction of playback—positive values fast forward, negative ones rewind.
According to the definition in the specification, the attribute video.playbackRate must be set to the value of video.defaultPlaybackRate each time video.play() is called. So as long as we do not crank up the defaultPlaybackRate, we can be sure that the original speed applies at each restart. To increase the speed, we therefore only need to change the video.playbackRate.
This makes the implementation of fastFwdBwd() very easy: The video is first stopped briefly. Then it is played again, and if 1 or -1 is assigned to the variable direct, the video.playbackRate is set accordingly and the speed is increased.
The functions _pause() and _play() contain code blocks for starting and stopping the video, previously found in the callback playPause(). With these functions, we can now not only control playback and pausing by clicking the Play button, but also directly via the script. To detach the functionality from the Play button, we need to define a reference to the button in initControl() via getElementById() and make it available as variable pButton. The split version of playPause() is shown in Listing 4.2:
Listing 4.2 Starting and stopping the video
var _play = function() {
video.play();
pButton.value = String.fromCharCode('0x25AE','0x25AE'),
};
var _pause = function() {
video.pause();
pButton.value = String.fromCharCode('0x25B6'),
};
var playPause = function() {
if (video.paused) {
_play();
}
else {
_pause();
}
};
4.7.5 Selecting Specific Scenes in the Film
To select individual scenes, we first need a list with timestamps and titles. A pull-down menu provides the basis:
<select name="scenes" onchange="selectScene(this)" size=19>
<option value="0:00" selected>0:00 Opening scene</option>
<option value="0:23">0:23 Title sequence</option>
<!-- 17 other entries -->
</select>
The rest is simple and taken care of by the callback selectScene(). We pass it the selected entry as the argument. Then we convert its timestamp to seconds and set video.currentTime to the resulting value. The method _play() serves us well once again and starts playing the video at the desired point:
var selectScene = function(ctrl) {
arr = ctrl.value.split(":");
video.currentTime = parseFloat((arr[0]*60)+(arr[1]*1));
updateProgress();
_play();
};
4.7.6 Set Volume to High, Low, or Mute
All that’s left is the volume control. Let’s start with a simple exercise—on/off. Once more, we need a button in the HTML code with a label formed from a Unicode symbol, this time BEAMED EIGHTH NOTES (♫):
<input type="button"
value="♫"
onclick="mute(this)">
The mute() function uses the read/write attribute video.muted to switch to mute or loud, depending on the initial setting. To give the user optical feedback, the button label is displayed in the CSS color silver when the tone is muted and in black when the volume is switched on:
var mute = function(ctrl) {
if (video.muted) {
video.muted = false;
ctrl.style.color = 'black';
}
else {
video.muted = true;
ctrl.style.color = 'silver';
}
};
Setting the volume is not complicated, either. In addition to the slider as input type range, we also need to control the label in a span. The basic HTML structure then looks like this:
<input type="range"
min=0.0 max=1.0 step=0.1 value=1.0
onchange="adjustVolume(this)"/>
<span id="currentVolume"> </span>
We define a reference to the span element in initControls(), as before, and use video.volume to initialize the volume with 100 %:
curVol = document.getElementById("currentVolume");
curVol.innerHTML = "100 %";
video.volume = 1;
The callback function adjustVolume() reacts if the slider is changed. The slider reflects with min=0 and max=1 the exact value range of video.volume and changes the volume via step=0.1 in 10% steps if the slider is dragged:
var adjustVolume = function(ctrl) {
video.volume = ctrl.value;
curVol.innerHTML = (Math.round(ctrl.value*100))+'%';
};
Our video player is now complete. This practical example has given you the chance to explore about half of the attributes and methods of the HTMLMediaElement interface. A few interesting attributes and methods are still missing; we will look at those next.
4.7.8 Other Attributes and Methods of the “HTMLMediaElement” Interface
All media elements (including not only video, but also audio) have five attributes in common, which are shown in the HTMLMediaElement interface. Apart from src as source of the media stream, there are the boolean attributes autoplay, loop, and controls, plus preload with its three values none, metadata, and auto. The code for dynamically creating a video could then look like this:
var video = document.createElement("VIDEO");
video.src = 'videos/bbb_240p_stereo.ogv';
video.autoplay = false;
video.loop = true;
video.controls = true;
video.preload = 'metadata';
But this video is not loaded yet. The loading process only starts with the next method of the HTMLMediaElement interface, video.load(). To be able to see the video in the browser, we need to append it to the DOM tree. So we add two lines to our listing:
video.load();
document.documentElement.appendChild(video);
The dynamic counterpart of the oncanplay attribute of our video player’s video element is an event listener with event type, callback function, and a flag that determines if the event should become active in the capture phase or not. Confused? Just use false for the third argument, which activates the event listener in the bubbling phase instead. If you want to know the details of how the event order works, look online at http://www.quirksmode.org/js/events_order.html. Our event listener listens for the event canplay and then immediately starts playing the film:
video.addEventListener("canplay", function() {
video.play();
}, false);
Note
The HTML version of our brief code example can of course be found online at http://html5.komplett.cc/code/chap_video/js_dynamicVideo_en.html.
As simple as this example may seem, the processes during loading a media stream are actually rather complicated. The specification distinguishes between network state and ready state, devoting two readonly attributes to these two states in the HTMLMediaElement interface, with several constants for describing the relevant state.
The attribute networkState is for monitoring the network state. It can be queried at any time and returns the possible values listed in Table 4.5.
Table 4.5 Constants of the “networkState” attribute
When selecting a suitable source, you need to remember that there are two options for doing this: either via the src attribute of the relevant element or via several source elements from which the browser can choose the most suitable one. If we are working with several source elements for a video, the question arises as to how we know which of the offered elements was in fact chosen by the browser. The answer is in the readonly attribute video.currentSrc. In the screen shot of the video player, you can see it at the bottom left before the copyright.
Actively asking if media types are supported by the relevant browser or not can be done not only by the browser when selecting the suitable source element, but also by the programmer with a script. The method we use for this is canPlayType(type) and requires a corresponding media type as an argument. The answer is probably if the browser is fairly sure that it can play the format, maybe if the browser is rather skeptical, or '' as an empty character chain if it can definitely not deal with it.
Note
See for yourself what selection of common types canPlayType(type) returns for your browser at http://html5.komplett.cc/code/chap_video/js_canPlayType.html.
The attribute readyState describes which state a media element is currently in. It has the possible values listed in Table 4.6.
Table 4.6 Constants of the “readyState” attribute
If anything should really go wrong during loading or playback, an error event is fired, narrowing down the relevant error in its code attribute:
video.addEventListener("error", function(e) {
alert(e.code);
}, false);
This callback function therefore returns one of the possible values shown in Table 4.7 in e.code.
Table 4.7 Constants in the “code” attribute of the “MediaError” interface
We have nearly reached the end of our journey through the HTMLMediaElement interface. The remaining attributes are:
• Two boolean attributes for displaying if the browser is currently searching for other data (seeking) or if the end of the stream has been reached (ended)
• An attribute for giving information on the start time of the stream (initialTime)
• An attribute that represents the current timeline offset as a Date object (startOffsetTime)
• Three attributes for implementing the TimeRanges interface—buffered, played, and seekable.
The basic idea of TimeRanges is, as its name indicates, recording periods of time:
interface TimeRanges {
readonly attribute unsigned long length;
float start(in unsigned long index);
float end(in unsigned long index);
};
Using the example of played helps you understand how this works: If we are playing the intro of the Big Buck Bunny video and then click Pause, we get a first time range consisting of a start and an end time. The corresponding attributes are played.start(0) and played.end(0), and the number of existing time ranges in played.length is 1. If we then switch to the eighth chapter and continue playback there for a bit, we create the next time range with played.start(1) and played.end(1), and the played.length becomes 2. If two time ranges should overlap, they are combined into one. All ranges are sorted in the TimeRanges object.
This way we can track which areas of a media stream are buffered, played, or marked as seekable. A little online example helps visualize the individual TimeRanges while playing the Big Buck Bunny video—take a look at http://html5.komplett.cc/code/chap_video/js_timeRanges.html.
4.7.9 The Long List of Media Events
The list of events fired on loading or playing of a media stream at certain times is long and basically reflects the three main status conditions of the HTMLMediaElement interface.
In the network state, we encounter loadstart, progress, suspend, abort, error, emptied, and stalled, and their names indicate in which network scenarios they appear. In the ready state are loadedmetadata, loadeddata, waiting, playing, canplay, or canplaythrough, all relating directly to the availability of data for the current or future playback position. In the playback state are play, pause, timeupdate, ended, ratechange, and durationchange, and again their names are as self-explanatory as is the last element we need to mention, volumechange.
When and how each event is used depends entirely on the purpose of your script. For our video player, we needed only two, oncanplay and ontimeupdate. But if we wanted to refine the details, we would almost certainly need many others as well.
If you want to read details on the various events, you should refer to the very helpful Event summary in the specification. There you will find not only a description of each event, but also indications as to when it is actually fired. Browse to http://www.w3.org/TR/html5/video.html#mediaevents.
If you want to see media events live in action, go to Philippe Le Hégaret’s HTML5 Video, Media Events, and Media Properties test page at W3C: http://www.w3.org/2010/05/video/mediaevents.html.
4.8 And What About Audio?
There is not much new to announce about audio in HTML5. Conveniently, video and audio share the HTMLMediaElement interface, which means that everything we have told you about scripting and video is also applicable to audio elements. Understandably, the additional video attributes for width, height, audio, and poster frame of the HTMLVideoElement interface are omitted. audio elements can be easily created via a constructor and have a src attribute assigned to them at the same time:
var audio = new Audio(src);
Following the pattern of our video player, let’s program an audio player for the Big Buck Bunny soundtrack. Slider, time display, and starting or stopping work in the same way as in the video example. A new feature is the menu for selecting the track: Different audio files are involved plus two buttons for jumping ahead or backward on the track list. Additionally, we implement looping at the end of all tracks plus random selection of the next track. You can see the result in Figure 4.10.
Figure 4.10 Screen shot of the JavaScript HTML5 audio player
Note
The individual tracks were extracted from the video’s soundtrack using the free, cross-platform, sound editor Audacity (http://audacity.sourceforge.net). For private use, you can also download the soundtrack without background noises for free from the homepage of the score’s composer, Jan Morgenstern, at http://www.wavemage.com/category/music.
The screen shot of the audio player will look familiar, because the new buttons once more use certain Unicode symbols for their labels. To be specific, you can see the symbols listed in Table 4.8.
Table 4.8 Unicode symbols for audio player buttons
The pull-down menu also looks familiar, but this time we do not jump to certain points in the playback time as in the video player; instead, we switch between whole tracks. The menu and the Skip backward, forward, Loop, and Shuffle buttons have this effect of changing from one track to the next, so the script logic becomes a bit more complicated.
Let’s start with the audio element:
<audio src="music/bbb_01_intro.ogg"
oncanplay="canPlay()"
ontimeupdate="updateProgress()"
onended="continueOrStop()">
</audio>
On loading the page, we set the src attribute to the first track and define three callbacks. You have already encountered the updateProgress() function, which moves the slider along and updates the time display (see Listing 4.1). The two new callbacks are canPlay(), which is called when a track is ready to play, and continueOrStop(), which decides what to do next at the end of a track. The oncanplay callback canPlay() is rather short and looks like this:
canPlay = function() {
curPos.max = audio.duration;
if (pbStatus.keepPlaying == true) {
_play();
}
};
Obviously, curPos.max adapts the slider’s max attribute, just as in the video player, but what is the subsequent if block all about? The answer is simple: We try to take the current playback status into account and only keep playing if the player was already in play mode.
So the status of the Play button determines if the audio player starts playing after switching to another track. If it is playing, it should keep playing after every track change, but if it is paused, it should only switch tracks and stay paused. This may sound complicated, but the implementation in the play button’s callback is easy; we just add the following code:
pbStatus.keepPlaying =
(pbStatus.keepPlaying == true) ? false : true;
This alternates the status variable pbStatus.keepPlaying between true and false with every click, and the correct decision is reached in canPlay().
To gain a better understanding of the audio player’s structure and functionality, look at the HTML, JavaScript, and CSS source code. You can find them online at these URLs:
• http://html5.komplett.cc/code/chap_video/js_audioPlayer_en.html
• http://html5.komplett.cc/code/chap_video/js_audioPlayer.js
• http://html5.komplett.cc/code/chap_video/js_audioPlayer.css
Back to our example. With canPlay() and pbStatus.keepPlaying, we now have control of the situation if the track is ready to play. But how do we manage switching from one track to the next? As mentioned earlier, there are several options for this: We can choose via the menu, click the Skip back and Skip forward buttons, or let the audio player do it automatically at the end of a track as a result of the settings for the Loop and Shuffle buttons. All of these options have one thing in common: They need to load a new track, and that is done via the method loadTrack():
var loadTrack = function(idx) {
audio.src = 'music/'+tracks.options[idx].value;
audio.load();
};
Two details need explaining:
1. What is hiding behind the argument idx? Hiding behind idx is the index of the track to be loaded from the pull-down menu in the variable tracks, from which we can extract file names.
2. What does the call audio.load() do? As you may have guessed, it starts loading the new track, which can be played as soon as it has reached the status canplay.
Note
To keep things simple, we use only Ogg Vorbis audio files in our example. If we wanted to offer several versions, we would first need to find the suitable format via the method canPlayType() and then load it. Try to add this function to the script when you have reached the end of this chapter!
loadTrack() is called in various ways. First, when changing tracks directly in the menu via the onchange event handler changeTrack(this):
changeTrack = function(ctrl) {
loadTrack(ctrl.options.selectedIndex);
};
Of course it is also called by the Skip forward and Skip backward buttons; their respective onclick event handler calls the callback function advanceTrack(n) and passes it the step value in the argument n as well as the desired direction via the positive or negative sign. The step value is the same in both cases, which means -1 is skip backward and 1 is skip forward:
advanceTrack = function(n) {
var idx = tracks.options.selectedIndex + n;
if (idx < 0) {
idx = idx + tracks.options.length;
}
if (idx > tracks.options.length-1) {
idx = idx - tracks.options.length;
}
tracks.options.selectedIndex = idx;
loadTrack(idx);
};
The algorithm for determining the new track is simple and consists of two phases. We first add n to the index of the selected track, and then we deal with two special cases that may arise from this: If we are currently in the first track and click Skip backward, the index becomes negative and we therefore have to keep playing the last track. If we are in the last track and click Skip forward, this also does not work, so we have to make sure the player selects the first track as next track.
The advantage of the method advanceTrack() is that we can use it even for the last two features—looping at the end of the track and random track selection. First, we quickly need to discuss exactly how the two buttons signal inactive and active. Switching between the two modes is done via onclick event handlers, which trigger the callback toggleOnOff(node) and assign the appropriate button in the argument node:
toggleOnOff = function(node) {
var cls = node.getAttribute("class");
node.setAttribute("class",
(cls == 'off') ? 'on' : 'off'
);
pbStatus[node.id] = node.getAttribute("class");
};
As the first line of the function indicates, the status is determined by the button element’s class attribute, defining the appearance via CSS. The formats for on and off can be found in the stylesheet js_audioPlayer.css:
.off {
opacity: 0.2;
}
.on {
opacity: 1.0;
}
Additionally, the current status of the relevant button is specified in the status variable pbStatus[node.id] where the node.id indicates loop or shuffle and therefore pbStatus.loop or pbStatus.shuffle is assigned on or off. The correct moment for reacting to this status is always at the end of a track. Now the callback function continueOrStop() takes effect:
continueOrStop = function() {
if (pbStatus.shuffle == 'on') {
advanceTrack(
Math.round(Math.random()*tracks.options.length)
);
}
else if (tracks.options.selectedIndex ==
tracks.options.length-1) {
if (pbStatus.loop == 'on') {
advanceTrack(1);
}
else {
pbStatus.keepPlaying = false;
}
}
else {
advanceTrack(1);
}
};
If we are in shuffle mode, rounding the result of Math.random(), multiplied by the number of all tracks, generates a random number between 0 and the total number of tracks. We then advance by this value in advanceTrack(), and it does not matter by how much we overshoot the target: If we are, for example, in the second-last track and want to skip forward five positions, the algorithm in advanceTrack() ensures that the fourth item on the menu is played.
The question “To loop or not to loop?” only ever arises in the last track. If the corresponding button is set to on mode, we start again from the beginning with advanceTrack(1); if it is in off mode, we stop here and set pbStatus.keepPlaying to false. In all other cases we simply go to the next track and start playing it.
At this point we have not only completed our audio player, but also reached the end of the chapter on video and audio. Many of the features we programmed manually in the video and audio player are of course also implemented by the browser and can be activated more easily via the controls attribute. But it still makes sense to look behind the scenes to discover the options available when scripting video and audio.
Summary
With video and audio, two important functions that previously required plug-ins become part of the HTML specification. It is difficult to predict which video codec will eventually prevail, although in light of Google’s commitment in favor of WebM, we can hope for a patent-free open format.
As the second part of the chapter shows, the HTMLMediaElement Interface makes video and audio accessible for scripting. Using JavaScript allows for an interaction that was not possible previously with the available plug-in solutions.
As for every HTML5 topic, there are many more impressive examples to be found online. Take some time to search and discover them for yourself! By reading this chapter you have laid the foundation for understanding these new and fascinating HTML5 features.