In this chapter, we will cover:
Playing audio from a mobile browser
Streaming video on the go
Using Appcache for offline viewing
Using Web Storage for feed or e-mail applications
Using web workers for heavy computation work
Creating Flash-like navigation with session and history API
With HTML5, you can build rich media applications to display for mobile devices. There are unlimited ways to use HTML5; the only limit is one's imagination.
In the previous chapters, we have covered semantic naming, CSS3, and Device Access categories of HTML5. In this chapter, we will go through three more categories:
Multimedia—More and more people are playing video and audio on the go, we will see how to embed these elements on mobile devices.
Offline and Storage—Offline is an important feature for mobile as the connectivity isn't consistent on a mobile device. Storage is useful for mobile to store data on the device to reduce fetching each time the user revisits the page.
Performance and Integration—With support of web workers on iOS and Blackberry, we could achieve better performance on mobile browsers.
Target browsers: iOS, Android, Blackberry, webOS, Opera Mobile, Firefox Mobile
Multimedia consists of audio and video. Playing audio on mobile can be tricky. There are a few supported audio formats on mobile browsers—Ogg Vorbis, MP3, and WAV. One issue with these formats is that all of them are not supported by all browsers.
Enter the following code in the document:
<!doctype html> <html> <head> <title>Mobile Cookbook</title> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> </head> <body> <div id="main"> <audio src="resources/snake_charmer.mp3" controls preload="auto" autobuffer> </audio> </div> </body> </html>
Now when rendering it in the browser, you will see a music player displayed as follows, and when you press play, the music should stream:
Using the audio tag is fairly simple. The audio is enclosed in the<audio></audio>
tags.
controls
tells the audio element to show visual controls such as pause, play, and so on.
autobuffer
lets the browser handle the buffering and streaming. The autobuffer
attribute has Boolean value. If it is in audio tag; audio will buffer automatically. preload=auto
makes the stream preload even before playing.
A problem with audio streaming on mobile is the format support. Here is a table showing the support comparison:
Browser |
Ogg Vorbis |
MP3 |
WAV |
---|---|---|---|
Android WebKit |
Yes |
Yes | |
Opera Mobile |
Yes | ||
Firefox Mobile |
Yes |
Yes | |
iOS Safari |
Yes |
Yes |
As shown in the table, the support has been largely inconsistent. This can be quite troublesome for cross-browser audio streaming. One way you can do it is to use multiple tracks. If a browser can't recognize a track in the first source tags, it will just try the next one. As we can see from the preceding table, the most widely supported format is MP3.
It is supported by most mobile browsers except Firefox. For Firefox, we can use Ogg, so the following code is more cross-mobile browser compatible:
<!doctype html> <html> <head> <title>Mobile Cookbook</title> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> </head> <body> <div id="main"> <audio controls preload="auto" autobuffer> <source src="resources/snake_charmer.mp3" /> <source src="resources/snake_charmer.ogg" /> </audio> </div> </body> </html>
You may ask, 'What about browsers that don't support HTML5 audio tags?' There are audio polyfills, but generally, I don't see the point of using polyfills for mobile audio. One reason is because these polyfills are made using Flash, and Flash Lite is only supported on limited mobile devices such as Symbian. One solution is to simply include a link within the audio tags. It won't be rendered by browsers that support audio tags, but it will show on browsers that don't support audio tags. You can do so by adding a download link inside just before the closing audio tags:
<div id="main"> <audio controls preload="auto" autobuffer> <a href="resources/snake_charmer.mp3">play or download here</a> </audio> </div>
Now if you render this in Windows Phone, the following will be displayed:
And if you click on the link, it will simply be opened by the system's default music player:
The current audio element lacks a client-side API. W3C Audio Working Group (http://www.w3.org/2011/audio/) was set up to address this issue. The API will support the features required by advanced interactive applications including the ability to process and synthesize audio streams directly in script. You can subscribe to participate in the discussion at: [email protected].
Target browsers: iOS, Android, Blackberry, webOS, Opera Mobile, Firefox Mobile
Some of the most visited websites from desktop platforms are video sites such as http://www.youtube.com and http://www.vimeo.com. They have a version optimized for mobiles. Video streaming is an important part of mobile. People enjoy watching videos on the go, especially short videos such as those on YouTube. They take less time to buffer and it doesn't take much time to finish watching. So how does the video work on a mobile device? Let's first create an example.
Enter the following code in to the HTML document:
<!doctype html> <html> <head> <title>Mobile Cookbook</title> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> </head> <body> <div id="main"> <video id="movie" width="320" height="240" preload controls> <source src=" http://diveintohtml5.info/i/test.mp4" /> <source src=" http://diveintohtml5.info/i/pr6.webm" type='video/webm; codecs="vp8, vorbis"' /> <source src=" http://diveintohtml5.info/i/pr6.ogv" type='video/ogg; codecs="theora, vorbis"' /> <object width="320" height="240" type="application/x-shockwave-flash"data=" http://releases.flowplayer.org/swf/flowplayer-3.2.1.swfflowplayer-3.2.1.swf"> data="flowplayer-3.2.1.swf"> <param name="movie" value=" http://releases.flowplayer.org/swf/flowplayer-3.2.1.swf" /> <param name="allowfullscreen" value="true" /> <param name="flashvars" value='config={"clip": {"url":http://diveintohtml5.info/i//test.mp4", "autoPlay":false, "autoBuffering":true}}' /> <p>Download video as <a href=" http://diveintohtml5.info/i/pr6.mp4">MP4</a>, <a href=" http://diveintohtml5.info/i/pr6.webm">WebM</a>, or <a href=" http://diveintohtml5.info/i/pr6.ogv">Ogg</a>.</p> </object> </video> <p>Try this page in Safari 4! Or you can <a href=" http://diveintohtml5.info/i//test.mp4">download the video</a> instead.</p> </div> </body> </html>
Now if you open it in a mobile browser, you should see the video player rendered.
Part of the code is taken from Mark Pilgrim's Dive into HTML5. You must be thinking, that's a hell of a lot of work to get video working! Here let's see what each part does. Both iOS and Android support H.264 (mp4
) format, webm
and ogv
versions are added to make sure it will also render in other desktop and mobile devices.
If you have multiple<source>
elements, iOS will only recognize the first one. Since iOS devices only support H.264+AAC+MP4, you have to always list your MP4 first. This bug is fixed in iOS 4.0. So in the example, we listed test.mp4
as the first one.
<source src=" http://diveintohtml5.info/i/test.mp4" /> <source src=" http://diveintohtml5.info/i/pr6.webm" type='video/webm; codecs="vp8, vorbis"' /> <source src=" http://diveintohtml5.info/i/pr6.ogv" type='video/ogg; codecs="theora, vorbis"' />
The following Flash fallback is added to make sure sites don't support HTML5 video could play the video:
<object width="320" height="240" type="application/x-shockwave-flash"data=" http://releases.flowplayer.org/swf/flowplayer-3.2.1.swfflowplayer-3.2.1.swf"> data="flowplayer-3.2.1.swf"> <param name="movie" value=" http://releases.flowplayer.org/swf/flowplayer-3.2.1.swf" /> <param name="allowfullscreen" value="true" /> <param name="flashvars" value='config={"clip": {"url": "resources http://diveintohtml5.info/i//test.mp4", "autoPlay":false, "autoBuffering":true}}' /> <p>Download video as <a href="test.mp4">MP4</a>, <a href="test.webm">WebM</a>, or <a href="test.ogv">Ogg</a>.</p> </object>
Mark Pilgrim's Dive into HTML5 has detailed information about issues that are faced while rendering a video on different browsers. You can have a read at: http://diveintohtml5.info/video.html
Versions of Android before 2.3 had a couple of issues with HTML5 video. The type attribute on<source>
elements confused earlier versions of Android greatly. The only way to get it to recognize a video source is, ironically, to omit the type attribute altogether and ensure that your H.264+AAC+MP4 video file's name ends with a .mp4
extension. You can still include the type attribute on your other video sources, as H.264 is the only video format that Android 2.2 supports. This bug is fixed in Android 2.3.
The controls
attribute was not supported. There are no ill effects to including it, but Android will not display any user interface controls for a video. You will need to provide your own user interface controls. As a minimum, you should provide a script that starts playing the video when the user clicks it. This bug is also fixed in Android 2.3.
Target browsers: iOS, Android, Opera Mobile, webOS, Firefox Mobile
Apart from Device Access, offline caching is one of the most important features for mobile devices. One of the biggest differences between desktop browsing and mobile browsing is that mobile users are always on the go. Unlike desktop browsing, which typically uses a single stable connection, mobile browsing may take place in transit, switching between 3G and WiFi and going offline entirely in such places as tunnels. Offline caching can help with issues caused by disconnection from the Internet.
Devices |
Supported |
---|---|
iOS |
Yes (3.2+) |
Android |
Yes (2.1+) |
Windows Mobile |
No |
Blackberry v6.0 and above |
No |
Symbian 60 |
No |
Palm webOS |
Yes |
Opera Mobile |
Yes |
Firefox Mobile |
Yes |
In the default.appcache
file we've just created, type in the following content:
CACHE MANIFEST # version 1 img/apple-touch-icon.png #img/splash.png NETWORK: #http://example.com/api/ FALLBACK:
Now create an HTML document and name it ch06r03.html:
<!doctype html> <html manifest="default.appcache"> <head> <title>Mobile Cookbook</title> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> </head> <body> <img src="img/apple-touch-icon.png" alt="Apple Touch Icon" /> </body> </html>
Now if you load the page, disable the Internet connection and load the page again. You can see the page still loads.
Anything under the CACHE MANIFEST
comprises the files that will be cached for offline viewing. The file that includes the cache manifest file will automatically be included and that means:
CACHE MANIFEST # version 1 img/apple-touch-icon.png #img/splash.png
The NETWORK
section lists all the URLs you DON'T want to be cached. These are the files that should be loaded each time the page is reloaded. An example of such a file is API calls. You don't want the browser to cache dynamic API returns. If all your API calls are from the same prefix, you don't have to include them all. Instead, you only have to include the prefix. For example, if you have a list of URLs as follows:
http://example.com/api/?loc=paris http://example.com/api/?loc=london
Instead of adding them one by one to the list, you can just add one:
NETWORK: http://example.com/api/
The FALLBACK
section is a place to list page URL replacements for network URLs to be used when the browser is offline or the remote server is not available.
One question you might be asking is why do we use .appcache
instead of .manifest
as the extension? It is because .appcache
is recommended by WHATWG. As it is a standard and there is no issue with browser support, it is best to use .appcache
.
Another thing you might be wondering is whether these extensions are recognized by the browsers. No worries, the following AddType
will help both .appcache
and .manifest
render with the proper MIME type. Add the following to the .htaccess
file:
AddType text/cache-manifest appcache manifest
To know more about Appcache, one can go over to the Appcache Facts site (http://appcachefacts.info/). It has much useful and valuable information about Appcache. It also maintains a list of links to sites exploring Appcache:
Dive Into HTML5 Let's Take This Offline: (http://diveintohtml5.info/offline.html)
Google Code blog Using AppCache to Launch Offline: (http://googlecode.blogspot.com/2009/04/gmail-for-mobile-html5-series-using.html)
HTML5 Rocks A Beginner's Guide to Using the Application Cache: (http://www.html5rocks.com/tutorials/appcache/beginner/)
MDN Doc Center Offline resources in Firefox: (https://developer.mozilla.org/en/offline_resources_in_firefox)
Safari Developer Library Storing Data on the Client: (http://developer.apple.com/library/safari/#documentation/appleapplications/reference/SafariWebContent/Client-SideStorage/Client-SideStorage.html)
Cache Manifest Validator Online validator, JSON(P) validation API, and TextMate
bundle: (http://manifest-validator.com/)
If you want to dig deeper into specs, read the official description of the HTML Living Standard at:
http://www.whatwg.org/specs/web-apps/current-work/multipage/ offline.html
Target browsers: cross-browser
Web Storage is very useful for offline applications, especially news feeds or e-mail web apps. When people talk about Web Storage, they usually mean the localStorage
part. It is a key/value persistence system. Apart from web storage, there are two more HTML5 storage features; they are Indexed Database API and Web SQL Database.
So let's see the pros and cons of Web Storage, Indexed Database, and Web SQL Database.
Storage type |
Pros |
Cons |
---|---|---|
Web Storage |
Simple, easy to use API Supported by major browsers |
No data privacy |
Indexed Database |
No SQL-like structured storage |
Not yet supported by most mobile browsers No SQL (obviously) |
Web SQL Database |
Fast Feature-rich SQL implementation Supported by major new mobile browsers |
W3C working group has put in on hold on the standard |
From a mobile browser support perspective, Web Storage is the most widely supported, followed by Web SQL Database.
Web SQL Database has a better feature set than Web Storage. So in this recipe, we will focus on Web Storage and Web SQL Database, and not on Indexed Database (at least for now).
First, enter the following code:
<!doctype html> <html> <head> <title>Mobile Cookbook</title> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <script src="js/modernizr.custom.54685.js"></script> </head> <body> <section> <p>Values are stored on <code>keyup</code></p> <p>Content loaded from previous sessions:</p> <div id="previous"></div> </section> <section> <div> <label for="local">localStorage:</label> <input type="text" name="local" value="" id="local" /> </div>
Now we are adding in the JavaScript portion:
<script> var addEvent = (function () { if (document.addEventListener) { return function (el, type, fn) { if (el && el.nodeName || el === window) { el.addEventListener(type, fn, false); } else if (el && el.length) { for (var i = 0; i < el.length; i++) { addEvent(el[i], type, fn); } } }; } else { return function (el, type, fn) { if (el && el.nodeName || el === window) { el.attachEvent('on' + type, function () { return fn.call(el, window.event); }); } else if (el && el.length) { for (var i = 0; i < el.length; i++) { addEvent(el[i], type, fn); } } }; } })(); function getStorage(type) { var storage = window[type + 'Storage'], delta = 0, li = document.createElement('li'), if (!window[type + 'Storage']) return; if (storage.getItem('value')) { delta = ((new Date()).getTime() - (new Date()).setTime(storage.getItem('timestamp'))) / 1000; li.innerHTML = type + 'Storage: ' + storage.getItem('value') + ' (last updated: ' + delta + 's ago)'; } else { li.innerHTML = type + 'Storage is empty'; } document.querySelector('#previous').appendChild(li); } getStorage('local'), addEvent(document.querySelector('#local'), 'keyup', function () { localStorage.setItem('value', this.value); localStorage.setItem('timestamp', (new Date()).getTime()); }); </script>
Now at the end of the file, let's close the HTML document:
</section> </body> </html>
localStorage
even works in Dolphin, a browser used by Samsung and that can be installed on any Android device. When rendering the page using the Dolphin browser, you can enter any words. For this case, if you enter "hullo world", once you hit refresh, it will display this information:
As mentioned, it is really as simple as value/key pair, and you can store data using set
and get
methods.
To set data, you use setItem
method:
localStorage.setItem('value', this.value);
To get data, you use:
storage.getItem('value')
Looking for a polyfill? jQuery Offline is a nice offline storage plugin. It uses the HTML5 localStorage
API for persistence. You can use the same API for browsers that do not support localStorage
. jQuery Offline will simply fall back to making a request to the server each time. You can learn more about it at https://github.com/wycats/jquery-offline.
Web SQL Database is an alternative to localStorage
, and it's loved by people who use SQL. Remy Sharp has a very good demo on github that shows how to use Web SQL Database. You can learn more about it at: http://html5demos.com/database.
The Web Storage Portability Layer library allows you to write offline storage code easily for browsers that support either HTML5 databases or Gears.
Gears is an earlier offline storage system developed by Google. It is supported on browsers like IE6 and IE Mobile 4.0.1, but it is no longer under development.
You can learn more about this library at: http://google-opensource.blogspot.com/2009/05/web-storage-portability-layer-common.html.
You can read more about localStorage vs. IndexedDB vs. Web SQL at: http://csimms.botonomy.com/2011/05/html5-storage-wars-localstorage-vs-indexeddb-vs-web-sql.html.
Target browsers: Opera Mobile, Firefox Mobile, iOS5, Blackberry
Most programmers with Java/Python/.NET backgrounds should be familiar with multi-threaded or concurrent programming. JavaScript was once laughed at for its lack of high-level threading, but with the advent of HTML5 its API has been expanded to allow concurrency, substantially increasing its effective power! JavaScript is no longer just a scripting language. With more and more sophisticated tasks created using JavaScript, it has to perform more while dealing with heavy frontend computing.
Devices |
Supported |
---|---|
iOS |
Yes (5.0+) |
Android |
No |
Windows Mobile |
No |
Blackberry |
Yes (6.0+) |
Symbian |
No |
Palm webOS |
No |
Opera Mobile |
Yes |
Firefox Mobile |
Yes |
Enter the following code into the document:
/* math.js */ function addNumbers(x,y) { return x + y; } function minNumbers(x,y) { return x - y; } /* Add an eventlistener to the worker, this will be called when the worker receives a message from the main page. */ this.onmessage = function (event) { var data = event.data; switch(data.op) { case 'mult': postMessage(minNumbers(data.x, data.y)); break; case 'add': postMessage(addNumbers(data.x, data.y)); break; default: postMessage("Wrong operation specified"); } };
Now, let's create an HTML document and name it ch06r05.html
. Enter the following code into the HTML file:
<!doctype html> <html> <head> <title>Mobile Cookbook</title> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <script src="js/modernizr.custom.54685.js"></script> </head> <body onload="loadDeals()"> <input type="text" id="x" value="6" /> <br /> <input type="text" id="y" value="3" /> <br /> <input type="text" id="output" /> <br /> <input type="button" id="minusButton" value="Subtract" /> <input type="button" id="addButton" value="Add" /> <script> if (Modernizr.webworkers){ alert('hi'), } /* Create a new worker */ arithmeticWorker = new Worker("js/math.js"); /* Add an event listener to the worker, this will be called whenever the worker posts any message. */ arithmeticWorker.onmessage = function (event) { document.getElementById("output").value = event.data; }; /* Register events for buttons */ document.getElementById("minusButton").onclick = function() { /* Get the values to do operation on */ x = parseFloat(document.getElementById("x").value); y = parseFloat(document.getElementById("y").value); message = { 'op' : 'min', 'x' : x, 'y' : y }; arithmeticWorker.postMessage(message); } document.getElementById("addButton").onclick = function() { /* Get the values to do operation on */ x = parseFloat(document.getElementById("x").value); y = parseFloat(document.getElementById("y").value); message = { 'op' : 'add', 'x' : x, 'y' : y }; arithmeticWorker.postMessage(message); } </script> </body> </html>
While rendering this page in a mobile browser, we can see three fields and two buttons for calculation. In the following example screenshot, I entered 6 and 3 and pressed the Add button to see 9 shown as the result:
We can break math.js
into three parts:
The actual math functions
get
event from master (HTML document)
post
event to master (HTML document)
The actual math functions are fairly easy to understand, addNumbers
is a function to add numbers and minNumbers
is for deduction:
/* math.js */ function addNumbers(x,y) { return x + y; } function minNumbers(x,y) { return x - y; }
The next is the onmessage
. This is the information the math.js
gets from the HTML document:
this.onmessage = function (event) { var data = event.data; ... };
Once the math.js
worker gets the information from the master (HTML document), it will start to do the math and post back to the master by using postMessage:
switch(data.op) { case 'mult': postMessage(minNumbers(data.x, data.y)); break; case 'add': postMessage(addNumbers(data.x, data.y)); break; default: postMessage("Wrong operation specified"); }
In the HTML document also, there are three parts as follows:
Create a worker
post
information to worker to do the math
get
the math done by worker
It is fairly easy to create a worker. It's created by calling new Worker("math.js"):
/* Create a new worker */ arithmeticWorker = new Worker("js/math.js");
For posting information to the worker, you can use the same postMessage
method as explained in math.js
. The message itself can be an object with name/value pairs:
message = { 'op' : 'min', 'x' : x, 'y' : y }; arithmeticWorker.postMessage(message);
For getting the information back once the math is done by the worker, we use the same onmessage
method explained in math.js:
Target browsers: cross-browser
In the past, people had to use hash-tag to fake URL as a compromise between SEO and smooth page transition. Now, with the history API, that hack is no longer needed. With the history API together with Ajax calls, one can dynamically update a URL.
Device platform |
Supported |
---|---|
iOS |
Yes (4.2+) |
Android |
Yes (2.2+) |
Windows Mobile |
No |
Blackberry |
No |
Symbian |
Yes (5.2+) |
Palm webOS |
No |
Opera Mobile |
No |
Firefox Mobile |
Yes |
Enter the following code in the HTML document:
<!doctype html> <html> <head> <title>Mobile Cookbook</title> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <script src="js/modernizr.custom.54685.js"></script> <style> section {width:300px; background:#ccc; padding:5px; margin:20px auto;} html, body, figure {padding:0; margin:0;} figcaption {display:block;} </style> </head> <body> <section id="gallery"> <p class="photonav"><a id="photoprev" href="ch06r06_b.html">< Previous</a> <a id="photonext" href="ch06r06_a.html">Next ></a></p> <figure id="photo"> <img id="photoimg" src="http://placekitten.com/300/300" alt="Fer" width="300" height="300"><br /> <figcaption>Adagio, 1982</figcaption> </figure> </section> <script src="js/nav.js"></script> </body> </html>
Now let's create another document and name it ch06r06_a.html
. Enter the following code into it:
<p class="photonav"><a id="photoprev" href="ch06r06_b.html">< Previous</a> <a id="photonext" href="ch06r06_b.html">Next ></a></p> <figure id="photo"> <img id="photoimg" src="http://placekitten.com/300/301" alt="Fer" width="300" height="300"> <figcaption>Aida, 1990</figcaption> </figure>
Now let's create yet another document and name it ch06r06_b.html
. Enter the following code to the document:
<p class="photonav"><a id="photoprev" href="ch06r06_a.html">< Previous</a> <a id="photonext" href="ch06r06_a.html">Next ></a></p> <figure id="photo"> <img id="photoimg" src="http://placekitten.com/300/299" alt="Fer" width="300" height="300"> <figcaption>Air Cat, 2001</figcaption> </figure>
Now let's create a JavaScript file and enter the following code. Replace the URL in the following code with your own URL:
function supports_history_api() { return !!(window.history && history.pushState); } function swapPhoto(href) { var req = new XMLHttpRequest(); req.open("GET", "http://localhost /work/packt/ch06_code/" + href.split("/").pop(), false); req.send(null); if (req.status == 200) { document.getElementById("gallery").innerHTML = req.responseText; setupHistoryClicks(); return true; } return false; } function addClicker(link) { link.addEventListener("click", function(e) { if (swapPhoto(link.href)) { history.pushState(null, null, link.href); e.preventDefault(); } }, true); } function setupHistoryClicks() { addClicker(document.getElementById("photonext")); addClicker(document.getElementById("photoprev")); } window.onload = function() { if (!supports_history_api()) { return; } setupHistoryClicks(); window.setTimeout(function() { window.addEventListener("popstate", function(e) { swapPhoto(location.pathname); }, false); }, 1); }
Now let's render the page in a mobile browser. When you click on the Previous or Next buttons, the pages will not refresh. But if you take a look at the URLs, they are updated:
history.pushState
is used to push the new URL to the browser address bar:
history.pushState(null, null, link.href);
The actual page navigation is an Ajax request to the server, so the page never reloads. But the URL is updated with the following function:
function swapPhoto(href) { var req = new XMLHttpRequest(); req.open("GET", "http://192.168.1.11:8080/work/packt/ch06_code/" + href.split("/").pop(), false); req.send(null); if (req.status == 200) { document.getElementById("gallery").innerHTML = req.responseText; setupHistoryClicks(); return true; } return false; }
To learn more about history API, you can dig into the specification at: http://www.whatwg.org/specs/web-apps/current-work/multipage/history.html
Mark Pilgrim has a great detailed explanation at Dive into HTML5:http://diveintohtml5.info/history.html
You can also learn more at Mozilla's MDC Docs:https://developer.mozilla.org/en/DOM/Manipulating_the_browser_history
Wondering where the kitten pictures have come from? It's from a site called http://placekitten.com/. A quick and simple service for getting pictures of kittens for using them as placeholders in your designs or code.