Chapter 17 and Chapter 18 demonstrated how to create and use web services, and by now you should have a pretty good idea of the types of services that are available. (If not, Appendix C is a good source of information.) Combining two or more web services can yield some functional and easy-to-use applications. On their own, each service may be good but harder to use, or less useful, than it would be if it were combined with other services. A mashup is what you get when you combine web services. The result is a Web 2.0 application that is more sophisticated than its parts and provides functionality that most likely did not exist before.
Mashups can aid in the development of Web 2.0 applications by giving them better interactivity with maps and associated data, and can aid in the manipulation of blogs, lists, photo and video sharing, and just about any other type of service found on the Web. Without the capabilities these mashups provide, some of these existing applications would lack the necessary functionality to truly be considered Web 2.0 applications. Even worse, they may not be as useful as they could be. Web services in general, but especially when they are combined into mashups, help to give web applications a dynamic and often flashy appearance. This, in turn, makes users feel like they are using an application, and not just viewing a web site.
Mashup creation ushered in the era of Web 2.0 applications and their underlying programming. Of course, Web 2.0 applications have since evolved into more than mashups, combining dynamic HTML with visual effects and better user interaction.
So, what exactly are mashups? A simple definition of a mashup is a web site or application that combines two or more sources of information into a new web application. Another way to look at a mashup is to think of it as a hybrid web application, where parts of the application come from public interfaces such as syndicated web feeds, site scrapings, and web services.
The word mashup first became popular in the music industry, as DJs from around the world began to combine parts of existing music tracks (sometimes of different genres) to create entirely new tracks. These mashups became hits in the club scene, as a lot of them were from the techno/dance genre. When web services started to be combined to form new applications for the Web, the term mashup easily transitioned to this new medium.
The first noticeable public web application that used two different APIs was launched in April 2005. This application, created by Paul Rademacher, was the (now famous) HousingMaps.com. Paul created this application because he needed an easier way to search the housing market. By hacking the JavaScript that Google used for its maps and combining it with the classified site Craigslist, Paul created a site that allows users to visually search houses in major U.S. cities. Figure 19-1 shows the results of his efforts.
Tim O’Reilly described Paul’s creation as “the first true Web 2.0 application.” Regardless of whether it actually was the first such application, HousingMaps.com prompted web developers to realize that many untapped resources could be applied in new and creative ways.
When Paul created his mashup, Google had not yet released its maps API to the public. Soon, though, Google and other search engine companies released publicly available APIs to their resources that allowed the web development community to respond with hundreds of mashups. As a result, according to David Berlind, executive editor at ZDNet, 2.5 new mashups were being created every day. He projected that this number would reach 10 new mashups every day by 2007, showing just how solid this type of application development is for programmers.
Tip
David Berlind has a great explanation on the basics of mashups in the YouTube video “What is a Mashup?” at http://www.youtube.com/watch?v=U9sENSA_sjI.
Most web services are more data-driven than anything else, and most likely will not overwhelm an application. This is due, in large part, to the level of control the developer has on this data. However, some web services could be distracting or overwhelming to an application if the developer is not careful. Mapping services are a good example of this type of service.
Other mashups strategically use different web services to create a usable and welcomed application for the Web. These mashups do not overwhelm a user with information, unless she specifically asks for it. They can stand on their own, and they need no additions to make them more useful.
Now, I am not saying that you should not use mapping services in applications. In fact, I find Internet mapping applications (and the subject of GIS in general) to be quite interesting. I am saying that a developer can face some general pitfalls if he fails to give much thought to how the services in the application can best be utilized. There are several general things to consider when using mashups either in an application or as the application itself:
Do not add services just because you can.
Avoid application clutter.
Disparate mashups do not necessarily make cohesive applications.
Do not reinvent the wheel.
I know that the first point seems kind of simple and intuitive, but it is an easy trap to fall into. So many web service APIs are available, with more being added every day, that you may be compelled to add and add and add as new services appear. Very quickly you can have a mashup that may contain a lot of useful functionality but is simply overwhelming to the average end user. Your intended audience will have some impact on how web services are best utilized—in terms of both functionality and quantity.
Sometimes a mashup is intended to provide as much information as possible from different sources. If users know this before using the application, fine; the number of mashups is justified. Figure 19-2 shows a good example of a mashup that may include various web services just because they were available.
Figure 19-2. The Optrata mashup (http://optrata.com/), which provides individual web services because they are available, and not because they are necessarily useful
Remember that one man’s collection can be another man’s clutter. “the_optrata_mashup_open_parenthesis"/> showed one example of clutter. However, clutter could be the result of an excellent mashup that has extraordinary functionality, but whose separate web services are used in such a way as to cause a problem. Clutter happens when the user cannot find the part of the application she is looking for, or when the different parts of the mashup seem to pile on top of one another in an ill-conceived way. Figure 19-3 shows this type of pitfall.
A very important step when developing a mashup is to make sure the individual web services being used make sense when combined. Without this cohesiveness among web services, the application’s usability may suffer. Even worse, the application’s overall function may not be obvious.
To make sure your application is not suffering from a lack of cohesiveness, ask yourself whether it makes sense for each web service to be part of your application. For example, does it make sense to have a web service that gives the latest stock quotes in an application that is an interactive map of the biblical Middle East? Probably not.
Remember that a mashup is meant to be a seamless combination of individual web services into one usable application.
A last point to remember—and this goes for any type of development, not just web development—is not to reinvent the wheel. There is no point in building something that has already been built, unless you want the satisfaction of knowing you could do it. For the most part, you should build on what others have done instead of spending time and resources to build it from scratch.
Certainly, if a mashup does almost everything you require but lacks some very important functionality, it may be necessary to build your own mashup from scratch. Another option may be to contact the author of the mashup and find out whether it is possible to add the needed functionality. If you are lucky and the mashup is open source, you should be able to add what you need to the existing application without having to rewrite the whole thing.
The important thing to remember is that more mashups are appearing on the Web daily, and because of this, you need to be careful not to flood the Web with applications that are too similar to those that are already available.
We have not really discussed what mashups can do. Mashups can do just about anything you want them to do. Because all mashups are really existing services that are combined to create a new service on the Web, the sky is the limit. I recommend that you think about what you need to accomplish, and determine whether others have already handled parts of that task. If so, determine whether you can leverage that work with what you are doing.
Search for public information; chances are good that what you need is available. Not all publicly available information is free, mind you, but the amount of data you can find on the Web is amazing. Look for open source web services. That may require a little extra searching, but finding a web service that adds no costs to your application is worth the extra effort.
You can tailor mashups to do anything you want. With a little bit of work, you can put together available web services to create brand-new functionality. Open source services will cut down on costs and development efforts, and publicly available data can effectively provide the data an application needs. So, you may be asking, “What can mashups not do?” Practically nothing. Sometimes there may be a better alternative, but overall, mashups can function in just about any situation you could think of.
A big part of what goes into a mashup is the data sources that are used to create the different components in the application. Plenty of fee-based services are on the Web, but unless the mashup supports an application for a large corporation the price tag can be unrealistic for most individuals and small companies to pay. For those groups of people, it is better to try to find free or open source services to provide the data for any application that is to be built.
A lot of times, finding the data sources for an application can be harder than coding it. The availability of data is only as good as the services are at advertising it to the world. Even data that is publicly available, or is at least supposed to be in the public record, can be buried within pages and impossible to find. Also, a lot of publicly accessible data is available only for a price.
Being able to factor these variables into the budget for a project can sometimes keep the project from being canceled completely. It is important to know what needs to go into your mashup.
When it comes to public data, a wide variety of information is available, from demographics to death records. A lot of this information comes at a cost, though some government agencies are beginning to allow access to some information for free. The idea that it is public can excite many developers before they realize that the data comes at a price. Remember that there is a clear distinction between public and free—they are not the same thing.
There is a lot out there for those who look for it; some examples of publicly available data are:
Public records
Background check records
Business records
People searches
Tip
The Freedom of Information Act provides the public with access to all agency records, except those that are specifically exempted. However, it applies only to agencies of the federal government. The laws vary in all of the 50 states, their counties, their cities, and so on.
Do not confuse this section with the "Public Data" section, as these are separate entities. Public data encompasses all the data we are trying to collect. I define public records as all the records available to anyone who walks into a county clerk’s office and asks for them. Examples include records of births, deaths, marriages, divorces, and bankruptcies, as well as property records. You can access these records online, but unfortunately they come with a price tag. Sites exist that allow a user to search for data and then pay a fee to get the information. Some examples are:
People Finders (http://www.peoplefinders.com/)
Public Record Finder (http://www.publicrecordfinder.com/)
The caveat to public records is that not all states supply the needed information in a way that you can access it easily via the Web. For example, you can search birth records only for the states of California and Texas. Death records are provided only for people who possessed Social Security numbers. Marriage records can be searched in only about one-quarter of the country’s states, and the date ranges for licenses vary by state:
California (1960–1985)
Colorado (1975–2004) *
Connecticut (1966–2002)
Florida (1970–1999) *
Iowa (1835–1926)
Illinois (1793–1920)
Kentucky (1973–1999)
Maine (1892–1996)
Minnesota (1976–2003) *
Nevada (1968–2000) *
Ohio (1970–2004)
Texas (1968–1998) *
Utah (1800–1999)
West Virginia (1931–1970)
Only the states followed by an asterisk (*) provide divorce records as well. So, you can see that there is still a long way to go before most public records actually become available.
Typically, people conduct background checks if they want to determine whether someone has a criminal or sex offender record. Thankfully, the federal government provides several sources for such information. The Department of Corrections keeps data of all superior court-level felonies where the sentence handed down was probation or more than 12 months in prison.
It costs more to obtain background check records than public records, because they are more difficult to obtain. However, they can be worth the added cost because you can obtain almost everything you want to know about a person through a background check.
Data on individuals is not the only public data available. Data on businesses (both large and small) is also publicly available. You can learn everything you want to know about a business from web sites that perform searches for you. A typical business would have the following information available: its legal name, officers or owners, address, state and federal tax liens, filing information, DBA business name filings, and property ownership. This is good information to have and could greatly enhance mapping mashups.
Professional licenses are also on record and are available for searching. You can generally find the name of the business, license owner, address, and other related information.
Finally, we have public data that is gathered and kept by corporations, mainly for marketing purposes. Most of the name or phone number searches performed on the Web are conducted from these marketing sources. Here are some of the common pieces of information marketing companies compile:
Full name
Age/date of birth
Address
Phone number
Social Security number
It is frightening to realize that a marketing company can purchase such information for its databases. Some of this information is also available for anyone that owns a land-line phone number—phone books are online, and include the names, addresses, and phone numbers of individuals.
Open source services are the way to go if you want to keep your mashups as inexpensive as possible. However, they may not provide the level of support that fee-based services provide. Nonetheless, open source services usually make it easy to get data that may not exist anywhere else on the Web, or data that was available but not easily accessible through an API.
Finding these services can be easy enough, as certain web sites are solely dedicated to listing available web services on the Internet. The following is a list of a few of the better sites that track web services:
Web Service List (http://www.webservicelist.com/)
WebserviceX.NET (http://www.webservicex.net/WS/default.aspx)
Programmable Web (http://www.programmableweb.com/)
Portlets are components that you can easily plug into applications and aggregate into a page. The web services I described in Chapter 17 and Chapter 18 that were integrated into applications to give a little Web 2.0 feel were basically portlets in the application. These web services (even when combined) did not really comprise a new mashup. Mashups are created only when individual services are directly combined.
On the other hand, I feel that an application composed of three or more web services or portlets that do not necessarily interact with one another should still be considered a mashup. Take a look at the mashup in Figure 19-4. The individual portlets comprise a mashup that is basically an information web portal. This mashup uses the following web services: Flickr, Technorati, Yahoo! Image Search, and YouTube.
Web portals usually are composed of individual portlets that contain information regarding a main theme. This theme makes the portal, but the inclusion of many web services makes it a mashup.
Now that you know the details regarding mashups, it’s time to build one. By following these four easy steps, you will be on your way to building your own unique mashups for the web world to consume:
Choose a subject.
Select data sources.
Decide on the backend environment and language.
Code it.
Simple enough, right? In case it is not, the sections that follow include a little more detail to clarify the matter.
It is very hard to program anything without a little bit of direction. Start with the simplest question about the new program you are going to create: what is this a mashup of? Although it may be tempting to jump in with both feet and build a mashup that combines data sources from different areas—such as maps, real estate data, photos, search capability, and more—don’t do it!
By narrowing down the subject of your mashup, you also help to eliminate web services that you will not need while deciding which ones you do need. Now that you know roughly the types of web services you require for your mashup, it is time to hunt for them on the Internet.
The data sources you choose will directly affect the web services (and their APIs) that you picked. For example, you know you want to use the mapping data that Google provides, so you are going to use its API out of necessity. On the other hand, you may have found the data you were looking for from a government site, and you have to scrape the data yourself to make it usable in your mashup. In such cases, you have direct control over how to get the data, so you have more choices.
I suggest that you choose APIs that have good documentation associated with them so that you will have an easier time programming with them. For sources that do not have APIs, it is best if the data is given in a straightforward manner that is easy to obtain. The way the data is presented to you must always be consistent (e.g., if it is given in a tabular manner, the columns should always be in the same order).
You need to make a couple of decisions regarding the backend of the mashup before you begin programming. The first (and most obvious) decision is the language you are going to use. It makes no difference whether you use PHP, C# .NET, Perl, or Java, as long as you know the language. Sometimes the API you are using works specifically with a certain language, but most times it will not matter.
A big factor to consider with the backend is the transport type being used. You must make sure you know how to create connections to the API and handle the data coming back, regardless of whether you are using SOAP, REST, or XML-RPC. I have shown examples throughout the book on how to handle the different transport types using PHP. Other languages may have similar methods, but then again they may be completely different. You should check out books that discuss the languages of your choosing. A good starting point is Programming Web Services with SOAP by Pavel Kulchenko, James Snell, and Doug Tidwell (O’Reilly). I recommend that you then check out a few language-specific books (all O’Reilly):
Real World Web Services by Will Iverson
Java Web Services in a Nutshell by Kim Topley
Programming Web Services with Perl by Pavel Kulchenko and Randy J. Ray
Java Web Services by Dave Chappell and Tyler Jewell
Programming Web Services with XML-RPC by Edd Dumbill, Joe Johnston, and Simon St.Laurent
You have the subject for your mashup, you have chosen the data sources you’ll need, and you have the backend of the application in hand. All you need to do now is to program the mashup. If you are looking for some profound insight here or words of wisdom on how to code a mashup, I will not be giving any. Everyone has her own way of approaching a new project, and it would be best if I leave this up to you. After all, this is going to be your mashup, not mine.
It is extraordinary how mashups have taken off to such an extent that they are now a viable business solution. This is especially true when it comes to mapping services and the types of mashups you can create with them. This blending of existing technologies can have a great impact on a business when leveraged properly. You may not see this without a couple of “for examples,” so I will provide them!
First, consider a delivery company and its need to know where all of its trucks are at any given time. A web application that blends mapping services with a custom-built GPS service could drastically change how the company does business, as an application of this sort could be used for a number of different things. It could alert a controller if any of the company’s drivers was exceeding the speed limit while out on a route. There is the reduced cost of not getting driving violations and all of those associated fines, plus there is the added benefit of not giving the company a negative reputation because of trucks that are speeding or have been pulled over. It could also track the time spent at a stop, or any of the statistical data an analyst would want to improve performance with routes and delivery methods.
Still not convinced? Consider a real estate company that wants to improve its online presence and increase its revenues at the same time. Again, using a mapping service combined with a real estate service and possibly other local data services, an application could be built to meet the company’s needs. Think of the possibilities for this company if it could provide census data along with real estate data. The ability for consumers to understand the demographics and density data in a neighborhood from the comfort of their own homes would be very beneficial. It would most likely increase the real estate company’s revenue as a direct side effect, as the agents would spend less time showing houses to customers that do not suit them. The Web provides great opportunities for searching beforehand so that making purchases is easier and less painful.
The point is that you can use mashups in just about any situation, for both corporate applications and personal sites. Mashups are everywhere, and if you haven’t done so already, it’s time you added something useful to the Web.