Chapter 1. Modern Search

SEO is part of design, and design is part of SEO—and the evolution of search shows us why. You don’t need to know every detail of SEO’s history to integrate search standards into your design process, but a little context will help a lot. We promise.

We’ll start off by defining what SEO is (and isn’t) and describing how search engines work, and then walk through the history of search algorithms and ranking factors. Finally, we’ll talk about the process Google uses so we can better understand how design, content, and UX impact the search experience—and ultimately how much visibility your site merits.

The Practice of SEO

Most user experiences don’t start on our carefully designed websites, but on a list of Google search results. In fact, over half of all website traffic comes from organic search, which means that SEO, for better or worse, is part of a user’s overall experience of a site (http://bkaprt.com/seo38/01-01).

While its name might imply that we’re optimizing for search engines, that’s only half of the story. The practice of SEO is as much about understanding people—the tasks they want to accomplish, the questions they have, the problems they’re trying to solve, and the language they use to describe things—as it is about making sure web crawlers (the kind that read web pages and organize them into a search index) can find and understand your content.

Optimization can take myriad forms, many of which fall into more than one category. To keep things simple, we’ll say SEO primarily affects the following five areas:

• Content: Looking at the subject, quality, clarity, accuracy, and depth of content, along with meta descriptions, alt text, keywords, and alignment with search intent.
• Information architecture: Aligning content organization, categorization, hierarchy, and labeling with search intent, keywords, and the language people use to describe what they’re looking for.
• User experience: Focusing on the features and functionality of the content, its ease of use and access, and its ability to satisfy search intent.
• Code: Addressing site speed, XML sitemaps, canonical links, robots.txt files, pagination, site migrations and redirect strategies, SSL and HTTPS standards, and more.
• Authority: Working with internal link-building, inbound links, brand reputation, marketing, credibility, reviews, social proof, and other trust factors.

And within these five areas, there’s a lot of subject matter to cover and gain expertise around. That’s why, much like the disciplines of user experience design or content strategy, SEO has different areas of focus, each with its own set of goals:

• Onsite: Optimizing web pages through information architecture, site design, usability, and the satisfaction of search intent in core web content. Most of this book will focus on this type of SEO.
• Technical: Developing websites with indexing, structured data implementation, CMS functionality, and security in mind. We’ll touch on this just enough to give you the tools to either code and build in a search-friendly way, or work effectively with developers.
• Editorial: Researching and identifying opportunities for content creation and content marketing. Editorial SEO uses a lot of the same techniques as onsite SEO, but with the goal of generating ongoing content.
• Local: Increasing visibility in organic searches for brick-and-mortar businesses. This emphasizes business listings, citations (online mentions) and contact details, and service areas and hours of operation (like “late-night coffee shops open now near me” or “24-hour urgent care”). We won’t cover this area much at all.
• Off-site: Strengthening relationships and influence with other websites and earning legitimate backlinks (inbound links to your site from other authoritative websites). Think of this as the public relations side of SEO. We won’t cover this area either.

It’s unusual to find someone who excels at every kind of SEO. Although there are a few unicorns out there, most professionals worth their salt will specialize in just one or two areas. If you’re working with consultants or adding an SEO professional to your team, make sure their strengths align with the kind of work you’re trying to do, or work with an agency that has a team of experts across the spectrum.

Mythbusting SEO

Even though you now have a picture of what the general practice of SEO looks like, you’re probably still wondering what exactly SEO can and can’t do. There are a lot of misconceptions around that, so let’s clear them up!

SEO can:

• make it easier for users to search for specific answers, information, products, or services you’re offering;
• help people find the content that’s most relevant to their geographic location or the language they’re searching in;
• help you leverage search data to improve user experiences and content;
• make your search listing stand out, feel more compelling, and provide users with a reason to click; and
• enable your content to be featured by Google in answer boxes, featured snippets, and other desirable formats.

SEO cannot:

• work overnight (it’s a long-term play);
• guarantee your site will be highly ranked in search results or used in any Google-controlled features, like answer boxes or featured snippets;
• get content on sites that lack domain or brand authority (like new microsites or sites built on subdomains) to rank highly in search results; or
• bring in lots of traffic for a product or service no one really needs, a problem no one really has, or anything people aren’t searching for in the first place.

Google Sets the Bar

Google is the standard for how we frame search, its evolution, and where things stand today. It’s the only search engine that really matters in SEO. Are there search engines besides Google? Sure. Baidu is important if you’re doing SEO for mainland China. Yandex is important if you’re optimizing for visibility in Russia. Does anyone still use Bing? Possibly. But with more than 90 percent of the search market share worldwide at the time of writing, Google easily represents the majority of users (http://bkaprt.com/seo38/01-02, http://bkaprt.com/seo38/01-03).

So Google doesn’t just lead the search market share—it dominates it. Google sets the standard for search algorithm development, as well as crawling and indexing technology (i.e., how search engines work). Therefore, we’ll focus on how SEO has evolved under Google’s leadership and the search standards the company has set. Now that we’ve gotten that out of the way, let’s dive in.

What Search Engines Do

In its Beginners Guide to SEO (which we highly recommend to anyone who wants to get a more comprehensive overview of SEO basics), SEO software and data company Moz says that search engines “exist to discover, understand, and organize the internet’s content in order to offer the most relevant results to the questions searchers are asking” (http://bkaprt.com/seo38/01-04).

To do this, search engines have four main directives (Fig 1.1):

• Crawling. Search engine bots crawl the web looking for as many URLs as they can find, and scan everything including page titles, images, keywords, code, and other linked pages. Errors in your metatags or robot.txt files (a text file that tells a search engine where you want (and don’t want) them to crawl on your site), or technology that isn’t search engine-friendly, can prevent search engines from being able to see and index your content.
• Indexing. After the search engine discovers your content in its crawl, it determines what your content is about (through semantic relevance and other factors), and then organizes and stores that content in a process called indexing. When people search Google, they’re not really searching the web—they’re searching Google’s database of crawled content.
• Ranking. Once your web pages are indexed, they’ll be shown in the search engine results page (SERP) when people make relevant queries. The algorithm takes many ranking signals into consideration to determine your content’s relevance to the search query. Organic results are then ordered from most to least relevant, under the paid results at the very top. (Keep in mind that whatever you may be doing in paid search has no effect whatsoever on your organic rankings.)
• Post-rank assessment. After your content is initially ranked, Google continuously evaluates how people interact with your page to monitor and adjust the quality of the ranking, if necessary. If Google sees any feedback signal or indication that a page isn’t satisfying the searcher’s intent or answering their question (for example, if people quickly abandon your page and return to the search engine or skip your search listing for a lower-ranking one), it tries to find a more relevant result to move higher on the SERP.

  

How SEO Earned a Bad Reputation

Google cares about giving people what they want. Even its webmaster guidelines on optimizing for search suggests asking yourself: “Does this help my users? Would I do this if search engines didn’t exist?” (http://bkaprt.com/seo38/01-05)

So why did anyone ever try to game the system? Well, for a while there, gaming algorithms was just plain easier than meeting peoples’ needs—and it worked. Early on, Google relied on overly simple ranking factors like keyword matching to indicate relevance, and backlink volume to indicate value. If a lot of people are linking to a site, then it must be good, right?

But this ranking system was pretty easy to manipulate. Google’s early algorithms weren’t sophisticated enough to take quality and legitimacy into account—so with a little strategic keyword stuffing and some backlinks from fake sites, you could be well on your way to appearing on the first page of Google search results! 

Back then, leveraging those loopholes in search algorithms became commonplace. But it was only a matter of time until Google figured out how to reinstate order amid the chaos.

How Algorithms Evolved to Set Things Right

SEO has changed a lot since its spammy early days, and Google has worked hard to codify the complexity of human search behavior. They’ve gradually built a smarter, more sophisticated search algorithm with the power to deliver meaningful, valuable, and relevant results.

Keep in mind that nobody but Google really knows the details of how their search algorithm works; it’s a highly guarded secret. To discourage people from trying to game the system, exact ranking factors aren’t disclosed.

However, Google does give us a lot of clues through their guidelines and best-practices documentation. It’s safe to say that the algorithm looks at:

• content quality,
• legitimate links and authority,
• relevance,
• search intent, and
• user experience.

Thankfully, the SEO community has run myriad tests to try to suss out which factors matter more than others, and our own experience has shown us what kind of site improvements really move the needle. We can use that collective knowledge to get a better understanding of how to optimize for relevant factors. Here’s a deeper look into what we know matters.

Content quality

Quality can be complex and subjective. It’s even more challenging to assess algorithmically, though Google tries. Of course, Google does not disclose the specifics of how they assess content quality, but they do offer writing guidelines that place a lot of focus on factors like trustworthiness, quality control, and alignment with readers’ genuine interests (http://bkaprt.com/seo38/01-06).

Legitimate links and authority

Think of links to your site from other sites as weighted votes. A link from a large, well-known, and respected site counts for a lot, while a link from a lesser-known site counts just a little. In aggregate, links help determine how authoritative your content is and exert a positive influence on how Google sees your site’s value.

But as we said earlier, this kind of system used to be easy to manipulate. After they adjusted the algorithm to home in on quality, Google turned its attention to filtering out sites engaging in fake link-building and keyword-stuffing practices. And, holy shit, was there ever a mountain of garbage to clean up. These days, you’ve got to earn your links if you want them to count.

Relevance

In 2013, Google released a significant algorithm update (more like an overhaul) called Hummingbird, which put greater emphasis on natural-language queries, focusing on context and meaning over individual keywords. It drastically improved the search engine’s semantic understanding of the web and how topics of information relate or connect to one another.

Hummingbird focused on synonyms and theme-related topics, so exact-match (meaning exactly the same word for word) keywords don’t matter as much as much as they used to. Google shows results that are relevant to the search, rather than only the searcher’s keywords. That means you don’t have to know the one correct phrase to match user queries, as long as you’re using semantically relevant language and navigation labels to clearly communicate your content.

Search intent

Following the Hummingbird update, Google made further refinements to their natural-language processing capabilities with a machine-learning artificial intelligence system called RankBrain, which continuously “teaches” Google how to understand semantics better.

More recently, Google implemented BERT, which stands for Bidirectional Encoder Representations from Transformers. BERT uses natural language processing (NLP) to understand not only nuances in words’ meanings, but also how meanings change when words are used together (http://bkaprt.com/seo38/01-07). It’s one of the biggest steps forward in the history of search.

You can see these advancements at work when you search for something like “Italian plumbers.” Previously, you might get an actual listing of plumbers in Italy, but now (unless you are performing this search in Italian while located in Italy), Google understands that you probably aren’t looking for plumbers—you’re searching for video game characters. Because BERT can understand these nuances, Google shows images of Mario and Luigi (Fig 1.2).

Google understands what searchers want and need—it understands their intent. That’s a big deal, and it means that doing the work to understand and address search intent on your site is incredibly important.

User experience

Design—including the content, code, and interface—matters in SEO because design shapes how users behave. Most SEO professionals believe Google assesses a site’s user experience by incorporating behavioral data in its ranking systems. As we’ve mentioned, Google keeps its exact use of behavioral data secret, but these are likely factors:

• Click-through rate: How many clicks a search listing receives in relation to how many times it is shown.
• Pogo-sticking: Returning to the search results for a different page. This implies that the first page didn’t satisfy the user’s intent, or the experience didn’t meet their expectations (because of slow loading times, say, or being hard to use on a mobile device).
• Dwell time: The amount of time a user spends on a page before returning to Google.
• Real-life behavior: Geopositioning a user’s physical location (via mobile apps) to see if they actually visit a place they searched for.
• Mobile experience: How your site is viewed on small screens and devices. Google’s change to mobile-first indexing determines your rankings for all screen sizes, even desktop (http://bkaprt.com/seo38/01-08).

Whether these signals are being used as direct ranking factors, feedback to train the algorithm, or something else entirely is unclear. But most SEO practitioners recognize that Google is analyzing some level of user satisfaction to ensure it sends users the most appropriate results.

No matter how the algorithm has evolved, one theme stands out: search intent matters. In a comparison of Google’s ranking signals from 2014 to 2019, Rand Fishkin, cofounder and CEO of SparkToro and formerly of Moz, demonstrated that intent had moved from the sixth most critical ranking factor all the way up to number one (Fig 1.3).

Furthermore, there’s a theme with all the ranking factors that now carry more weight in the algorithm, like intent matching, content structure, comprehensiveness, and content accuracy: they’re user-centric. As Google becomes more sophisticated in understanding what users are looking for, along with gaining a better understanding of the meaning of our content, people have become more important. At the same time, purely technical ranking factors that can be easily manipulated and have less of an impact on user experience, like keyword matching and inbound anchor text, have declined.

What Does It All Mean?

This evolution of search algorithms means that what Google really wants is for you to make pages primarily for users, not search engines! It wants you to avoid tricks intended to improve search engine rankings and do the hard work required to provide a good user experience.

It also means that one of the most important things you can do to improve search visibility for your site is to satisfy search intent—to think about why someone would come to your site in the first place, and ensure their expectations are met. In the next chapter, we’ll take a closer look at what search intent is and how you can uncover it on your own.