The SBD process is language-dependent and is often not straightforward. Common approaches to detect sentences include using a set of rules or training a model to detect them. A set of simple rules for detecting a sentence follows. The end of a sentence is detected if the following is true:

• The text is terminated by a period, question mark, or exclamation mark
• The period is not preceded by an abbreviation or followed by a digit

Although this works well for most sentences, it will not work for all of them. For example, it is not always easy to determine what an abbreviation is, and sequences such as ellipses may be confused with periods.

Most search engines are not concerned with SBD. They are only interested in a query's tokens and their positions. POS-taggers and other NLP tasks that perform the extraction of data will frequently process individual sentences. The detection of sentence boundaries will help separate phrases that might appear to span sentences. For example, consider the following sentences:

"The construction process was over. The hill where the house was built was short."

If we were searching for the phrase over the hill, we would inadvertently pick up it here.

Many of the examples in this chapter will use the following text to demonstrate SBD. This text consists of three simple sentences followed by a more complicated sentence:

private static String paragraph = "When determining the end of sentences " 
    + "we need to consider several factors. Sentences may end with " 
    + "exclamation marks! Or possibly questions marks? Within " 
    + "sentences we may find numbers like 3.14159, abbreviations " 
    + "such as found in Mr. Smith, and possibly ellipses either " 
    + "within a sentence ..., or at the end of a sentence...";