27. Splitting a string into tokens with a list of possible delimiters

Two different versions of a splitting function are listed as follows:

  • The first one uses a single character as the delimiter. To split the input string it uses a string stream initialized with the content of the input string, using std::getline() to read chunks from it until the next delimiter or an end-of-line character is encountered.
  • The second one uses a list of possible character delimiters, specified in an std::string. It uses std:string::find_first_of() to locate the first position of any of the delimiter characters, starting from a given position. It does so in a loop until the entire input string is being processed. The extracted substrings are added to the result vector:
template <class Elem>
using tstring = std::basic_string<Elem, std::char_traits<Elem>,
std::allocator<Elem>>;

template <class Elem>
using tstringstream = std::basic_stringstream<
Elem, std::char_traits<Elem>, std::allocator<Elem>>;
template<typename Elem>
inline std::vector<tstring<Elem>> split(tstring<Elem> text,
Elem const delimiter)
{
auto sstr = tstringstream<Elem>{ text };
auto tokens = std::vector<tstring<Elem>>{};
auto token = tstring<Elem>{};
while (std::getline(sstr, token, delimiter))
{
if (!token.empty()) tokens.push_back(token);
}
return tokens;
}

template<typename Elem>
inline std::vector<tstring<Elem>> split(tstring<Elem> text,
tstring<Elem> const & delimiters)
{
auto tokens = std::vector<tstring<Elem>>{};
size_t pos, prev_pos = 0;
while ((pos = text.find_first_of(delimiters, prev_pos)) !=
std::string::npos)
{
if (pos > prev_pos)
tokens.push_back(text.substr(prev_pos, pos - prev_pos));
prev_pos = pos + 1;
}
if (prev_pos < text.length())
tokens.push_back(text.substr(prev_pos, std::string::npos));
return tokens;
}

The following sample code shows two examples of how different strings can be split using either one delimiter character or multiple delimiters:

int main()
{
using namespace std::string_literals;
std::vector<std::string> expected{"this", "is", "a", "sample"};
assert(expected == split("this is a sample"s, ' '));
assert(expected == split("this,is a.sample!!"s, ",.! "s));
}
..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset