The Captain’s Log

« Previous—Next »

Boost tokenize for the win

21 May 2009

String manipulation has historically been a major shortfalling of the C++ language, or at least it was until boost came along. Something as simple as spliting a string up on a delimeter is now made so much easier thanks to the boost::tokenizer and boost string algorithms.

#include <iostream>
#include <string>
#include <boost/foreach.hpp>
#include <boost/tokenizer.hpp>

using namespace std;
using namespace boost;

int main(int argc, char** argv)
{
   string text = "token, test   string";

   char_separator<char> sep(", ");
   tokenizer<char_separator<char>> tokens(text, sep);
   for (tokenizer<char_separator<char> >::iterator it = tokens.begin(); it != tokens.end(); ++it)
   {
      cout << *it << "." << endl;
   }
}

And my personal favourite is to push the result of the tokenization into a vector/list/stl container.

#include 
#include 
#include

#include 
#include 

using namespace std;
using namespace boost;

int main(int argc, char** argv)
{
   string text = “token, test   string”;

   list tokenList;
   split(tokenList, text, is_any_of(”, “), token_compress_on);
   (list::iterator it = tokenList.begin(); it != tokenList.end(); ++it)
   {
      cout << *it << "." << endl;
   }
}

So awesome!

Stolen from the lazy programmer and reposted here for my own reference.

Feedback

Name (required)

Mail (will not be published) (required)

Website