code: public Token[] tokenize(String input) 
      {
      	char[] stringChars =  input.toCharArray();
          List<Token> tokens = new ArrayList<Token>();
          for (char c : stringChars) {
  			if(Character.isLetterOrDigit(c)){
  				tokens.add(internSomething(Character.toString(c)));
  			}
  			else if (!ignorePunctuation && !Character.isWhitespace(c)) {
  				tokens.add(internSomething(Character.toString(c)));
  			}
  		}
          return (Token[]) tokens.toArray(new BasicToken[tokens.size()]);
      }
docs: Return tokenized version of a string.  Tokens are sequences
  of alphanumerics, or any single punctuation character.