code: public String tokenizeToCoNLLOffsets() {
  
      final StringBuilder sb = new StringBuilder();
      final String[] sentences = segmenter.segmentSentence();
      final List<List<Token>> tokens = tokenizer.tokenize(sentences);
      for (final List<Token> tokSentence : tokens) {
        for (final Token token : tokSentence) {
          String tokenValue = token.getTokenValue();
          if (tokenValue.equals(RuleBasedSegmenter.PARAGRAPH)) {
            tokenValue = DEFAULT_TOKEN_VALUE;
          }
          sb.append(tokenValue.trim()).append(DELIMITER).append(token.startOffset())
              .append(DELIMITER).append(token.tokenLength()).append(LINE_BREAK);
        }
        sb.append(LINE_BREAK);
      }
      return sb.toString();
    }
docs: Tokenizes and segments input text. Outputs tokenized text in conll format:
  one token per sentence and two newlines to divide sentences plus offsets
  and lenght information about tokens.
  
  @return String tokenized text