code: private <T extends TokenBase> List<T> createTokenList(int offset, String text) {
          ArrayList<T> result = new ArrayList<>();
  
          ViterbiLattice lattice = viterbiBuilder.build(text);
          List<ViterbiNode> bestPath = viterbiSearcher.search(lattice);
  
          for (ViterbiNode node : bestPath) {
              int wordId = node.getWordId();
              if (node.getType() == ViterbiNode.Type.KNOWN && wordId == -1) { // Do not include BOS/EOS
                  continue;
              }
              @SuppressWarnings("unchecked")
              T token = (T) tokenFactory.createToken(wordId, node.getSurface(), node.getType(),
                              offset + node.getStartIndex(), dictionaryMap.get(node.getType()));
              result.add(token);
          }
  
          return result;
      }
docs: Tokenize input sentence.
  
  @param offset   offset of sentence in original input text
  @param text sentence to tokenize
  @return list of Token