code: private void addTokens(final Token curToken, final List<Token> tokens) {
      if (curToken.tokenLength() != 0) {
        if (this.unTokenizable) {
          tokens.add(curToken);
        } else if (!this.unTokenizable) {
          if (!replacement.matcher(curToken.getTokenValue()).matches()) {
            tokens.add(curToken);
          }
        }
      }
    }
docs: Add tokens if certain conditions are met.
  
  @param curToken
  the token to be added
  @param tokens
  the list of tokens