code: def corpus_token_counts(
      text_filepattern, corpus_max_lines, split_on_newlines=True):
    
    counts = collections.Counter()
    for doc in _read_filepattern(
        text_filepattern,
        max_lines=corpus_max_lines,
        split_on_newlines=split_on_newlines):
      counts.update(encode(_native_to_unicode(doc)))
  
    mlperf_log.transformer_print(
        key=mlperf_log.PREPROC_VOCAB_SIZE, value=len(counts))
    return counts
docs: Read the corpus and compute a dictionary of token counts.
  
    Args:
      text_filepattern: A pattern matching one or more files.
      corpus_max_lines: An integer; maximum total lines to read.
      split_on_newlines: A boolean. If true, then split files by lines and strip
          leading and trailing whitespace from each line. Otherwise, treat each
          file as a single string.
  
    Returns:
      a dictionary mapping token to count.