code: public static void main(String[] args) throws IOException {
      
      boolean eolIsSignificant = (args.length > 0 && args[0].equals("-cr"));
      Reader reader = ((args.length > 0 && 
                        !args[args.length - 1].equals("-cr")) ?
                       new InputStreamReader(new FileInputStream
                                             (args[args.length - 1]), "UTF-8") :
                       new InputStreamReader(System.in, "UTF-8"));
      WhitespaceTokenizer<Word> tokenizer = 
        new WhitespaceTokenizer<Word>(new WordTokenFactory(), reader, 
                                      eolIsSignificant);
      PrintWriter pw = 
        new PrintWriter(new OutputStreamWriter(System.out, "UTF-8"), true);
      while (tokenizer.hasNext()) {
        Word w = tokenizer.next();
        if (w.value().equals(WhitespaceLexer.NEWLINE)) {
          pw.println("***CR***");
        } else {
          pw.println(w);
        }
      }
    }
docs: Reads a file from the argument and prints its tokens one per line.
  This is mainly as a testing aid, but it can also be quite useful
  standalone to turn a corpus into a one token per line file of tokens.
  <p/>
  Usage: <code>java edu.stanford.nlp.process.WhitespaceTokenizer filename
  </code>
  
  @param args Command line arguments
  @throws IOException If can't open files, etc.