code: def doc_stream(path):
      
      with open(path, 'r') as f:
          for line in f:
              if line.strip():
                  yield line
docs: Generator to feed tokenized documents (treating each line as a document).