code: def get_vocab_and_terms(docs):
      
      vocab = set()
      docs_terms = {}
      sum_uniques_per_doc = 0
  
      # go through the documents
      for i, (doc_label, terms) in enumerate(docs.items()):
          terms_arr = np.array(terms)
          docs_terms[doc_label] = terms_arr
  
          # update the vocab set
          terms_unique = set(terms)
          vocab |= terms_unique
  
          # update the sum of unique values per document
          sum_uniques_per_doc += len(terms_unique)
  
      doc_labels = docs_terms.keys()
  
      return np.fromiter(vocab, dtype='<U%d' % max(map(len, vocab)), count=len(vocab)), \
             np.fromiter(doc_labels, dtype='<U%d' % max(map(len, doc_labels)), count=len(doc_labels)),\
             docs_terms, sum_uniques_per_doc
docs: From a dict `docs` with document ID -> terms/tokens list mapping, generate an array of vocabulary (i.e.
      unique terms of the whole corpus `docs`), an array of document labels (i.e. document IDs), a dict
      with document ID -> document terms array mapping and a sum of the number of unique terms per document.
  
      The returned variable `sum_uniques_per_doc` tells us how many elements will be non-zero in a DTM which
      will be created later. Hence this is the allocation size for the sparse DTM.
  
      This function provides the input for create_sparse_dtm().
  
      Return a tuple with:
      - np.array of vocabulary
      - np.array of document names
      - dict with mapping: document name -> np.array of document terms
      - overall sum of unique terms per document (allocation size for the sparse DTM)