code: def prepare_tokens(ctx, input, annotations, tout, lout):
      
      click.echo('chemdataextractor.chemdner.prepare_tokens')
      # Collect the annotations into a dict
      anndict = defaultdict(list)
      for line in annotations:
          pmid, ta, start, end, text, category = line.strip().split('\t')
          anndict[(pmid, ta)].append((int(start), int(end), text))
      # Process the corpus
      for line in input:
          pmid, title, abstract = line.strip().split(u'\t')
          for t, section, anns in [(Title(title), 'T', anndict.get((pmid, u'T'), [])), (Paragraph(abstract), u'A', anndict.get((pmid, u'A'), []))]:
              # Write our tokens with POS and IOB tags
              tagged = _prep_tags(t, anns)
              for i, sentence in enumerate(tagged):
                  tout.write(u' '.join([u'/'.join([token, tag, label]) for token, tag, label in sentence]))
                  lout.write(u' '.join([u'/'.join([token, label]) for token, tag, label in sentence]))
                  tout.write(u'\n')
                  lout.write(u'\n')
              tout.write(u'\n')
              lout.write(u'\n')
docs: Prepare tokenized and tagged corpus file from those supplied by CHEMDNER.