<kaggle_start><data_title>COVID-19 Open Research Dataset Challenge (CORD-19)<data_description>### Dataset Description

In response to the COVID-19 pandemic, the White House and a coalition of leading research groups have prepared the COVID-19 Open Research Dataset (CORD-19). CORD-19 is a resource of over 51,000 scholarly articles, including over 40,000 with full text, about COVID-19, SARS-CoV-2, and related coronaviruses. This freely available dataset is provided to the global research community to apply recent advances in natural language processing and other AI techniques to generate new insights in support of the ongoing fight against this infectious disease. There is a growing urgency for these approaches because of the rapid acceleration in new coronavirus literature, making it difficult for the medical research community to keep up.

### Call to Action

We are issuing a call to action to the world's artificial intelligence experts to develop text and data mining tools that can help the medical community develop answers to high priority scientific questions.  The CORD-19 dataset represents the most extensive machine-readable coronavirus literature collection available for data mining to date. This allows the worldwide AI research community the opportunity to apply text and data mining approaches to find answers to questions within, and connect insights across, this content in support of the ongoing COVID-19 response efforts worldwide. There is a growing urgency for these approaches because of the rapid increase in coronavirus literature, making it difficult for the medical community to keep up.

A list of our initial key questions can be found under the **[Tasks](https://www.kaggle.com/allen-institute-for-ai/CORD-19-research-challenge/tasks)** section of this dataset. These key scientific questions are drawn from the NASEM’s SCIED (National Academies of Sciences, Engineering, and Medicine’s Standing Committee on Emerging Infectious Diseases and 21st Century Health Threats) [research topics](https://www.nationalacademies.org/event/03-11-2020/standing-committee-on-emerging-infectious-diseases-and-21st-century-health-threats-virtual-meeting-1) and the World Health Organization’s [R&D Blueprint](https://www.who.int/blueprint/priority-diseases/key-action/Global_Research_Forum_FINAL_VERSION_for_web_14_feb_2020.pdf?ua=1) for COVID-19.  

Many of these questions are suitable for text mining, and we encourage researchers to develop text mining tools to provide insights on these questions. 

We are maintaining a summary of the [community's contributions](https://www.kaggle.com/covid-19-contributions). For guidance on how to make your contributions useful, we're maintaining a [forum thread](https://www.kaggle.com/allen-institute-for-ai/CORD-19-research-challenge/discussion/138484) with the feedback we're getting from the medical and health policy communities. 

### Prizes

Kaggle is sponsoring a *$1,000 per task* award to the winner whose submission is identified as best meeting the evaluation criteria. The winner may elect to receive this award as a charitable donation to COVID-19 relief/research efforts or as a monetary payment. More details on the prizes and timeline can be found on the [discussion post](https://www.kaggle.com/allen-institute-for-ai/CORD-19-research-challenge/discussion/135826).

### Accessing the Dataset

We have made this dataset available on Kaggle. Watch out for [periodic updates](https://www.kaggle.com/allen-institute-for-ai/CORD-19-research-challenge/discussion/137474). 

The dataset is also hosted on [AI2's Semantic Scholar](https://pages.semanticscholar.org/coronavirus-research). And you can search the dataset using AI2's new [COVID-19 explorer](https://cord-19.apps.allenai.org/).

The licenses for each dataset can be found in the all _ sources _ metadata csv file.<data_name>CORD-19-research-challenge
<code># # Introduction
# This project uses pretrained text classifier model to get the embedding in order to find the nearest texts (or sentences) in a collection of articles. The texts to find are from the following questions.
# - Real-time tracking of whole genomes and a mechanism for coordinating the rapid dissemination of that information to inform the development of diagnostics and therapeutics and to track variations of the virus over time. [Q1](#Q1)
# - Access to geographic and temporal diverse sample sets to understand geographic distribution and genomic differences, and determine whether there is more than one strain in circulation. Multi-lateral agreements such as the Nagoya Protocol could be leveraged. [Q2](#Q2)
# - Evidence that livestock could be infected (e.g., field surveillance, genetic sequencing, receptor binding) and serve as a reservoir after the epidemic appears to be over. [Q3](#Q3)
# - Evidence of whether farmers are infected, and whether farmers could have played a role in the origin. [Q3a](#Q3a)
# - Surveillance of mixed wildlife-livestock farms for SARS-CoV-2 and other coronaviruses in Southeast Asia. [Q3b](#Q3b)
# - Experimental infections to test host range for this pathogen. [Q3c](#Q3c)
# - Animal host(s) and any evidence of continued spill-over to humans. [Q4](#Q4)
# - Socioeconomic and behavioral risk factors for this spill-over. [Q5](#Q5)
# - Sustainable risk reduction strategies. [Q6](#Q6)
# This project uses models and datasets from the following works.
# - [CORD-19: Abstract and Conclusion Word Embedding (version 17)](https://www.kaggle.com/ekaakurniawan/cord-19-abstract-and-conclusion-word-embedding?scriptVersionId=31434959) or [`skip-gram__CORD-19` dataset (version 1)](https://www.kaggle.com/ekaakurniawan/skipgram-cord19-v17/version/1).
# - [CORD-19: Abstract and Conclusion Classification (version 20)](https://www.kaggle.com/ekaakurniawan/cord-19-abstract-and-conclusion-classification?scriptVersionId=31732781) or [`brnn-gru__CORD-19` dataset (version 2)](https://www.kaggle.com/ekaakurniawan/brnngru-cord19-v19/version/2).
# Instead of using the full classification architecture, the text embedding architecture will extract text features from bottleneck layer without the fully connected layers.
# # Software Versions
import sys

print("Python %s" % sys.version)
import os
import time
import math
import random
import gc
from pathlib import Path
from abc import ABC, abstractmethod
from typing import List, Dict
from collections import Counter
import json
from pprint import pprint  # for debugging
import numpy as np

print("NumPy %s" % np.__version__)
import matplotlib

print("matplotlib %s" % matplotlib.__version__)
import matplotlib.pyplot as plt

# Set plotting options
import tensorflow as tf

print("TensorFlow %s" % tf.__version__)
from tensorflow.keras.preprocessing.text import text_to_word_sequence
from tensorflow import keras
from tensorflow.keras import layers
import pandas as pd

print("Pandas %s" % pd.__version__)
pd.set_option("display.max_colwidth", 400)
import spacy

print("spaCy %s" % spacy.__version__)

# Get ScispaCy trained model for processing biomedical, scientific or clinical text.`[1]`
import en_core_sci_md

print("en_core_sci_md %s" % en_core_sci_md.__version__)

# # Hyperparameters
# ## Word Embedding Hyperparameters
# Taken from [CORD-19: Abstract and Conclusion Word Embedding](https://www.kaggle.com/ekaakurniawan/cord-19-abstract-and-conclusion-word-embedding).
# Data
WE_DATA_NAME = "CORD-19"
WE_INPUT_DIRECTORY = "/kaggle/input/skipgram-cord19-v17/embedding"
WE_WORKING_DIRECTORY = "/kaggle/working/embedding"
# Preprocessing
WE_TEXT_PROCESSING_MODEL = en_core_sci_md
# Training
WE_MODEL_ARCHITECTURE_NAME = "skip-gram"
WE_VOCABULARY_SIZE = 147843 + 1  # Vocabulary size + UNK
WE_EMBEDDING_DIM = 256
WE_WORD_BATCH_SIZE = 128  # How many words in one batch
WE_TOTAL_NEG_SAMPLES = 64  # Number of negative examples to sample
WE_LEARNING_RATE = 1.0
WE_EPOCHS = 150
WE_REPORT_EVERY = 50000
# Validation
WE_VALIDATION_SIZE = 16  # Random set of words to evaluate similarity on
WE_VALIDATION_WINDOW = 100  # Only pick dev samples in the head of the distribution

# ## Classification Hyperparameters
# Taken from [CORD-19: Abstract and Conclusion Classification](https://www.kaggle.com/ekaakurniawan/cord-19-abstract-and-conclusion-classification).
# Data
CL_DATA_NAME = "CORD-19"
CL_INPUT_DIRECTORY = "/kaggle/input/brnngru-cord19-v19/classification"
CL_WORKING_DIRECTORY = "/kaggle/working/classification"
# Preprocessing
CL_SEQUENCE_LENGTH = 400  # Feature fixed sequence length
CL_NUM_PARALLEL_CALLS = 4
# Training
CL_MODEL_ARCHITECTURES = [
    "brnn-gru",  # Bidirectional Recurrent Neural Networks using GRU
]
CL_MODEL_ARCHITECTURE_NAME = "brnn-gru"
CL_PRETRAINED_EMBEDDING_FLAG = True  # Use pretrained embedding
CL_VOCABULARY_SIZE = 147843 + 1  # Vocabulary size + UNK
CL_EMBEDDING_DIM = 256  # Embedding dimension
CL_HIDDEN_LAYER_UNITS = 256
CL_TOTAL_CLASSES = 6  # Total classes
CL_BATCH_SIZE = 128  # Bach size
CL_EPOCHS = 60  # Number of epochs
CL_NUM_WORKERS = 4

# ## Text Embedding Hyperparameters
# Data
TE_DATA_NAME = "CORD-19"
TE_INPUT_DIRECTORY = "/kaggle/input"
TE_WORKING_DIRECTORY = "/kaggle/working/text_embedding"
# Inference
TE_BATCH_SIZE = 128  # How many texts in one batch
TE_NUM_WORKERS = 4

# ## Text Finder Hyperparameters
# Application
TF_APPLICATION_NAME = "text-finder"
# Data
TF_DATA_NAME = "CORD-19"
TF_INPUT_DIRECTORY = "/kaggle/input"
TF_WORKING_DIRECTORY = "/kaggle/working/text_finder"
# Preprocessing
TF_NUM_PARALLEL_CALLS = 4
# Model
TF_MODEL_ARCHITECTURE_NAME = "brnn-gru"
# Inference
TF_BATCH_SIZE = 1  # How many texts in one batch
TF_NUM_WORKERS = 4

# # Analyze Dataset
# Dataset source is from [COVID-19 Open Research Dataset (CORD-19)](https://pages.semanticscholar.org/coronavirus-research).`[2]` Check the website for new updates.
# List all directories in the dataset.
for dirname, _, _ in os.walk(TE_INPUT_DIRECTORY):
    print(dirname)
publication_directory = os.path.join(TE_INPUT_DIRECTORY, "CORD-19-research-challenge")
publication_directory
publication_names = [
    "custom_license",
    "biorxiv_medrxiv",
    "comm_use_subset",
    "noncomm_use_subset",
]


# ## Create Publication Model
class PublicationModel(object):
    """Publication class to store and process publication information."""

    def __init__(self, name: str, directory: str):
        """Initialize publication object.
        Arguments:
            name {str} -- the name of the publication.
            directory {str} -- the directory of the publication.
        """
        self.name = name
        self.directory = directory
        self.article_filenames: List[str] = []

    def collect_articles(self):
        """Collect all article filenames in publication directory."""
        self.article_filenames = os.listdir(
            os.path.join(self.directory, self.name, self.name, "pdf_json")
        )

    def __repr__(self):
        """Return publication short information."""
        return f"{self.name} with {len(self.article_filenames)} articles."


publications: List[PublicationModel] = []
for publication_name in publication_names:
    publication = PublicationModel(publication_name, publication_directory)
    publication.collect_articles()
    print(publication.__repr__())
    publications.append(publication)
total_articles = 0
for publication in publications:
    total_articles += len(publication.article_filenames)
print("total articles:", total_articles)


# ## Create Article Model
class ArticleModel(object):
    """Article class to store the entire article information."""

    def __init__(self, paper_id: str, title: str, abstract: str, conclusion: str):
        """Initialize article object.
        Arguments:
            paper_id {str} -- 40-character sha1 of the PDF.
            title {str} -- the title of the article.
            abstract {str} -- the text abstract of the article.
            conclusion {str} -- the text conclusion of the article.
        """
        self.paper_id = paper_id
        self.title = title
        self.abstract = abstract
        self.conclusion = conclusion

    def __repr__(self):
        """Return article in readable string."""
        return (
            f"{self.paper_id}\n{self.title}\n\n"
            + f"Abstract:\n{self.abstract}\n\n"
            + f"Conclusion:\n{self.conclusion}\n"
        )


# ## Create Article Ingestor
class ArticleIngestorInterface(ABC):
    """Abstract class for ingestor classes."""

    allowed_extensions: List[str] = []

    @classmethod
    def can_ingest(cls, path: str) -> bool:
        """Check if file extension is supported for ingestion.
        Arguments:
            path {str} -- article file location.
        Returns:
            bool -- True if the extension is supported and False otherwise.
        """
        ext = path.split(".")[-1].lower()
        return ext in cls.allowed_extensions

    @classmethod
    @abstractmethod
    def parse(cls, path: str) -> ArticleModel:
        """Parse article from file and store them in ArticleModel object.
        Arguments:
            path {str} -- article file location.
        Returns:
            ArticleModel -- article object.
        """
        pass


class JsonArticleIngestor(ArticleIngestorInterface):
    """Object that ingests JSON file into article model."""

    allowed_extensions: List[str] = ["json"]

    @classmethod
    def parse(cls, path: str) -> ArticleModel:
        """Parse article from file and store them in ArticleModel object.
        Arguments:
            path {str} -- article file location.
        Returns:
            ArticleModel -- article object.
        """
        if not cls.can_ingest(path):
            raise Exception(f"Cannot ingest exception for {path}")
        with open(path, "r", encoding="utf-8-sig") as fh:
            article_json = json.load(fh)
            title = article_json.get("metadata", {}).get("title", "")
            abstract = " ".join(
                [p.get("text", "") for p in article_json.get("abstract", "")]
            )
            conclusion = " ".join(
                [
                    p.get("text", "")
                    for p in article_json.get("body_text", "")
                    if "conclusion" in p.get("section", "").lower()
                ]
            )
            article = ArticleModel(
                paper_id=article_json.get("paper_id", ""),
                title=title.strip(),
                abstract=abstract.strip(),
                conclusion=conclusion.strip(),
            )
        return article


# ## Check Article Ingestor
sample_json_article_path = os.path.join(
    publication_directory,
    "custom_license",
    "custom_license",
    "pdf_json",
    "0a52a3d2793f8ca8a4d6f6630e986ea1da115f80.json",
)
sample_article = JsonArticleIngestor.parse(sample_json_article_path)
sample_article

# System monitoring:
# - RAM usage up to this point: 1 GB
# ## Get Sentences
tic = time.time()
nlp = WE_TEXT_PROCESSING_MODEL.load()
toc = time.time()
print(f"Runtime: {int(toc-tic)} seconds.")

# System monitoring:
# - RAM usage up to this point: 1.3 GB
# Check getting sentences.
text = sample_article.abstract
[str(sentence).strip() for sentence in list(nlp(text).sents)]
text = sample_article.conclusion
[str(sentence).strip() for sentence in list(nlp(text).sents)]


# Get sentences from abstract and conclusion of all articles.
def get_sentences():
    """Get sentences from all articles."""
    paper_ids: List[str] = []
    publication_names: List[str] = []
    sections: List[str] = []
    sentences: List[str] = []
    for publication in publications:
        article_filenames = publication.article_filenames
        for article_filename in article_filenames:
            # Get and parse article
            article_path = os.path.join(
                publication.directory,
                publication.name,
                publication.name,
                "pdf_json",
                article_filename,
            )
            article = JsonArticleIngestor.parse(article_path)
            for section_name in ["abstract", "conclusion"]:
                if section_name == "abstract":
                    text = article.abstract
                elif section_name == "conclusion":
                    text = article.conclusion
                else:
                    continue
                # Get sentences.
                for sentence in list(nlp(text).sents):
                    sentence = str(sentence).strip()
                    # Ignore sentences lesser than three words
                    # (subject, predicate and object).
                    if len(sentence.split()) < 3:
                        continue
                    paper_ids.append(article.paper_id)
                    publication_names.append(publication.name)
                    sections.append(section_name)
                    sentences.append(sentence)
    sentences_df = pd.DataFrame(
        {
            "paper_id": paper_ids,
            "publication_name": publication_names,
            "section": sections,
            "sentence": sentences,
        }
    )
    return sentences_df


tic = time.time()
sentences_df = get_sentences()
toc = time.time()
print(f"Runtime: {int((toc-tic)/60)} minutes.")


# System monitoring:
# - CPU utilization: 100%
# - RAM usage up to this point: 6.4 GB
sentences = sentences_df["sentence"].values.tolist()

# Total sentences.
total_sentences = len(sentences)
total_sentences

# Sentence lengths.
sentence_lengths = [len(sentence.split()) for sentence in sentences]
fig = plt.figure()
ax = fig.add_subplot(111)
ax.boxplot([sentence_lengths])
plt.title("Sentence Lengths")
plt.xlabel("Sentence")
plt.ylabel("# Words")
plt.show()

# Mean value of sentence lengths.
int(np.mean(sentence_lengths))

# Sample of sentences table.
sentences_df.head(10)

# Save sentences table to file.
if not os.path.isdir(TF_WORKING_DIRECTORY):
    print(f"Creating directory... {TF_WORKING_DIRECTORY}")
    os.makedirs(TF_WORKING_DIRECTORY)
sentences_path = os.path.join(TF_WORKING_DIRECTORY, "sentences.csv")
sentences_path
sentences_df.to_csv(sentences_path, index=False)

# System monitoring:
# - Disk usage up to this point: 115 MB
# Remove `sentences_df` to free some memory.
if "sentences_df" in globals():
    del sentences_df
gc.collect()

# System monitoring:
# - RAM usage up to this point: 6.4 GB
# # Build Dataset
sentences_dataset = tf.data.Dataset.from_generator(
    lambda: sentences, (tf.string), (tf.TensorShape([]))
)
for sentence in sentences_dataset.take(10):
    print(f"{sentence}")


# # Create Data Pipeline
# ## Tokenize
def tokenize(words: str) -> List[str]:
    """Tokenize text of words into a list of individual words.
    Arguments:
        words {srt} -- tensor of words.
    Returns:
        List(str) -- list of words.
    """
    return text_to_word_sequence(words)


# ## Generate Words Dictionaries
def load_pretrained_words_dictionaries():
    """Get words dictionaries used by pretrained model."""
    print(f"Load dictionaries...")
    dictionary_path = os.path.join(WE_INPUT_DIRECTORY, "words.txt")
    with open(dictionary_path, "r") as fh:
        sorted_words = fh.readlines()
        sorted_words = [word.strip() for word in sorted_words]
    # Generate dictionaries
    int_to_word = {i + 1: word for i, word in enumerate(sorted_words)}
    int_to_word[0] = "<UNK>"
    word_to_int = {word: i for i, word in int_to_word.items()}
    vocabulary_size = len(int_to_word)
    if vocabulary_size != WE_VOCABULARY_SIZE:
        raise Exception(
            f"Vocabulary size is not the same "
            + f"({vocabulary_size} and {WE_VOCABULARY_SIZE})"
        )
    return int_to_word, word_to_int, vocabulary_size


int_to_word, word_to_int, vocabulary_size = load_pretrained_words_dictionaries()
vocabulary_size
word_to_int["<UNK>"]
word_to_int["the"]
int_to_word[0]
int_to_word[1]
assert int_to_word[word_to_int["the"]] == "the"


# ## Preprocess Dataset
sequence_length = CL_SEQUENCE_LENGTH
sequence_length


def preprocess(text: str) -> List[int]:
    """Pad/turncate and encode text of words from strings into integers.
    Arguments:
        text {str} -- input text.
    Returns:
        List[int] -- features in integers.
    """
    if type(text) is not str:
        # Convert tensor to string
        text = text.numpy().decode("utf-8")
    # Tokenize text
    tokenized_text = tokenize(text)
    # Turncate text
    if len(tokenized_text) >= sequence_length:
        features = tokenized_text[:sequence_length]
    # Pad text
    else:
        features = ["<UNK>"] * (sequence_length - len(tokenized_text))
        features += tokenized_text
    # Encode text
    features_int = [word_to_int.get(feature, 0) for feature in features]
    features_int = np.array(features_int)
    return features_int


features_int = preprocess(text=" ".join(["the"] * (sequence_length - 26)))
features_int.shape
features_int


# Check labeled dataset text smaller than sequence length.
features_int = preprocess(text=" ".join(["the"] * (sequence_length - 26)))
assert len(features_int) == sequence_length
assert Counter(features_int)[1] == sequence_length - 26
assert Counter(features_int)[0] == 26

# Check labeled dataset text equal sequence length.
features_int = preprocess(text=" ".join(["the"] * sequence_length))
assert len(features_int) == sequence_length
assert Counter(features_int)[1] == sequence_length
assert Counter(features_int)[0] == 0

# Check labeled dataset text bigger than sequence length.
features_int = preprocess(text=" ".join(["the"] * (sequence_length + 16)))
assert len(features_int) == sequence_length
assert Counter(features_int)[1] == sequence_length
assert Counter(features_int)[0] == 0

# ## Data Pipeline
num_parallel_calls = CL_NUM_PARALLEL_CALLS
preprocessing_lambda = lambda x: tf.py_function(
    func=preprocess, inp=(x,), Tout=(tf.int32)
)
sentences_features = (
    sentences_dataset.map(preprocessing_lambda, num_parallel_calls=num_parallel_calls)
    .batch(total_sentences)
    .prefetch(buffer_size=num_parallel_calls)
    .cache()
    .repeat()
)
sentences_features_iter = iter(sentences_features)
tic = time.time()
sentences_features_int = sentences_features_iter.get_next()
toc = time.time()
print(f"Runtime: {int((toc-tic)/60)} minutes.")

# Initial runtime is 4 minutes.
# System monitoring:
# - CPU utilization: 165%
# - RAM usage up to this point: 7.1 GB
sentences_features_int.shape
assert sentences_features_int.shape == (total_sentences, sequence_length)


# # Build Models
# Models to build are:
# - Pretrained word embedding model.
# - Pretrained classifier model that uses pretrained word embedding model.
# - Text embedding model is pretrained classifier model up to the bottleneck features without the dense or classifier layers. This model will be used to extract bottleneck features from all sentences in the articles.
# - Text finder model gets the input from questions being asked to produce questions bottleneck features from text embedding model. Then, this model performs cosine similarity of the questions bottleneck features against article sentences bottleneck features.
# ## Load Pretrained Word Embedding Model
class PretrainedWordEmbeddingModel(object):
    """Pretrained word embedding model."""

    def __init__(
        self,
        input_directory: str,
        data_name: str,
        model_architecture_name: str,
        vocabulary_size: int,
        embedding_dim: int,
        total_neg_samples: int,
        learning_rate: float,
        word_batch_size: int,
    ):
        """Initialize pretrained word embedding model.
        Arguments:
            input_directory {str} -- input directory.
            data_name {str} -- input data name.
            model_architecture_name {str} -- name of model architecture.
            vocabulary_size {int} -- vocabulary size.
            embedding_dim {int} -- embedding dimension.
            total_neg_samples {int} -- total negative samples.
            learning_rate {float} -- learning rate.
            word_batch_size {int} -- training batch size.
        """
        self.input_directory = input_directory
        # Dataset
        self.data_name = data_name
        # Model
        self.graph = tf.Graph()
        self.model_architecture_name = model_architecture_name
        self.vocabulary_size = vocabulary_size
        self.embedding_dim = embedding_dim
        self.total_neg_samples = total_neg_samples
        self.learning_rate = learning_rate
        self.word_batch_size = word_batch_size
        self.model_checkpoint_directory = os.path.join(
            self.input_directory, "models/checkpoint"
        )
        self.model_checkpoint_file_name = "%s__%s" % (
            self.model_architecture_name,
            self.data_name,
        )
        self.model_checkpoint_path = os.path.join(
            self.model_checkpoint_directory, self.model_checkpoint_file_name
        )
        self.train_dataset = None
        self.train_labels = None
        self.loss = None
        self.optimizer = None
        # Initial processing
        self.build_model()

    def build_model(self):
        """Build model."""
        with self.graph.as_default():
            # Input data
            self.train_dataset = tf.compat.v1.placeholder(
                shape=(self.word_batch_size), dtype=tf.int32
            )
            self.train_labels = tf.compat.v1.placeholder(
                shape=(self.word_batch_size, 1), dtype=tf.int32
            )
            # Variables
            embeddings = tf.compat.v1.Variable(
                tf.random.uniform([self.vocabulary_size, self.embedding_dim], -1.0, 1.0)
            )
            softmax_weights = tf.compat.v1.Variable(
                tf.random.truncated_normal(
                    [self.vocabulary_size, self.embedding_dim],
                    stddev=1.0 / math.sqrt(self.embedding_dim),
                )
            )
            softmax_biases = tf.compat.v1.Variable(tf.zeros([self.vocabulary_size]))
            # Model
            # Look up embeddings for inputs.
            embed = tf.compat.v1.nn.embedding_lookup(embeddings, self.train_dataset)
            # Compute the softmax loss, using a sample of the negative labels
            # each time.
            self.loss = tf.compat.v1.reduce_mean(
                tf.nn.sampled_softmax_loss(
                    weights=softmax_weights,
                    biases=softmax_biases,
                    inputs=embed,
                    labels=self.train_labels,
                    num_sampled=self.total_neg_samples,
                    num_classes=self.vocabulary_size,
                )
            )
            # Optimizer
            # Note: The optimizer will optimize the softmax_weights AND
            # the embeddings.
            # This is because the embeddings are defined as a variable quantity
            # and the optimizer's `minimize` method will by default modify all
            # variable quantities that contribute to the tensor it is passed.
            # See docs on `tf.train.Optimizer.minimize()` for more details.
            self.optimizer = tf.compat.v1.train.AdagradOptimizer(
                learning_rate=self.learning_rate
            ).minimize(self.loss)

    def get_embedding_variable(self):
        with tf.compat.v1.Session(graph=self.graph) as sess:
            saver = tf.compat.v1.train.Saver()
            saver.restore(sess, self.model_checkpoint_path)
            return sess.graph.get_collection("variables")[0].eval()


word_embedding = PretrainedWordEmbeddingModel(
    input_directory=WE_INPUT_DIRECTORY,
    data_name=WE_DATA_NAME,
    vocabulary_size=WE_VOCABULARY_SIZE,
    embedding_dim=WE_EMBEDDING_DIM,
    total_neg_samples=WE_TOTAL_NEG_SAMPLES,
    learning_rate=WE_LEARNING_RATE,
    model_architecture_name=WE_MODEL_ARCHITECTURE_NAME,
    word_batch_size=WE_WORD_BATCH_SIZE,
)
word_embedding_variable = word_embedding.get_embedding_variable()
type(word_embedding_variable)
word_embedding_variable.shape


# Check if the variable is pretrained. Initialization values are in between -1.0 and 1.0. Any maximum and minimum values beyond initial values indicates pretrained variable.
np.min(word_embedding_variable)
np.max(word_embedding_variable)


# ## Load Pretrained Text Classifier Model
class PretrainedClassifierModel(object):
    """Pretrained classifier model."""

    def __init__(
        self,
        input_directory: str,
        data_name: str,
        model_architecture_name: str,
        pretrained_embedding_flag: bool,
        vocabulary_size: int,
        embedding_dim: int,
        hidden_layer_units: int,
        total_classes: int,
        pretrained_embedding_variable: np.ndarray = None,
    ):
        """Initialize pretrained word embedding.
        Arguments:
            input_directory {str} -- input directory.
            data_name {str} -- input data name.
            model_architecture_name {str} -- name of model architecture.
            pretrained_embedding_flag {bool} -- use pretrained embedding.
            vocabulary_size {int} -- vocabulary size.
            embedding_dim {int} -- embedding dimension.
            hidden_layer_units {int} -- hidden layer units.
            total_classes {int} -- total classes.
            pretrained_embedding_variable {numpy.ndarray} -- pretrained embedding.
        """
        self.input_directory = input_directory
        # Dataset
        self.data_name = data_name
        # Model
        self.model = None
        self.model_architecture_name = model_architecture_name
        self.pretrained_embedding_flag = pretrained_embedding_flag
        self.pretrained_embedding_variable = pretrained_embedding_variable
        self.vocabulary_size = vocabulary_size
        self.embedding_dim = embedding_dim
        self.hidden_layer_units = hidden_layer_units
        self.total_classes = total_classes
        self.model_checkpoint_directory = os.path.join(
            self.input_directory, "models/checkpoint"
        )
        self.model_checkpoint_file_name = "%s__%s.h5" % (
            self.model_architecture_name,
            self.data_name,
        )
        self.model_checkpoint_path = os.path.join(
            self.model_checkpoint_directory, self.model_checkpoint_file_name
        )
        # Initial processing
        self.build_model()

    def build_model(self):
        """Build model."""
        if self.model_architecture_name == "brnn-gru":
            if self.pretrained_embedding_flag:
                self.model = keras.Sequential(
                    [
                        keras.layers.Embedding(
                            self.vocabulary_size,
                            self.embedding_dim,
                            weights=[self.pretrained_embedding_variable],
                            trainable=False,
                        ),
                        keras.layers.Bidirectional(
                            keras.layers.GRU(self.hidden_layer_units)
                        ),
                        keras.layers.Dense(self.hidden_layer_units, activation="relu"),
                        keras.layers.Dense(self.total_classes),
                    ]
                )
            else:
                self.model = keras.Sequential(
                    [
                        keras.layers.Embedding(
                            self.vocabulary_size,
                            self.embedding_dim,
                            embeddings_initializer="uniform",
                        ),
                        keras.layers.Bidirectional(
                            keras.layers.GRU(self.hidden_layer_units)
                        ),
                        keras.layers.Dense(self.hidden_layer_units, activation="relu"),
                        keras.layers.Dense(self.total_classes),
                    ]
                )
        self.model.compile(
            optimizer="adam",
            loss=tf.losses.BinaryCrossentropy(from_logits=True),
            metrics=["accuracy"],
        )

    def load_weights(self):
        self.model.load_weights(self.model_checkpoint_path)


text_classifier = PretrainedClassifierModel(
    input_directory=CL_INPUT_DIRECTORY,
    data_name=CL_DATA_NAME,
    model_architecture_name=CL_MODEL_ARCHITECTURE_NAME,
    pretrained_embedding_flag=CL_PRETRAINED_EMBEDDING_FLAG,
    vocabulary_size=CL_VOCABULARY_SIZE,
    embedding_dim=CL_EMBEDDING_DIM,
    hidden_layer_units=CL_HIDDEN_LAYER_UNITS,
    total_classes=CL_TOTAL_CLASSES,
    pretrained_embedding_variable=word_embedding_variable,
)
text_classifier.model.summary()


# ### Load Pretrained Model
text_classifier.load_weights()

# ## Build Text Embedding Model
text_embedding = keras.Model(
    inputs=text_classifier.model.input,
    outputs=text_classifier.model.get_layer("bidirectional").output,
)
text_embedding.summary()


# ## Extract Bottleneck Features using Text Embedding
def extract_bottleneck_features(model, features_int, batch_size, num_workers):
    bottleneck_features = model.predict(
        x=features_int,
        batch_size=batch_size,
        verbose=1,
        workers=num_workers,
        use_multiprocessing=True,
    )
    return bottleneck_features


bottleneck_features = extract_bottleneck_features(
    model=text_embedding,
    features_int=sentences_features_int,
    batch_size=TE_BATCH_SIZE,
    num_workers=TE_NUM_WORKERS,
)


# Runtime is 53 minutes.
# System monitoring:
# - CPU utilization: 386%
# - RAM usage up to this point: 9.3 GB
# Delete sentences and NLP variables to free some memory.
if "sentences" in globals():
    del sentences
if "sentences_dataset" in globals():
    del sentences_dataset
if "sentences_features" in globals():
    del sentences_features
if "sentences_features_iter" in globals():
    del sentences_features_iter
if "sentences_features_int" in globals():
    del sentences_features_int
if "nlp" in globals():
    del nlp
gc.collect()

# System monitoring:
# - RAM usage up to this point: 8.4 GB
type(bottleneck_features)
bottleneck_features.shape
np.max(bottleneck_features)
np.min(bottleneck_features)

# Normalize bottleneck features for cosine similarity.
normalized_bottleneck_features = (
    bottleneck_features
    / tf.sqrt(tf.reduce_sum(tf.square(bottleneck_features), 1, keepdims=True))
).numpy()
type(normalized_bottleneck_features)
normalized_bottleneck_features.shape
np.min(normalized_bottleneck_features)
np.max(normalized_bottleneck_features)

# System monitoring:
# - RAM usage up to this point: 9.2 GB
# Save normalized bottleneck features to file.
normalized_bottleneck_features_path = os.path.join(
    TF_WORKING_DIRECTORY, "normalized_bottleneck_features.npy"
)
normalized_bottleneck_features_path
normalized_bottleneck_features.save(normalized_bottleneck_features_path)

# ## Build Nearest Text Finder Model
normalized_bottleneck_features_tensor = tf.Variable(
    normalized_bottleneck_features, trainable=False, dtype=tf.float32
)
normalized_bottleneck_features_tensor
normalized_text_embedding_tensor = text_embedding.output / tf.sqrt(
    tf.reduce_sum(tf.square(text_embedding.output), 1, keepdims=True)
)
normalized_text_embedding_tensor
cosine_similarity_tensor = tf.matmul(
    normalized_text_embedding_tensor,
    tf.transpose(normalized_bottleneck_features_tensor),
)
cosine_similarity_tensor
text_finder = keras.Model(inputs=text_embedding.input, outputs=cosine_similarity_tensor)
text_finder.summary()

# System monitoring:
# - RAM usage up to this point: 10.8 GB
# Delete unused models and the variables to free some memory.
if "word_embedding" in globals():
    del word_embedding
if "word_embedding_variable" in globals():
    del word_embedding_variable
if "text_classifier" in globals():
    del text_classifier
if "text_embedding" in globals():
    del text_embedding
if "bottleneck_features" in globals():
    del bottleneck_features
if "normalized_bottleneck_features" in globals():
    del normalized_bottleneck_features
gc.collect()

# System monitoring:
# - RAM usage up to this point: 9 GB
# Save text finder model to file.
model_directory = os.path.join(TF_WORKING_DIRECTORY, "models")
model_directory
if not os.path.isdir(model_directory):
    print(f"Creating directory... {model_directory}")
    os.makedirs(model_directory)
model_checkpoint_directory = os.path.join(model_directory, "checkpoint")
model_checkpoint_directory
if not os.path.isdir(model_checkpoint_directory):
    print(f"Creating directory... {model_checkpoint_directory}")
    os.makedirs(model_checkpoint_directory)
model_architecture_name = TF_MODEL_ARCHITECTURE_NAME
model_architecture_name
data_name = TF_DATA_NAME
data_name
application_name = TF_APPLICATION_NAME
application_name
model_checkpoint_file_name = "%s__%s__%s.h5" % (
    model_architecture_name,
    data_name,
    application_name,
)
model_checkpoint_file_name
model_checkpoint_path = os.path.join(
    model_checkpoint_directory, model_checkpoint_file_name
)
model_checkpoint_path
tic = time.time()
text_finder.save(model_checkpoint_path)
toc = time.time()
print(f"Runtime: {int((toc-tic)/60)} minutes.")


# System monitoring:
# - RAM usage up to this point: 5.1 GB
# - Disk usage up to this point: 700 MB
# # Make Similarity Prediction
def make_similarity_prediction(
    dataset,
    dataset_size,
    preprocessing_lambda,
    model,
    num_parallel_calls,
    batch_size,
    num_workers,
):
    """Make prediction on text finder."""
    # Free some memory
    if "sentences_df" in globals():
        del sentences_df
        gc.collect()
    # Make data pipeline
    features = (
        dataset.map(preprocessing_lambda, num_parallel_calls=num_parallel_calls)
        .batch(dataset_size)
        .prefetch(buffer_size=num_parallel_calls)
    )
    features_iter = iter(features)
    features_int = features_iter.get_next()
    # Make prediction
    predictions = model.predict(
        x=features_int,
        batch_size=batch_size,
        verbose=1,
        workers=num_workers,
        use_multiprocessing=True,
    )
    return predictions


def get_prediction_info(prediction: List[int], idx: int = 0, top_k: int = 10):
    """Get sentences info from prediction index."""
    if "sentences_df" not in globals():
        sentences_df = pd.read_csv(sentences_path)
    nearest = (-prediction[idx]).argsort()[:top_k]
    return sentences_df[sentences_df.index.isin(nearest)]


# ## Initialize Model
# Initializing model may take some time (72 seconds) to initiate the model graph.
questions = [
    "Much",
]
questions_dataset = tf.data.Dataset.from_generator(
    lambda: questions, (tf.string), (tf.TensorShape([]))
)
similarity_prediction = make_similarity_prediction(
    dataset=questions_dataset,
    dataset_size=len(questions),
    preprocessing_lambda=preprocessing_lambda,
    model=text_finder,
    num_parallel_calls=TF_NUM_PARALLEL_CALLS,
    batch_size=TF_BATCH_SIZE,
    num_workers=TF_NUM_WORKERS,
)
type(similarity_prediction)
similarity_prediction.shape
get_prediction_info(prediction=similarity_prediction)

# ## Test using Existing Text in the Articles
questions = [
    "Much progress has been made in understanding the role of structural and accessory proteins in the pathogenesis of severe acute respiratory syndrome coronavirus (SARS-CoV) infections.",
]
questions_dataset = tf.data.Dataset.from_generator(
    lambda: questions, (tf.string), (tf.TensorShape([]))
)
similarity_prediction = make_similarity_prediction(
    dataset=questions_dataset,
    dataset_size=len(questions),
    preprocessing_lambda=preprocessing_lambda,
    model=text_finder,
    num_parallel_calls=TF_NUM_PARALLEL_CALLS,
    batch_size=TF_BATCH_SIZE,
    num_workers=TF_NUM_WORKERS,
)
get_prediction_info(prediction=similarity_prediction)

# ## Find Nearest Sentences to the Questions
questions = [
    "Real-time tracking of whole genomes and a mechanism for coordinating the rapid dissemination of that information to inform the development of diagnostics and therapeutics and to track variations of the virus over time.",
    "Access to geographic and temporal diverse sample sets to understand geographic distribution and genomic differences, and determine whether there is more than one strain in circulation. Multi-lateral agreements such as the Nagoya Protocol could be leveraged.",
    "Evidence that livestock could be infected (e.g., field surveillance, genetic sequencing, receptor binding) and serve as a reservoir after the epidemic appears to be over.",
    "Evidence of whether farmers are infected, and whether farmers could have played a role in the origin.",
    "Surveillance of mixed wildlife-livestock farms for SARS-CoV-2 and other coronaviruses in Southeast Asia.",
    "Experimental infections to test host range for this pathogen.",
    "Animal host(s) and any evidence of continued spill-over to humans.",
    "Socioeconomic and behavioral risk factors for this spill-over.",
    "Sustainable risk reduction strategies.",
]
questions_dataset = tf.data.Dataset.from_generator(
    lambda: questions, (tf.string), (tf.TensorShape([]))
)
similarity_prediction = make_similarity_prediction(
    dataset=questions_dataset,
    dataset_size=len(questions),
    preprocessing_lambda=preprocessing_lambda,
    model=text_finder,
    num_parallel_calls=TF_NUM_PARALLEL_CALLS,
    batch_size=TF_BATCH_SIZE,
    num_workers=TF_NUM_WORKERS,
)

# # Q1
# #### Real-time tracking of whole genomes and a mechanism for coordinating the rapid dissemination of that information to inform the development of diagnostics and therapeutics and to track variations of the virus over time.
get_prediction_info(prediction=similarity_prediction, idx=0)

# # Q2
# #### Access to geographic and temporal diverse sample sets to understand geographic distribution and genomic differences, and determine whether there is more than one strain in circulation. Multi-lateral agreements such as the Nagoya Protocol could be leveraged.
get_prediction_info(prediction=similarity_prediction, idx=1)

# # Q3
# #### Evidence that livestock could be infected (e.g., field surveillance, genetic sequencing, receptor binding) and serve as a reservoir after the epidemic appears to be over.
get_prediction_info(prediction=similarity_prediction, idx=2)

# # Q3a
# #### Evidence of whether farmers are infected, and whether farmers could have played a role in the origin.
get_prediction_info(prediction=similarity_prediction, idx=3)

# # Q3b
# #### Surveillance of mixed wildlife-livestock farms for SARS-CoV-2 and other coronaviruses in Southeast Asia.
get_prediction_info(prediction=similarity_prediction, idx=4)

# # Q3c
# #### Experimental infections to test host range for this pathogen.
get_prediction_info(prediction=similarity_prediction, idx=5)

# # Q4
# #### Animal host(s) and any evidence of continued spill-over to humans.
get_prediction_info(prediction=similarity_prediction, idx=6)

# # Q5
# #### Socioeconomic and behavioral risk factors for this spill-over.
get_prediction_info(prediction=similarity_prediction, idx=7)

# # Q6
# #### Sustainable risk reduction strategies.
get_prediction_info(prediction=similarity_prediction, idx=8)
<|endoftext|>
[{"Id": 32195719, "ScriptId": 8850712, "ParentScriptVersionId": NaN, "ScriptLanguageId": 9, "AuthorUserId": 273193, "CreationDate": "04/17/2020 18:31:59", "VersionNumber": 8.0, "Title": "CORD-19: Abstract and Conclusion Text Embedding", "EvaluationDate": "04/17/2020", "IsChange": true, "TotalLines": 1268.0, "LinesInsertedFromPrevious": 10.0, "LinesChangedFromPrevious": 0.0, "LinesUnchangedFromPrevious": 1258.0, "LinesInsertedFromFork": NaN, "LinesDeletedFromFork": NaN, "LinesChangedFromFork": NaN, "LinesUnchangedFromFork": NaN, "TotalVotes": 0}]
[{"Id": 34837114, "KernelVersionId": 32195719, "SourceDatasetVersionId": 1073914}, {"Id": 34837113, "KernelVersionId": 32195719, "SourceDatasetVersionId": 1071857}, {"Id": 34837112, "KernelVersionId": 32195719, "SourceDatasetVersionId": 1063983}]
[{"Id": 1073914, "DatasetId": 551982, "DatasourceVersionId": 1103617, "CreatorUserId": 1314380, "LicenseName": "Other (specified in description)", "CreationDate": "04/11/2020 13:32:24", "VersionNumber": 7.0, "Title": "COVID-19 Open Research Dataset Challenge (CORD-19)", "Slug": "CORD-19-research-challenge", "Subtitle": "An AI challenge with AI2, CZI, MSR, Georgetown, NIH & The White House", "Description": "### Dataset Description\n\nIn response to the COVID-19 pandemic, the White House and a coalition of leading research groups have prepared the COVID-19 Open Research Dataset (CORD-19). CORD-19 is a resource of over 51,000 scholarly articles, including over 40,000 with full text, about COVID-19, SARS-CoV-2, and related coronaviruses. This freely available dataset is provided to the global research community to apply recent advances in natural language processing and other AI techniques to generate new insights in support of the ongoing fight against this infectious disease. There is a growing urgency for these approaches because of the rapid acceleration in new coronavirus literature, making it difficult for the medical research community to keep up.\n\n### Call to Action\n\nWe are issuing a call to action to the world's artificial intelligence experts to develop text and data mining tools that can help the medical community develop answers to high priority scientific questions.  The CORD-19 dataset represents the most extensive machine-readable coronavirus literature collection available for data mining to date. This allows the worldwide AI research community the opportunity to apply text and data mining approaches to find answers to questions within, and connect insights across, this content in support of the ongoing COVID-19 response efforts worldwide. There is a growing urgency for these approaches because of the rapid increase in coronavirus literature, making it difficult for the medical community to keep up.\n\nA list of our initial key questions can be found under the **[Tasks](https://www.kaggle.com/allen-institute-for-ai/CORD-19-research-challenge/tasks)** section of this dataset. These key scientific questions are drawn from the NASEM\u2019s SCIED (National Academies of Sciences, Engineering, and Medicine\u2019s Standing Committee on Emerging Infectious Diseases and 21st Century Health Threats) [research topics](https://www.nationalacademies.org/event/03-11-2020/standing-committee-on-emerging-infectious-diseases-and-21st-century-health-threats-virtual-meeting-1) and the World Health Organization\u2019s [R&D Blueprint](https://www.who.int/blueprint/priority-diseases/key-action/Global_Research_Forum_FINAL_VERSION_for_web_14_feb_2020.pdf?ua=1) for COVID-19.  \n\nMany of these questions are suitable for text mining, and we encourage researchers to develop text mining tools to provide insights on these questions. \n\nWe are maintaining a summary of the [community's contributions](https://www.kaggle.com/covid-19-contributions). For guidance on how to make your contributions useful, we're maintaining a [forum thread](https://www.kaggle.com/allen-institute-for-ai/CORD-19-research-challenge/discussion/138484) with the feedback we're getting from the medical and health policy communities. \n\n### Prizes\n\nKaggle is sponsoring a *$1,000 per task* award to the winner whose submission is identified as best meeting the evaluation criteria. The winner may elect to receive this award as a charitable donation to COVID-19 relief/research efforts or as a monetary payment. More details on the prizes and timeline can be found on the [discussion post](https://www.kaggle.com/allen-institute-for-ai/CORD-19-research-challenge/discussion/135826).\n\n### Accessing the Dataset\n\nWe have made this dataset available on Kaggle. Watch out for [periodic updates](https://www.kaggle.com/allen-institute-for-ai/CORD-19-research-challenge/discussion/137474). \n\nThe dataset is also hosted on [AI2's Semantic Scholar](https://pages.semanticscholar.org/coronavirus-research). And you can search the dataset using AI2's new [COVID-19 explorer](https://cord-19.apps.allenai.org/).\n\nThe licenses for each dataset can be found in the all _ sources _ metadata csv file.\n\n### Acknowledgements\n\n![](https://www.kaggle.com/static/images/covid-19.png)\n\nThis dataset was created by the Allen Institute for AI in partnership  with the Chan Zuckerberg Initiative, Georgetown University\u2019s Center for Security and Emerging Technology, Microsoft Research, and the National Library of Medicine - National Institutes of Health, in coordination with The White House Office of Science and Technology Policy.", "VersionNotes": "2020-04-10", "TotalCompressedBytes": 0.0, "TotalUncompressedBytes": 0.0}]
[{"Id": 551982, "CreatorUserId": 2931338, "OwnerUserId": NaN, "OwnerOrganizationId": 3737.0, "CurrentDatasetVersionId": 3756201.0, "CurrentDatasourceVersionId": 3810704.0, "ForumId": 565591, "Type": 2, "CreationDate": "03/12/2020 20:05:08", "LastActivityDate": "03/12/2020", "TotalViews": 4468011, "TotalDownloads": 163921, "TotalVotes": 10518, "TotalKernels": 1717}]
# # Introduction
# This project uses pretrained text classifier model to get the embedding in order to find the nearest texts (or sentences) in a collection of articles. The texts to find are from the following questions.
# - Real-time tracking of whole genomes and a mechanism for coordinating the rapid dissemination of that information to inform the development of diagnostics and therapeutics and to track variations of the virus over time. [Q1](#Q1)
# - Access to geographic and temporal diverse sample sets to understand geographic distribution and genomic differences, and determine whether there is more than one strain in circulation. Multi-lateral agreements such as the Nagoya Protocol could be leveraged. [Q2](#Q2)
# - Evidence that livestock could be infected (e.g., field surveillance, genetic sequencing, receptor binding) and serve as a reservoir after the epidemic appears to be over. [Q3](#Q3)
# - Evidence of whether farmers are infected, and whether farmers could have played a role in the origin. [Q3a](#Q3a)
# - Surveillance of mixed wildlife-livestock farms for SARS-CoV-2 and other coronaviruses in Southeast Asia. [Q3b](#Q3b)
# - Experimental infections to test host range for this pathogen. [Q3c](#Q3c)
# - Animal host(s) and any evidence of continued spill-over to humans. [Q4](#Q4)
# - Socioeconomic and behavioral risk factors for this spill-over. [Q5](#Q5)
# - Sustainable risk reduction strategies. [Q6](#Q6)
# This project uses models and datasets from the following works.
# - [CORD-19: Abstract and Conclusion Word Embedding (version 17)](https://www.kaggle.com/ekaakurniawan/cord-19-abstract-and-conclusion-word-embedding?scriptVersionId=31434959) or [`skip-gram__CORD-19` dataset (version 1)](https://www.kaggle.com/ekaakurniawan/skipgram-cord19-v17/version/1).
# - [CORD-19: Abstract and Conclusion Classification (version 20)](https://www.kaggle.com/ekaakurniawan/cord-19-abstract-and-conclusion-classification?scriptVersionId=31732781) or [`brnn-gru__CORD-19` dataset (version 2)](https://www.kaggle.com/ekaakurniawan/brnngru-cord19-v19/version/2).
# Instead of using the full classification architecture, the text embedding architecture will extract text features from bottleneck layer without the fully connected layers.
# # Software Versions
import sys

print("Python %s" % sys.version)
import os
import time
import math
import random
import gc
from pathlib import Path
from abc import ABC, abstractmethod
from typing import List, Dict
from collections import Counter
import json
from pprint import pprint  # for debugging
import numpy as np

print("NumPy %s" % np.__version__)
import matplotlib

print("matplotlib %s" % matplotlib.__version__)
import matplotlib.pyplot as plt

# Set plotting options
import tensorflow as tf

print("TensorFlow %s" % tf.__version__)
from tensorflow.keras.preprocessing.text import text_to_word_sequence
from tensorflow import keras
from tensorflow.keras import layers
import pandas as pd

print("Pandas %s" % pd.__version__)
pd.set_option("display.max_colwidth", 400)
import spacy

print("spaCy %s" % spacy.__version__)

# Get ScispaCy trained model for processing biomedical, scientific or clinical text.`[1]`
import en_core_sci_md

print("en_core_sci_md %s" % en_core_sci_md.__version__)

# # Hyperparameters
# ## Word Embedding Hyperparameters
# Taken from [CORD-19: Abstract and Conclusion Word Embedding](https://www.kaggle.com/ekaakurniawan/cord-19-abstract-and-conclusion-word-embedding).
# Data
WE_DATA_NAME = "CORD-19"
WE_INPUT_DIRECTORY = "/kaggle/input/skipgram-cord19-v17/embedding"
WE_WORKING_DIRECTORY = "/kaggle/working/embedding"
# Preprocessing
WE_TEXT_PROCESSING_MODEL = en_core_sci_md
# Training
WE_MODEL_ARCHITECTURE_NAME = "skip-gram"
WE_VOCABULARY_SIZE = 147843 + 1  # Vocabulary size + UNK
WE_EMBEDDING_DIM = 256
WE_WORD_BATCH_SIZE = 128  # How many words in one batch
WE_TOTAL_NEG_SAMPLES = 64  # Number of negative examples to sample
WE_LEARNING_RATE = 1.0
WE_EPOCHS = 150
WE_REPORT_EVERY = 50000
# Validation
WE_VALIDATION_SIZE = 16  # Random set of words to evaluate similarity on
WE_VALIDATION_WINDOW = 100  # Only pick dev samples in the head of the distribution

# ## Classification Hyperparameters
# Taken from [CORD-19: Abstract and Conclusion Classification](https://www.kaggle.com/ekaakurniawan/cord-19-abstract-and-conclusion-classification).
# Data
CL_DATA_NAME = "CORD-19"
CL_INPUT_DIRECTORY = "/kaggle/input/brnngru-cord19-v19/classification"
CL_WORKING_DIRECTORY = "/kaggle/working/classification"
# Preprocessing
CL_SEQUENCE_LENGTH = 400  # Feature fixed sequence length
CL_NUM_PARALLEL_CALLS = 4
# Training
CL_MODEL_ARCHITECTURES = [
    "brnn-gru",  # Bidirectional Recurrent Neural Networks using GRU
]
CL_MODEL_ARCHITECTURE_NAME = "brnn-gru"
CL_PRETRAINED_EMBEDDING_FLAG = True  # Use pretrained embedding
CL_VOCABULARY_SIZE = 147843 + 1  # Vocabulary size + UNK
CL_EMBEDDING_DIM = 256  # Embedding dimension
CL_HIDDEN_LAYER_UNITS = 256
CL_TOTAL_CLASSES = 6  # Total classes
CL_BATCH_SIZE = 128  # Bach size
CL_EPOCHS = 60  # Number of epochs
CL_NUM_WORKERS = 4

# ## Text Embedding Hyperparameters
# Data
TE_DATA_NAME = "CORD-19"
TE_INPUT_DIRECTORY = "/kaggle/input"
TE_WORKING_DIRECTORY = "/kaggle/working/text_embedding"
# Inference
TE_BATCH_SIZE = 128  # How many texts in one batch
TE_NUM_WORKERS = 4

# ## Text Finder Hyperparameters
# Application
TF_APPLICATION_NAME = "text-finder"
# Data
TF_DATA_NAME = "CORD-19"
TF_INPUT_DIRECTORY = "/kaggle/input"
TF_WORKING_DIRECTORY = "/kaggle/working/text_finder"
# Preprocessing
TF_NUM_PARALLEL_CALLS = 4
# Model
TF_MODEL_ARCHITECTURE_NAME = "brnn-gru"
# Inference
TF_BATCH_SIZE = 1  # How many texts in one batch
TF_NUM_WORKERS = 4

# # Analyze Dataset
# Dataset source is from [COVID-19 Open Research Dataset (CORD-19)](https://pages.semanticscholar.org/coronavirus-research).`[2]` Check the website for new updates.
# List all directories in the dataset.
for dirname, _, _ in os.walk(TE_INPUT_DIRECTORY):
    print(dirname)
publication_directory = os.path.join(TE_INPUT_DIRECTORY, "CORD-19-research-challenge")
publication_directory
publication_names = [
    "custom_license",
    "biorxiv_medrxiv",
    "comm_use_subset",
    "noncomm_use_subset",
]


# ## Create Publication Model
class PublicationModel(object):
    """Publication class to store and process publication information."""

    def __init__(self, name: str, directory: str):
        """Initialize publication object.
        Arguments:
            name {str} -- the name of the publication.
            directory {str} -- the directory of the publication.
        """
        self.name = name
        self.directory = directory
        self.article_filenames: List[str] = []

    def collect_articles(self):
        """Collect all article filenames in publication directory."""
        self.article_filenames = os.listdir(
            os.path.join(self.directory, self.name, self.name, "pdf_json")
        )

    def __repr__(self):
        """Return publication short information."""
        return f"{self.name} with {len(self.article_filenames)} articles."


publications: List[PublicationModel] = []
for publication_name in publication_names:
    publication = PublicationModel(publication_name, publication_directory)
    publication.collect_articles()
    print(publication.__repr__())
    publications.append(publication)
total_articles = 0
for publication in publications:
    total_articles += len(publication.article_filenames)
print("total articles:", total_articles)


# ## Create Article Model
class ArticleModel(object):
    """Article class to store the entire article information."""

    def __init__(self, paper_id: str, title: str, abstract: str, conclusion: str):
        """Initialize article object.
        Arguments:
            paper_id {str} -- 40-character sha1 of the PDF.
            title {str} -- the title of the article.
            abstract {str} -- the text abstract of the article.
            conclusion {str} -- the text conclusion of the article.
        """
        self.paper_id = paper_id
        self.title = title
        self.abstract = abstract
        self.conclusion = conclusion

    def __repr__(self):
        """Return article in readable string."""
        return (
            f"{self.paper_id}\n{self.title}\n\n"
            + f"Abstract:\n{self.abstract}\n\n"
            + f"Conclusion:\n{self.conclusion}\n"
        )


# ## Create Article Ingestor
class ArticleIngestorInterface(ABC):
    """Abstract class for ingestor classes."""

    allowed_extensions: List[str] = []

    @classmethod
    def can_ingest(cls, path: str) -> bool:
        """Check if file extension is supported for ingestion.
        Arguments:
            path {str} -- article file location.
        Returns:
            bool -- True if the extension is supported and False otherwise.
        """
        ext = path.split(".")[-1].lower()
        return ext in cls.allowed_extensions

    @classmethod
    @abstractmethod
    def parse(cls, path: str) -> ArticleModel:
        """Parse article from file and store them in ArticleModel object.
        Arguments:
            path {str} -- article file location.
        Returns:
            ArticleModel -- article object.
        """
        pass


class JsonArticleIngestor(ArticleIngestorInterface):
    """Object that ingests JSON file into article model."""

    allowed_extensions: List[str] = ["json"]

    @classmethod
    def parse(cls, path: str) -> ArticleModel:
        """Parse article from file and store them in ArticleModel object.
        Arguments:
            path {str} -- article file location.
        Returns:
            ArticleModel -- article object.
        """
        if not cls.can_ingest(path):
            raise Exception(f"Cannot ingest exception for {path}")
        with open(path, "r", encoding="utf-8-sig") as fh:
            article_json = json.load(fh)
            title = article_json.get("metadata", {}).get("title", "")
            abstract = " ".join(
                [p.get("text", "") for p in article_json.get("abstract", "")]
            )
            conclusion = " ".join(
                [
                    p.get("text", "")
                    for p in article_json.get("body_text", "")
                    if "conclusion" in p.get("section", "").lower()
                ]
            )
            article = ArticleModel(
                paper_id=article_json.get("paper_id", ""),
                title=title.strip(),
                abstract=abstract.strip(),
                conclusion=conclusion.strip(),
            )
        return article


# ## Check Article Ingestor
sample_json_article_path = os.path.join(
    publication_directory,
    "custom_license",
    "custom_license",
    "pdf_json",
    "0a52a3d2793f8ca8a4d6f6630e986ea1da115f80.json",
)
sample_article = JsonArticleIngestor.parse(sample_json_article_path)
sample_article

# System monitoring:
# - RAM usage up to this point: 1 GB
# ## Get Sentences
tic = time.time()
nlp = WE_TEXT_PROCESSING_MODEL.load()
toc = time.time()
print(f"Runtime: {int(toc-tic)} seconds.")

# System monitoring:
# - RAM usage up to this point: 1.3 GB
# Check getting sentences.
text = sample_article.abstract
[str(sentence).strip() for sentence in list(nlp(text).sents)]
text = sample_article.conclusion
[str(sentence).strip() for sentence in list(nlp(text).sents)]


# Get sentences from abstract and conclusion of all articles.
def get_sentences():
    """Get sentences from all articles."""
    paper_ids: List[str] = []
    publication_names: List[str] = []
    sections: List[str] = []
    sentences: List[str] = []
    for publication in publications:
        article_filenames = publication.article_filenames
        for article_filename in article_filenames:
            # Get and parse article
            article_path = os.path.join(
                publication.directory,
                publication.name,
                publication.name,
                "pdf_json",
                article_filename,
            )
            article = JsonArticleIngestor.parse(article_path)
            for section_name in ["abstract", "conclusion"]:
                if section_name == "abstract":
                    text = article.abstract
                elif section_name == "conclusion":
                    text = article.conclusion
                else:
                    continue
                # Get sentences.
                for sentence in list(nlp(text).sents):
                    sentence = str(sentence).strip()
                    # Ignore sentences lesser than three words
                    # (subject, predicate and object).
                    if len(sentence.split()) < 3:
                        continue
                    paper_ids.append(article.paper_id)
                    publication_names.append(publication.name)
                    sections.append(section_name)
                    sentences.append(sentence)
    sentences_df = pd.DataFrame(
        {
            "paper_id": paper_ids,
            "publication_name": publication_names,
            "section": sections,
            "sentence": sentences,
        }
    )
    return sentences_df


tic = time.time()
sentences_df = get_sentences()
toc = time.time()
print(f"Runtime: {int((toc-tic)/60)} minutes.")


# System monitoring:
# - CPU utilization: 100%
# - RAM usage up to this point: 6.4 GB
sentences = sentences_df["sentence"].values.tolist()

# Total sentences.
total_sentences = len(sentences)
total_sentences

# Sentence lengths.
sentence_lengths = [len(sentence.split()) for sentence in sentences]
fig = plt.figure()
ax = fig.add_subplot(111)
ax.boxplot([sentence_lengths])
plt.title("Sentence Lengths")
plt.xlabel("Sentence")
plt.ylabel("# Words")
plt.show()

# Mean value of sentence lengths.
int(np.mean(sentence_lengths))

# Sample of sentences table.
sentences_df.head(10)

# Save sentences table to file.
if not os.path.isdir(TF_WORKING_DIRECTORY):
    print(f"Creating directory... {TF_WORKING_DIRECTORY}")
    os.makedirs(TF_WORKING_DIRECTORY)
sentences_path = os.path.join(TF_WORKING_DIRECTORY, "sentences.csv")
sentences_path
sentences_df.to_csv(sentences_path, index=False)

# System monitoring:
# - Disk usage up to this point: 115 MB
# Remove `sentences_df` to free some memory.
if "sentences_df" in globals():
    del sentences_df
gc.collect()

# System monitoring:
# - RAM usage up to this point: 6.4 GB
# # Build Dataset
sentences_dataset = tf.data.Dataset.from_generator(
    lambda: sentences, (tf.string), (tf.TensorShape([]))
)
for sentence in sentences_dataset.take(10):
    print(f"{sentence}")


# # Create Data Pipeline
# ## Tokenize
def tokenize(words: str) -> List[str]:
    """Tokenize text of words into a list of individual words.
    Arguments:
        words {srt} -- tensor of words.
    Returns:
        List(str) -- list of words.
    """
    return text_to_word_sequence(words)


# ## Generate Words Dictionaries
def load_pretrained_words_dictionaries():
    """Get words dictionaries used by pretrained model."""
    print(f"Load dictionaries...")
    dictionary_path = os.path.join(WE_INPUT_DIRECTORY, "words.txt")
    with open(dictionary_path, "r") as fh:
        sorted_words = fh.readlines()
        sorted_words = [word.strip() for word in sorted_words]
    # Generate dictionaries
    int_to_word = {i + 1: word for i, word in enumerate(sorted_words)}
    int_to_word[0] = "<UNK>"
    word_to_int = {word: i for i, word in int_to_word.items()}
    vocabulary_size = len(int_to_word)
    if vocabulary_size != WE_VOCABULARY_SIZE:
        raise Exception(
            f"Vocabulary size is not the same "
            + f"({vocabulary_size} and {WE_VOCABULARY_SIZE})"
        )
    return int_to_word, word_to_int, vocabulary_size


int_to_word, word_to_int, vocabulary_size = load_pretrained_words_dictionaries()
vocabulary_size
word_to_int["<UNK>"]
word_to_int["the"]
int_to_word[0]
int_to_word[1]
assert int_to_word[word_to_int["the"]] == "the"


# ## Preprocess Dataset
sequence_length = CL_SEQUENCE_LENGTH
sequence_length


def preprocess(text: str) -> List[int]:
    """Pad/turncate and encode text of words from strings into integers.
    Arguments:
        text {str} -- input text.
    Returns:
        List[int] -- features in integers.
    """
    if type(text) is not str:
        # Convert tensor to string
        text = text.numpy().decode("utf-8")
    # Tokenize text
    tokenized_text = tokenize(text)
    # Turncate text
    if len(tokenized_text) >= sequence_length:
        features = tokenized_text[:sequence_length]
    # Pad text
    else:
        features = ["<UNK>"] * (sequence_length - len(tokenized_text))
        features += tokenized_text
    # Encode text
    features_int = [word_to_int.get(feature, 0) for feature in features]
    features_int = np.array(features_int)
    return features_int


features_int = preprocess(text=" ".join(["the"] * (sequence_length - 26)))
features_int.shape
features_int


# Check labeled dataset text smaller than sequence length.
features_int = preprocess(text=" ".join(["the"] * (sequence_length - 26)))
assert len(features_int) == sequence_length
assert Counter(features_int)[1] == sequence_length - 26
assert Counter(features_int)[0] == 26

# Check labeled dataset text equal sequence length.
features_int = preprocess(text=" ".join(["the"] * sequence_length))
assert len(features_int) == sequence_length
assert Counter(features_int)[1] == sequence_length
assert Counter(features_int)[0] == 0

# Check labeled dataset text bigger than sequence length.
features_int = preprocess(text=" ".join(["the"] * (sequence_length + 16)))
assert len(features_int) == sequence_length
assert Counter(features_int)[1] == sequence_length
assert Counter(features_int)[0] == 0

# ## Data Pipeline
num_parallel_calls = CL_NUM_PARALLEL_CALLS
preprocessing_lambda = lambda x: tf.py_function(
    func=preprocess, inp=(x,), Tout=(tf.int32)
)
sentences_features = (
    sentences_dataset.map(preprocessing_lambda, num_parallel_calls=num_parallel_calls)
    .batch(total_sentences)
    .prefetch(buffer_size=num_parallel_calls)
    .cache()
    .repeat()
)
sentences_features_iter = iter(sentences_features)
tic = time.time()
sentences_features_int = sentences_features_iter.get_next()
toc = time.time()
print(f"Runtime: {int((toc-tic)/60)} minutes.")

# Initial runtime is 4 minutes.
# System monitoring:
# - CPU utilization: 165%
# - RAM usage up to this point: 7.1 GB
sentences_features_int.shape
assert sentences_features_int.shape == (total_sentences, sequence_length)


# # Build Models
# Models to build are:
# - Pretrained word embedding model.
# - Pretrained classifier model that uses pretrained word embedding model.
# - Text embedding model is pretrained classifier model up to the bottleneck features without the dense or classifier layers. This model will be used to extract bottleneck features from all sentences in the articles.
# - Text finder model gets the input from questions being asked to produce questions bottleneck features from text embedding model. Then, this model performs cosine similarity of the questions bottleneck features against article sentences bottleneck features.
# ## Load Pretrained Word Embedding Model
class PretrainedWordEmbeddingModel(object):
    """Pretrained word embedding model."""

    def __init__(
        self,
        input_directory: str,
        data_name: str,
        model_architecture_name: str,
        vocabulary_size: int,
        embedding_dim: int,
        total_neg_samples: int,
        learning_rate: float,
        word_batch_size: int,
    ):
        """Initialize pretrained word embedding model.
        Arguments:
            input_directory {str} -- input directory.
            data_name {str} -- input data name.
            model_architecture_name {str} -- name of model architecture.
            vocabulary_size {int} -- vocabulary size.
            embedding_dim {int} -- embedding dimension.
            total_neg_samples {int} -- total negative samples.
            learning_rate {float} -- learning rate.
            word_batch_size {int} -- training batch size.
        """
        self.input_directory = input_directory
        # Dataset
        self.data_name = data_name
        # Model
        self.graph = tf.Graph()
        self.model_architecture_name = model_architecture_name
        self.vocabulary_size = vocabulary_size
        self.embedding_dim = embedding_dim
        self.total_neg_samples = total_neg_samples
        self.learning_rate = learning_rate
        self.word_batch_size = word_batch_size
        self.model_checkpoint_directory = os.path.join(
            self.input_directory, "models/checkpoint"
        )
        self.model_checkpoint_file_name = "%s__%s" % (
            self.model_architecture_name,
            self.data_name,
        )
        self.model_checkpoint_path = os.path.join(
            self.model_checkpoint_directory, self.model_checkpoint_file_name
        )
        self.train_dataset = None
        self.train_labels = None
        self.loss = None
        self.optimizer = None
        # Initial processing
        self.build_model()

    def build_model(self):
        """Build model."""
        with self.graph.as_default():
            # Input data
            self.train_dataset = tf.compat.v1.placeholder(
                shape=(self.word_batch_size), dtype=tf.int32
            )
            self.train_labels = tf.compat.v1.placeholder(
                shape=(self.word_batch_size, 1), dtype=tf.int32
            )
            # Variables
            embeddings = tf.compat.v1.Variable(
                tf.random.uniform([self.vocabulary_size, self.embedding_dim], -1.0, 1.0)
            )
            softmax_weights = tf.compat.v1.Variable(
                tf.random.truncated_normal(
                    [self.vocabulary_size, self.embedding_dim],
                    stddev=1.0 / math.sqrt(self.embedding_dim),
                )
            )
            softmax_biases = tf.compat.v1.Variable(tf.zeros([self.vocabulary_size]))
            # Model
            # Look up embeddings for inputs.
            embed = tf.compat.v1.nn.embedding_lookup(embeddings, self.train_dataset)
            # Compute the softmax loss, using a sample of the negative labels
            # each time.
            self.loss = tf.compat.v1.reduce_mean(
                tf.nn.sampled_softmax_loss(
                    weights=softmax_weights,
                    biases=softmax_biases,
                    inputs=embed,
                    labels=self.train_labels,
                    num_sampled=self.total_neg_samples,
                    num_classes=self.vocabulary_size,
                )
            )
            # Optimizer
            # Note: The optimizer will optimize the softmax_weights AND
            # the embeddings.
            # This is because the embeddings are defined as a variable quantity
            # and the optimizer's `minimize` method will by default modify all
            # variable quantities that contribute to the tensor it is passed.
            # See docs on `tf.train.Optimizer.minimize()` for more details.
            self.optimizer = tf.compat.v1.train.AdagradOptimizer(
                learning_rate=self.learning_rate
            ).minimize(self.loss)

    def get_embedding_variable(self):
        with tf.compat.v1.Session(graph=self.graph) as sess:
            saver = tf.compat.v1.train.Saver()
            saver.restore(sess, self.model_checkpoint_path)
            return sess.graph.get_collection("variables")[0].eval()


word_embedding = PretrainedWordEmbeddingModel(
    input_directory=WE_INPUT_DIRECTORY,
    data_name=WE_DATA_NAME,
    vocabulary_size=WE_VOCABULARY_SIZE,
    embedding_dim=WE_EMBEDDING_DIM,
    total_neg_samples=WE_TOTAL_NEG_SAMPLES,
    learning_rate=WE_LEARNING_RATE,
    model_architecture_name=WE_MODEL_ARCHITECTURE_NAME,
    word_batch_size=WE_WORD_BATCH_SIZE,
)
word_embedding_variable = word_embedding.get_embedding_variable()
type(word_embedding_variable)
word_embedding_variable.shape


# Check if the variable is pretrained. Initialization values are in between -1.0 and 1.0. Any maximum and minimum values beyond initial values indicates pretrained variable.
np.min(word_embedding_variable)
np.max(word_embedding_variable)


# ## Load Pretrained Text Classifier Model
class PretrainedClassifierModel(object):
    """Pretrained classifier model."""

    def __init__(
        self,
        input_directory: str,
        data_name: str,
        model_architecture_name: str,
        pretrained_embedding_flag: bool,
        vocabulary_size: int,
        embedding_dim: int,
        hidden_layer_units: int,
        total_classes: int,
        pretrained_embedding_variable: np.ndarray = None,
    ):
        """Initialize pretrained word embedding.
        Arguments:
            input_directory {str} -- input directory.
            data_name {str} -- input data name.
            model_architecture_name {str} -- name of model architecture.
            pretrained_embedding_flag {bool} -- use pretrained embedding.
            vocabulary_size {int} -- vocabulary size.
            embedding_dim {int} -- embedding dimension.
            hidden_layer_units {int} -- hidden layer units.
            total_classes {int} -- total classes.
            pretrained_embedding_variable {numpy.ndarray} -- pretrained embedding.
        """
        self.input_directory = input_directory
        # Dataset
        self.data_name = data_name
        # Model
        self.model = None
        self.model_architecture_name = model_architecture_name
        self.pretrained_embedding_flag = pretrained_embedding_flag
        self.pretrained_embedding_variable = pretrained_embedding_variable
        self.vocabulary_size = vocabulary_size
        self.embedding_dim = embedding_dim
        self.hidden_layer_units = hidden_layer_units
        self.total_classes = total_classes
        self.model_checkpoint_directory = os.path.join(
            self.input_directory, "models/checkpoint"
        )
        self.model_checkpoint_file_name = "%s__%s.h5" % (
            self.model_architecture_name,
            self.data_name,
        )
        self.model_checkpoint_path = os.path.join(
            self.model_checkpoint_directory, self.model_checkpoint_file_name
        )
        # Initial processing
        self.build_model()

    def build_model(self):
        """Build model."""
        if self.model_architecture_name == "brnn-gru":
            if self.pretrained_embedding_flag:
                self.model = keras.Sequential(
                    [
                        keras.layers.Embedding(
                            self.vocabulary_size,
                            self.embedding_dim,
                            weights=[self.pretrained_embedding_variable],
                            trainable=False,
                        ),
                        keras.layers.Bidirectional(
                            keras.layers.GRU(self.hidden_layer_units)
                        ),
                        keras.layers.Dense(self.hidden_layer_units, activation="relu"),
                        keras.layers.Dense(self.total_classes),
                    ]
                )
            else:
                self.model = keras.Sequential(
                    [
                        keras.layers.Embedding(
                            self.vocabulary_size,
                            self.embedding_dim,
                            embeddings_initializer="uniform",
                        ),
                        keras.layers.Bidirectional(
                            keras.layers.GRU(self.hidden_layer_units)
                        ),
                        keras.layers.Dense(self.hidden_layer_units, activation="relu"),
                        keras.layers.Dense(self.total_classes),
                    ]
                )
        self.model.compile(
            optimizer="adam",
            loss=tf.losses.BinaryCrossentropy(from_logits=True),
            metrics=["accuracy"],
        )

    def load_weights(self):
        self.model.load_weights(self.model_checkpoint_path)


text_classifier = PretrainedClassifierModel(
    input_directory=CL_INPUT_DIRECTORY,
    data_name=CL_DATA_NAME,
    model_architecture_name=CL_MODEL_ARCHITECTURE_NAME,
    pretrained_embedding_flag=CL_PRETRAINED_EMBEDDING_FLAG,
    vocabulary_size=CL_VOCABULARY_SIZE,
    embedding_dim=CL_EMBEDDING_DIM,
    hidden_layer_units=CL_HIDDEN_LAYER_UNITS,
    total_classes=CL_TOTAL_CLASSES,
    pretrained_embedding_variable=word_embedding_variable,
)
text_classifier.model.summary()


# ### Load Pretrained Model
text_classifier.load_weights()

# ## Build Text Embedding Model
text_embedding = keras.Model(
    inputs=text_classifier.model.input,
    outputs=text_classifier.model.get_layer("bidirectional").output,
)
text_embedding.summary()


# ## Extract Bottleneck Features using Text Embedding
def extract_bottleneck_features(model, features_int, batch_size, num_workers):
    bottleneck_features = model.predict(
        x=features_int,
        batch_size=batch_size,
        verbose=1,
        workers=num_workers,
        use_multiprocessing=True,
    )
    return bottleneck_features


bottleneck_features = extract_bottleneck_features(
    model=text_embedding,
    features_int=sentences_features_int,
    batch_size=TE_BATCH_SIZE,
    num_workers=TE_NUM_WORKERS,
)


# Runtime is 53 minutes.
# System monitoring:
# - CPU utilization: 386%
# - RAM usage up to this point: 9.3 GB
# Delete sentences and NLP variables to free some memory.
if "sentences" in globals():
    del sentences
if "sentences_dataset" in globals():
    del sentences_dataset
if "sentences_features" in globals():
    del sentences_features
if "sentences_features_iter" in globals():
    del sentences_features_iter
if "sentences_features_int" in globals():
    del sentences_features_int
if "nlp" in globals():
    del nlp
gc.collect()

# System monitoring:
# - RAM usage up to this point: 8.4 GB
type(bottleneck_features)
bottleneck_features.shape
np.max(bottleneck_features)
np.min(bottleneck_features)

# Normalize bottleneck features for cosine similarity.
normalized_bottleneck_features = (
    bottleneck_features
    / tf.sqrt(tf.reduce_sum(tf.square(bottleneck_features), 1, keepdims=True))
).numpy()
type(normalized_bottleneck_features)
normalized_bottleneck_features.shape
np.min(normalized_bottleneck_features)
np.max(normalized_bottleneck_features)

# System monitoring:
# - RAM usage up to this point: 9.2 GB
# Save normalized bottleneck features to file.
normalized_bottleneck_features_path = os.path.join(
    TF_WORKING_DIRECTORY, "normalized_bottleneck_features.npy"
)
normalized_bottleneck_features_path
normalized_bottleneck_features.save(normalized_bottleneck_features_path)

# ## Build Nearest Text Finder Model
normalized_bottleneck_features_tensor = tf.Variable(
    normalized_bottleneck_features, trainable=False, dtype=tf.float32
)
normalized_bottleneck_features_tensor
normalized_text_embedding_tensor = text_embedding.output / tf.sqrt(
    tf.reduce_sum(tf.square(text_embedding.output), 1, keepdims=True)
)
normalized_text_embedding_tensor
cosine_similarity_tensor = tf.matmul(
    normalized_text_embedding_tensor,
    tf.transpose(normalized_bottleneck_features_tensor),
)
cosine_similarity_tensor
text_finder = keras.Model(inputs=text_embedding.input, outputs=cosine_similarity_tensor)
text_finder.summary()

# System monitoring:
# - RAM usage up to this point: 10.8 GB
# Delete unused models and the variables to free some memory.
if "word_embedding" in globals():
    del word_embedding
if "word_embedding_variable" in globals():
    del word_embedding_variable
if "text_classifier" in globals():
    del text_classifier
if "text_embedding" in globals():
    del text_embedding
if "bottleneck_features" in globals():
    del bottleneck_features
if "normalized_bottleneck_features" in globals():
    del normalized_bottleneck_features
gc.collect()

# System monitoring:
# - RAM usage up to this point: 9 GB
# Save text finder model to file.
model_directory = os.path.join(TF_WORKING_DIRECTORY, "models")
model_directory
if not os.path.isdir(model_directory):
    print(f"Creating directory... {model_directory}")
    os.makedirs(model_directory)
model_checkpoint_directory = os.path.join(model_directory, "checkpoint")
model_checkpoint_directory
if not os.path.isdir(model_checkpoint_directory):
    print(f"Creating directory... {model_checkpoint_directory}")
    os.makedirs(model_checkpoint_directory)
model_architecture_name = TF_MODEL_ARCHITECTURE_NAME
model_architecture_name
data_name = TF_DATA_NAME
data_name
application_name = TF_APPLICATION_NAME
application_name
model_checkpoint_file_name = "%s__%s__%s.h5" % (
    model_architecture_name,
    data_name,
    application_name,
)
model_checkpoint_file_name
model_checkpoint_path = os.path.join(
    model_checkpoint_directory, model_checkpoint_file_name
)
model_checkpoint_path
tic = time.time()
text_finder.save(model_checkpoint_path)
toc = time.time()
print(f"Runtime: {int((toc-tic)/60)} minutes.")


# System monitoring:
# - RAM usage up to this point: 5.1 GB
# - Disk usage up to this point: 700 MB
# # Make Similarity Prediction
def make_similarity_prediction(
    dataset,
    dataset_size,
    preprocessing_lambda,
    model,
    num_parallel_calls,
    batch_size,
    num_workers,
):
    """Make prediction on text finder."""
    # Free some memory
    if "sentences_df" in globals():
        del sentences_df
        gc.collect()
    # Make data pipeline
    features = (
        dataset.map(preprocessing_lambda, num_parallel_calls=num_parallel_calls)
        .batch(dataset_size)
        .prefetch(buffer_size=num_parallel_calls)
    )
    features_iter = iter(features)
    features_int = features_iter.get_next()
    # Make prediction
    predictions = model.predict(
        x=features_int,
        batch_size=batch_size,
        verbose=1,
        workers=num_workers,
        use_multiprocessing=True,
    )
    return predictions


def get_prediction_info(prediction: List[int], idx: int = 0, top_k: int = 10):
    """Get sentences info from prediction index."""
    if "sentences_df" not in globals():
        sentences_df = pd.read_csv(sentences_path)
    nearest = (-prediction[idx]).argsort()[:top_k]
    return sentences_df[sentences_df.index.isin(nearest)]


# ## Initialize Model
# Initializing model may take some time (72 seconds) to initiate the model graph.
questions = [
    "Much",
]
questions_dataset = tf.data.Dataset.from_generator(
    lambda: questions, (tf.string), (tf.TensorShape([]))
)
similarity_prediction = make_similarity_prediction(
    dataset=questions_dataset,
    dataset_size=len(questions),
    preprocessing_lambda=preprocessing_lambda,
    model=text_finder,
    num_parallel_calls=TF_NUM_PARALLEL_CALLS,
    batch_size=TF_BATCH_SIZE,
    num_workers=TF_NUM_WORKERS,
)
type(similarity_prediction)
similarity_prediction.shape
get_prediction_info(prediction=similarity_prediction)

# ## Test using Existing Text in the Articles
questions = [
    "Much progress has been made in understanding the role of structural and accessory proteins in the pathogenesis of severe acute respiratory syndrome coronavirus (SARS-CoV) infections.",
]
questions_dataset = tf.data.Dataset.from_generator(
    lambda: questions, (tf.string), (tf.TensorShape([]))
)
similarity_prediction = make_similarity_prediction(
    dataset=questions_dataset,
    dataset_size=len(questions),
    preprocessing_lambda=preprocessing_lambda,
    model=text_finder,
    num_parallel_calls=TF_NUM_PARALLEL_CALLS,
    batch_size=TF_BATCH_SIZE,
    num_workers=TF_NUM_WORKERS,
)
get_prediction_info(prediction=similarity_prediction)

# ## Find Nearest Sentences to the Questions
questions = [
    "Real-time tracking of whole genomes and a mechanism for coordinating the rapid dissemination of that information to inform the development of diagnostics and therapeutics and to track variations of the virus over time.",
    "Access to geographic and temporal diverse sample sets to understand geographic distribution and genomic differences, and determine whether there is more than one strain in circulation. Multi-lateral agreements such as the Nagoya Protocol could be leveraged.",
    "Evidence that livestock could be infected (e.g., field surveillance, genetic sequencing, receptor binding) and serve as a reservoir after the epidemic appears to be over.",
    "Evidence of whether farmers are infected, and whether farmers could have played a role in the origin.",
    "Surveillance of mixed wildlife-livestock farms for SARS-CoV-2 and other coronaviruses in Southeast Asia.",
    "Experimental infections to test host range for this pathogen.",
    "Animal host(s) and any evidence of continued spill-over to humans.",
    "Socioeconomic and behavioral risk factors for this spill-over.",
    "Sustainable risk reduction strategies.",
]
questions_dataset = tf.data.Dataset.from_generator(
    lambda: questions, (tf.string), (tf.TensorShape([]))
)
similarity_prediction = make_similarity_prediction(
    dataset=questions_dataset,
    dataset_size=len(questions),
    preprocessing_lambda=preprocessing_lambda,
    model=text_finder,
    num_parallel_calls=TF_NUM_PARALLEL_CALLS,
    batch_size=TF_BATCH_SIZE,
    num_workers=TF_NUM_WORKERS,
)

# # Q1
# #### Real-time tracking of whole genomes and a mechanism for coordinating the rapid dissemination of that information to inform the development of diagnostics and therapeutics and to track variations of the virus over time.
get_prediction_info(prediction=similarity_prediction, idx=0)

# # Q2
# #### Access to geographic and temporal diverse sample sets to understand geographic distribution and genomic differences, and determine whether there is more than one strain in circulation. Multi-lateral agreements such as the Nagoya Protocol could be leveraged.
get_prediction_info(prediction=similarity_prediction, idx=1)

# # Q3
# #### Evidence that livestock could be infected (e.g., field surveillance, genetic sequencing, receptor binding) and serve as a reservoir after the epidemic appears to be over.
get_prediction_info(prediction=similarity_prediction, idx=2)

# # Q3a
# #### Evidence of whether farmers are infected, and whether farmers could have played a role in the origin.
get_prediction_info(prediction=similarity_prediction, idx=3)

# # Q3b
# #### Surveillance of mixed wildlife-livestock farms for SARS-CoV-2 and other coronaviruses in Southeast Asia.
get_prediction_info(prediction=similarity_prediction, idx=4)

# # Q3c
# #### Experimental infections to test host range for this pathogen.
get_prediction_info(prediction=similarity_prediction, idx=5)

# # Q4
# #### Animal host(s) and any evidence of continued spill-over to humans.
get_prediction_info(prediction=similarity_prediction, idx=6)

# # Q5
# #### Socioeconomic and behavioral risk factors for this spill-over.
get_prediction_info(prediction=similarity_prediction, idx=7)

# # Q6
# #### Sustainable risk reduction strategies.
get_prediction_info(prediction=similarity_prediction, idx=8)
