(doc_or_tokens: types.DocOrTokens, sample_size: int = 42)
return sum(type_contributions)
Compute the Hypergeometric Distribution Diversity (HD-D) of ``doc_or_tokens``,
    which calculates the mean contribution that each unique word (aka type) makes
    to the TTR of all possible combinations of random samples of words of a given size,
    then sums all contributions together.

    Args:
        doc_or_tokens: If a spaCy ``Doc``, non-punctuation tokens (words) are extracted;
            if an iterable of spaCy ``Token`` s, all non-punct elements are used.
        sample_size: Number of words randomly sampled without replacement when
            computing unique word appearance probabilities.
            Value should be in the range [35, 50].

    Note:
        The popular vocd-D index of lexical diversity is actually just an approximation
        of HD-D, and should not be used.

    References:
        - McCarthy, P. M., & Jarvis, S. (2010). MTLD, vocd-D, and HD-D: A validation study
          of sophisticated approaches to lexical diversity assessment. Behavior research
          methods, 42(2), 381-392.
        - McCarthy, P. M., & Jarvis, S. (2007). A theoretical and empirical evaluation
          of vocd. Language Testing, 24, 459-488.
Compute the Hypergeometric Distribution Diversity (HD-D) of ``doc_or_tokens``,
    which calculates the mean contribution that each unique word (aka type) makes
    to the TTR of all possible combinations of random samples of words of a given size,
    then sums all contributions together.
Compute the Hypergeometric Distribution Diversity ( HD - D ) of doc_or_tokens which calculates the mean contribution that each unique word ( aka type ) makes to the TTR of all possible combinations of random samples of words of a given size then sums all contributions together .
def hdd(doc_or_tokens: types.DocOrTokens, sample_size: int = 42) -> float:
    """
    Compute the Hypergeometric Distribution Diversity (HD-D) of ``doc_or_tokens``,
    which calculates the mean contribution that each unique word (aka type) makes
    to the TTR of all possible combinations of random samples of words of a given size,
    then sums all contributions together.

    Args:
        doc_or_tokens: If a spaCy ``Doc``, non-punctuation tokens (words) are extracted;
            if an iterable of spaCy ``Token`` s, all non-punct elements are used.
        sample_size: Number of words randomly sampled without replacement when
            computing unique word appearance probabilities.
            Value should be in the range [35, 50].

    Note:
        The popular vocd-D index of lexical diversity is actually just an approximation
        of HD-D, and should not be used.

    References:
        - McCarthy, P. M., & Jarvis, S. (2010). MTLD, vocd-D, and HD-D: A validation study
          of sophisticated approaches to lexical diversity assessment. Behavior research
          methods, 42(2), 381-392.
        - McCarthy, P. M., & Jarvis, S. (2007). A theoretical and empirical evaluation
          of vocd. Language Testing, 24, 459-488.
    """
    words = utils.get_words(doc_or_tokens)
    type_counts = itertoolz.frequencies(word.lower for word in words)
    n_words = sum(type_counts.values())
    if n_words < sample_size:
        LOGGER.warning(
            "number of words in document (%s) must be greater than sample size (%s) "
            "to compute HD-D; setting sample size equal to number of words, "
            "which effectively reduces this method to standard TTR",
            n_words,
            sample_size,
        )
        sample_size = n_words

    type_contributions = (
        (1.0 - hypergeom.pmf(0, n_words, type_count, sample_size)) / sample_size
        for type_count in type_counts.values()
    )
    return sum(type_contributions)
def hdd ( doc_or_tokens : types . DocOrTokens , sample_size : int = 42 ) -> float : words = utils . get_words ( doc_or_tokens ) type_counts = itertoolz . frequencies ( word . lower for word in words ) n_words = sum ( type_counts . values ( ) ) if n_words < sample_size : LOGGER . warning ( "number of words in document (%s) must be greater than sample size (%s) " "to compute HD-D; setting sample size equal to number of words, " "which effectively reduces this method to standard TTR" , n_words , sample_size , ) sample_size = n_words type_contributions = ( ( 1.0 - hypergeom . pmf ( 0 , n_words , type_count , sample_size ) ) / sample_size for type_count in type_counts . values ( ) ) return sum ( type_contributions )