---
title: "4 Benchmarking Performance"
author: "Adriano Rutz"
creative_commons: CC BY-SA
date: "`r Sys.Date()`"
output: rmarkdown::html_vignette
opengraph:
    image:
      src: https://github.com/taxonomicallyinformedannotation/tima/blob/main/man/figures/logo.svg
      alt: "Taxonomically Informed Metabolite Annotation"
    twitter:
      creator: "@adafede"
vignette: >
  %\VignetteIndexEntry{4 Benchmarking Performance}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
resource_files:
  - benchmark_full_pos.svg
  - benchmark_zoomed_pos.svg
  - benchmark_comparison_pos.svg
  - benchmark_full_neg.svg
  - benchmark_zoomed_neg.svg
  - benchmark_comparison_neg.svg
---

```{r setup, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)
```

This vignette simply shows the actual performance of TIMA.

The benchmarking dataset was built using [https://zenodo.org/record/5186176](https://zenodo.org/record/5186176).

It contained positive and negative MS^2^ spectra of multiple ion species ([M+H]^+^, [M+Na]^+^, [M+H~4~N]^+^, ...) coming from different mass spectrometers.

In positive mode,
It was filtered to 27,789 spectra, representing 17,822 structures without stereo.
Of those, only 15,005 spectra (54.0%) corresponded to structures present in the library we used to annotate.

In negative mode,
It was filtered to 12,060 spectra, representing 9,112 structures without stereo.
Of those, only 6,282 spectra (52.1%) corresponded to structures present in the library we used to annotate.

## Best 100 candidates
### Positive

![Benchmark ROC curves (positive mode)](./benchmark_full_pos.svg)\

### Negative

![Benchmark ROC curves (negative mode)](./benchmark_full_neg.svg)\

## Best 25 candidates (zoomed)
### Positive

![Benchmark ROC curves zoomed on the first 25 candidates (positive mode)](./benchmark_zoomed_pos.svg)\

### Negative

![Benchmark ROC curves zoomed on the first 25 candidates (negative mode)](./benchmark_zoomed_neg.svg)\

## Candidates distribution
### Positive

![Benchmark comparison (positive mode)](./benchmark_comparison_pos.svg)\

### Negative

![Benchmark comparison (negative mode)](./benchmark_comparison_neg.svg)\
