The scientific publication score – a new tool for summarizing evidence and data quality criteria of biomedical publications

Abstract. The number of biomedical research articles increases by
over 2.5 million publications each year, making it difficult to stay up to
date. In this study, we introduce a standardized search and evaluation tool
to combat this issue. Employing crowdsourcing, a large database of publications
is gathered. Using a standardized data entry format, coined the “scientific
publication score” (SPS), specific publication results can be easily
aggregated, thereby allowing fast and accurate comparisons for clinical
questions. The SPS combines two quality dimensions. The first captures the
quality of evidence of the study using the evidence criteria defined by the
Centre for Evidence-Based Medicine, Oxford, UK. The second is more
fine-grained and considers the magnitude of statistical analyses on
individual and specific results. From 2014 to 2019, experts of the European Bone and Joint Infection Society
(EBJIS) were asked to enter data of relevant publications about prosthetic
joint infection. Data and evidence levels of specific results were averaged,
summarized and ranked. A total of 366 publications were divided into two groups: (I) risk factors (e.g., host-related factors, pre- and postoperative issues) with 243
publications and (II) diagnostic methods (e.g., laboratory tests, imaging methods)
with 123 publications. After ranking, the highest score for risk factors of prosthetic joint
infection were calculated by the SPS for anemia (mean 3.50± SD 0.91),
malignancy (mean 3.17± SD 0.29) and previous alloarthroplasty (mean
3.00± SD 0.35). A comparison of the full SPS ranking with the ranking
determined at the 2018 International Consensus Meeting (ICM) on Musculoskeletal
Infection resulted in a Spearman rank correlation coefficient of 0.48
and a p value of 0.0382. The diagnostic methods ranked highest by the SPS
were aspirate leucocyte count (mean 3.15± SD 1.21), interleukin 6
(mean 3.14± SD 1.07) and aspirate (neutrophils over 80 %) (mean
3.12± SD 0.63). The comparison to the ICM ranking yielded a
Spearman rank correlation coefficient of 0.91 and a p value of 0.0015. Our pilot study evaluated a new tool for the quality assessment of specific
results based on the quality of the source publication. The SPS is suitable
for a ranking of specific results by evidence and data quality criteria
important for systematic reviews.
