Benchmarking
702279
Fashion MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
We present Fashion MNIST, a new dataset comprising of 28x28 grayscale images of 70,000 fashion products from 10 categories, with 7,000 images per category. The training set has 60,000 images and the test set has 10,000 images. Fashion MNIST is intended to serve as a direct drop in replacement for the original MNIST dataset for benchmarking machine learning algorithms, as it shares the same image size, data format and the structure of training and testing splits. The dataset is freely available at this https URL
ArXiv
Han Xiao Kashif Rasul Roland Vollgraf
56657912
Benchmarking Neural Network Robustness to Common Corruptions and Perturbations
In this paper we establish rigorous benchmarks for image classifier robustness. Our first benchmark, ImageNet C, standardizes and expands the corruption robustness topic, while showing which classifiers are preferable in safety critical applications. Then we propose a new dataset called ImageNet P which enables researchers to benchmark a classifier's robustness to common perturbations. Unlike recent robustness research, this benchmark evaluates performance on common corruptions and perturbations not worst case adversarial perturbations. We find that there are negligible changes in relative corruption robustness from AlexNet classifiers to ResNet classifiers. Afterward we discover ways to enhance corruption and perturbation robustness. We even find that a bypassed adversarial defense provides substantial common perturbation robustness. Together our benchmarks may aid future work toward networks that robustly generalize.
ICLR
Dan Hendrycks Thomas G Dietterich
39760169
Benchmarking Single Image Dehazing and Beyond
We present a comprehensive study and evaluation of existing single image dehazing algorithms, using a new large scale benchmark consisting of both synthetic and real world hazy images, called REalistic Single Image DEhazing (RESIDE) RESIDE highlights diverse data sources and image contents, and is divided into five subsets, each serving different training or evaluation purposes. We further provide a rich variety of criteria for dehazing algorithm evaluation, ranging from full reference metrics to no reference metrics and to subjective evaluation, and the novel task driven evaluation. Experiments on RESIDE shed light on the comparisons and limitations of the state of the art dehazing algorithms, and suggest promising future directions.
IEEE Transactions on Image Processing
Boyi Li Wenqi Ren Dengpan Fu Dacheng Tao Dan Feng Wenjun Zeng Zhangyang Wang
14502739
Multitask learning and benchmarking with clinical time series data
Health care is one of the most exciting frontiers in data mining and machine learning. Successful adoption of electronic health records (EHRs) created an explosion in digital clinical data available for analysis, but progress in machine learning for healthcare research has been difficult to measure because of the absence of publicly available benchmark data sets. To address this problem, we propose four clinical prediction benchmarks using data derived from the publicly available Medical Information Mart for Intensive Care (MIMIC III) database. These tasks cover a range of clinical problems including modeling risk of mortality, forecasting length of stay, detecting physiologic decline, and phenotype classification. We propose strong linear and neural baselines for all four tasks and evaluate the effect of deep supervision, multitask training and data specific architectural modifications on the performance of neural models.
Scientific Data
Hrayr Harutyunyan Hrant Khachatrian David C Kale A G Galstyan
9885279
Benchmarking Complications Associated with Esophagectomy
Objective: Utilizing a standardized dataset with specific definitions to prospectively collect international data to provide a benchmark for complications and outcomes associated with esophagectomy. Summary of Background Data: Outcome reporting in oncologic surgery has suffered from the lack of a standardized system for reporting operative results particularly complications. This is particularly the case for esophagectomy affecting the accuracy and relevance of international outcome assessments, clinical trial results, and quality improvement projects. Methods: The Esophageal Complications Consensus Group (ECCG) involving 24 high volume esophageal surgical centers in 14 countries developed a standardized platform for recording complications and quality measures associated with esophagectomy. Using a secure online database (ESODATA.org) ECCG centers prospectively recorded data on all resections according to the ECCG platform from these centers over a 2 year period. Results: Between January 2015 and December 2016, 2704 resections were entered into the database. All demographic and follow up data fields were 100% complete. The majority of operations were for cancer (95.6% and typically located in the distal esophagus (56.2% Some 1192 patients received neoadjuvant chemoradiation (46.1% and 763 neoadjuvant chemotherapy (29.5% Surgical approach involved open procedures in 52.1% and minimally invasive operations in 47.9% Chest anastomoses were done most commonly (60.7% and R0 resections were accomplished in 93.4% of patients. The overall incidence of complications was 59% with the most common individual complications being pneumonia (14.6% and atrial dysrhythmia (14.5% Anastomotic leak, conduit necrosis, chyle leaks, recurrent nerve injury occurred in 11.4% 1.3% 4.7% and 4.2% of cases, respectively. Clavien Dindo complications IIIb occurred in 17.2% of patients. Readmissions occurred in 11.2% of cases and 30 and 90 day mortality was 2.4% and 4.5% respectively. Conclusion: Standardized methods provide contemporary international benchmarks for reporting outcomes after esophagectomy.
Annals of surgery
Donald E Low Madhan Kumar Kuppusamy Derek Alderson Ivan Cecconello Andrew C Chang Gail E Darling A R Davies Xavier Benoit D'Journo Suzanne S Gisbertz S Michael Griffin Richard H Hardwick Arnulf Heinrich Hoelscher Wayne L Hofstetter Blair A Jobe Yuko Kitagawa Simon Ying-Kit Law Christophe Mariette Nicholas David Maynard Christopher R Morse Philippe R Nafteux Manuel Pera C S Pramesh Sonia Puig John V Reynolds Wolfgang Schroeder Mark Smithers Bas P L Wijnhoven
44832976
Benchmarking heterogeneous electrocatalysts for the oxygen evolution reaction.
Objective evaluation of the activity of electrocatalysts for water oxidation is of fundamental importance for the development of promising energy conversion technologies including integrated solar water splitting devices, water electrolyzers, and Li air batteries. However, current methods employed to evaluate oxygen evolving catalysts are not standardized, making it difficult to compare the activity and stability of these materials. We report a protocol for evaluating the activity, stability, and Faradaic efficiency of electrodeposited oxygen evolving electrocatalysts. In particular, we focus on methods for determining electrochemically active surface area and measuring electrocatalytic activity and stability under conditions relevant to an integrated solar water splitting device. Our primary figure of merit is the overpotential required to achieve a current density of 10 mA cm( 2) per geometric area, approximately the current density expected for a 10% efficient solar to fuels conversion device. Utilizing the aforementioned surface area measurements, one can determine electrocatalyst turnover frequencies. The reported protocol was used to examine the oxygen evolution activity of the following systems in acidic and alkaline solutions: CoO(x) CoPi, CoFeO(x) NiO(x) NiCeO(x) NiCoO(x) NiCuO(x) NiFeO(x) and NiLaO(x) The oxygen evolving activity of an electrodeposited IrO(x) catalyst was also investigated for comparison. Two general observations are made from comparing the catalytic performance of the OER catalysts investigated: (1) in alkaline solution, every non noble metal system achieved 10 mA cm( 2) current densities at similar operating overpotentials between 0.35 and 0.43 V, and (2) every system but IrO(x) was unstable under oxidative conditions in acidic solutions.
Journal of the American Chemical Society
Charles C L McCrory Suho Jung Jonas C Peters Thomas Francisco Jaramillo
59336626
Benchmarking Aggression Identification in Social Media
In this paper, we present the report and findings of the Shared Task on Aggression Identification organised as part of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC 1) at COLING 2018. The task was to develop a classifier that could discriminate between Overtly Aggressive, Covertly Aggressive, and Non aggressive texts. For this task, the participants were provided with a dataset of 15,000 aggression annotated Facebook Posts and Comments each in Hindi (in both Roman and Devanagari script) and English for training and validation. For testing, two different sets one from Facebook and another from a different social media were provided. A total of 130 teams registered to participate in the task, 30 teams submitted their test runs, and finally 20 teams also sent their system description paper which are included in the TRAC workshop proceedings. The best system obtained a weighted F score of 0.64 for both Hindi and English on the Facebook test sets, while the best scores on the surprise set were 0.60 and 0.50 for English and Hindi respectively. The results presented in this report depict how challenging the task is. The positive response from the community and the great levels of participation in the first edition of this shared task also highlights the interest in this topic.
TRAC@COLING 2018
Ritesh Kumar Atul Kr Ojha Shervin Malmasi Marcos Zampieri
4644084
Benchmarking 6DOF Outdoor Visual Localization in Changing Conditions
Visual localization enables autonomous vehicles to navigate in their surroundings and augmented reality applications to link virtual to real worlds. Practical visual localization approaches need to be robust to a wide variety of viewing condition, including day night changes, as well as weather and seasonal variations, while providing highly accurate 6 degree of freedom (6DOF) camera pose estimates. In this paper, we introduce the first benchmark datasets specifically designed for analyzing the impact of such factors on visual localization. Using carefully created ground truth poses for query images taken under a wide variety of conditions, we evaluate the impact of various factors on 6DOF camera pose estimation accuracy through extensive experiments with state of the art localization approaches. Based on our results, we draw conclusions about the difficulty of different conditions, showing that long term localization is far from solved, and propose promising avenues for future work, including sequence based localization approaches and the need for better local features. Our benchmark is available at visuallocalization.net.
2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
Torsten Sattler William P Maddern Carl Toft Akihiko Torii Lars Hammarstrand Erik Stenborg Daniel Safari M Okutomi Marc Pollefeys Josef Sivic Fredrik Kahl Tomas Pajdla
2589691
Benchmarking cloud serving systems with YCSB
While the use of MapReduce systems (such as Hadoop) for large scale data analysis has been widely recognized and studied, we have recently seen an explosion in the number of systems developed for cloud data serving. These newer systems address "cloud OLTP" applications, though they typically do not support ACID transactions. Examples of systems proposed for cloud serving use include BigTable, PNUTS, Cassandra, HBase, Azure, CouchDB, SimpleDB, Voldemort, and many others. Further, they are being applied to a diverse range of applications that differ considerably from traditional (e.g. TPC C like) serving workloads. The number of emerging cloud serving systems and the wide range of proposed applications, coupled with a lack of apples to apples performance comparisons, makes it difficult to understand the tradeoffs between systems and the workloads for which they are suited. We present the "Yahoo! Cloud Serving Benchmark" (YCSB) framework, with the goal of facilitating performance comparisons of the new generation of cloud data serving systems. We define a core set of benchmarks and report results for four widely used systems: Cassandra, HBase, Yahoo!'s PNUTS, and a simple sharded MySQL implementation. We also hope to foster the development of additional cloud benchmark suites that represent other classes of applications by making our benchmark tool available via open source. In this regard, a key feature of the YCSB framework/tool is that it is extensible it supports easy definition of new workloads, in addition to making it easy to benchmark new systems.
SoCC '10
Brian F Cooper Adam Silberstein Erwin Tam Raghu Ramakrishnan Russell Sears
12296499
Benchmarking Deep Reinforcement Learning for Continuous Control
Recently, researchers have made significant progress combining the advances in deep learning for learning feature representations with reinforcement learning. Some notable examples include training agents to play Atari games based on raw pixel data and to acquire advanced manipulation skills using raw sensory inputs. However, it has been difficult to quantify progress in the domain of continuous control due to the lack of a commonly adopted benchmark. In this work, we present a benchmark suite of continuous control tasks, including classic tasks like cart pole swing up, tasks with very high state and action dimensionality such as 3D humanoid locomotion, tasks with partial observations, and tasks with hierarchical structure. We report novel findings based on the systematic evaluation of a range of implemented reinforcement learning algorithms. Both the benchmark and reference implementations are released at this https URL in order to facilitate experimental reproducibility and to encourage adoption by other researchers.
ICML
Yan Duan Xi Chen Rein Houthooft John Schulman P Abbeel