Please use this identifier to cite or link to this item: http://hdl.handle.net/11667/109
Appears in Collections:University of Stirling Research Data
Title: Data for the paper "Investigating benchmark correlations when comparing algorithms with parameter tuning"
Creator(s): Brownlee, Alexander E I
Christie, Lee
Woodward, John R
Contact Email: alexander.brownlee@stir.ac.uk
Keywords: benchmarks
BBOB
ranking
differential evolution
continuous optimisation
parameter tuning
automated design of algorithms
Date Available: 5-Apr-2018
Citation: Brownlee, AEI; Christie, L; Woodward, JR (2018): Data for the paper "Investigating benchmark correlations when comparing algorithms with parameter tuning". University of Stirling. Dataset. http://hdl.handle.net/11667/109
Dataset Description (Abstract): Benchmarks are important to demonstrate the utility of optimisation algorithms, but there is controversy about the practice of benchmarking; we could select instances that present our algorithm favourably, and dismiss those on which our algorithm under-performs. Several papers highlight the pitfalls concerned with benchmarking, some of which concern the context of the automated design of algorithms, where we use a set of problem instances (benchmarks) to train our algorithm. As with machine learning, if the training set does not reflect the test set, the algorithm will not generalize. This raises some open questions concerning the use of test instances to automatically design algorithms. We use differential evolution, and sweep the parameter settings to investigate the practice of benchmarking using the BBOB benchmarks. We make three key findings. Firstly, several benchmark functions are highly correlated. This may lead to the false conclusion that an algorithm performs well in general, when it performs poorly on a few key instances, possibly introducing unwanted bias to a resulting automatically designed algorithm. Secondly, the number of evaluations can have a large effect on the conclusion. Finally, a systematic sweep of the parameters shows how performance varies with time across the space of algorithm configurations. This data set includes the experimental results and correlations reported in the paper.
Dataset Description (TOC): Data sets for the paper "Investigating Benchmark Correlations when Comparing Algorithms with Parameter Tuning"; Lee A. Christie, Alexander E.I. Brownlee, John R. Woodward; Proceedings of GECCO 2018, Kyoto Japan. vote-si1.xlsx - ranks for the coarse-grained sweep. finished-results/*.csv - these are the output files from which were calculated the correlations for the fine-grained sweep. correlations.csv - the spearman's rank correlation data for the fine-grained sweep between functions for generations 1-25. Additional details are provided in the readme.txt file. Dedicated UnZip software is recommended for accessing the dataset, for example, IZArc.
Type: dataset
Contract/Grant Title: FAIME: A Feature based Framework to Automatically Integrate and Improve Metaheuristics via Examples
DAASE: Dynamic Adaptive Automated Software Engineering
Funder(s): EPSRC - Engineering and Physical Sciences Research Council
Contract/Grant Number: EP/N002849/1
EP/J017515/1
URI: http://hdl.handle.net/11667/109
Rights: Rights covered by the standard CC-BY 4.0 licence: https://creativecommons.org/licenses/by/4.0/
Affiliation(s) of Dataset Creator(s): University of Stirling (Computing Science - CSM Dept)
Queen Mary University of London

Files in This Item:
File Description SizeFormat 
data.zip1.02 MBUnknownView/Open
readme.txt520 BTextView/Open


This item is protected by original copyright



Items in DataSTORRE are protected by copyright, with all rights reserved, unless otherwise indicated.