Creating and Managing Benchmark Suites with ABM
When researching the analysis and manipulation of source code in domains such as static analysis and meta-programming, researchers often need to create test corpora that are representative of real-world projects in the domains for which their research is relevant. Characterizing relevance in terms of code metrics is a difficult task, so we present the ABM (the Automated Benchmark Management application) web application which assists analysis developers in creating test corpora for their analyses. ABM uses Delphi, a data curation platform which crawls software repositories and indexes code features of software projects. With Delphi, ABM allows its users to search for projects matching certain code features, and to create and manage collections of such projects. ABM collections can also be persisted to the Zenodo platform (with a DOI), which also allows the reuse of collections by others. We further illustrate the usage of ABM by creating benchmark suites for an existing analysis tool. With this presentation, we encourage researchers to create reproducible research and available test suites, through the ABM website.
Slides (presentation.pdf) | 6.25MiB |
Tue 16 JulDisplayed time zone: Belfast change
13:30 - 15:00 | |||
13:30 30mTalk | A Central and Evolving Benchmark BenchWork File Attached | ||
14:00 30mTalk | Creating and Managing Benchmark Suites with ABM BenchWork Lisa Nguyen Quang Do Paderborn University File Attached | ||
14:30 30mTalk | Hermes: Towards Representative Benchmarks BenchWork Michael Eichberg TU Darmstadt, Germany Media Attached |