How to reliably evaluate storage systems The blkreplay toolkit replays I/O-loads on storage systems on the block-level. This allows measurements and comparisons of storage systems under close-to-real-life conditions. With this tool deep insights into workloads and storage systems can be obtained easily. The blkreplay toolkit was developed by 1&1 Internet AG in order to reproduce natural loads, recorded via blktrace in our data centers. It automates large laboratory projects, e.g. benchmarking comparisons of a wide variety of storage hardware and its multi-dimensional parameter space. The presentation will explain how the blkreplay toolkit works and discuss the block layer storage loads that have been recorded on various servers of a large ISP. The loads available cover a large range of applications including for example webhosting, databases and backup servers. The application of these loads to commercial and open source / commodity storage systems has led to a deep insight into their behaviour. Some of which are surprising and eye-opening. Finally it will be shown that OpenSource based hard- and software systems can compete with commercial ones in many areas, provided that certain conditions are met.