I’d like to show you how to find bugs in Linux systems using machine learning, when paired with the totally seemingly useless and annoying false positives that come out of your integration tests. We’ve all been frustrated unreproducible bugs in Linux … And also by stupid test-flakes that show up as failures in integration tests even though nothing related has changed. Both unproducible bugs and test flakes are the result of permutations in timing, load, ordering. I want to prove that unreproducible bugs are the same thing as test flakes. And we have a massive source of data here, millions of records of test flakes. Lets take a look at how the Cockpit project trains bots to correlate these flakes into unsupervised clusters. and automatically make use of the data, identifying real bugs, or simply retriggering tests. We’ll dive into details about Normalized Compression Distance, Unsupervised Clustering, TF-IDF and many other simple techniques used to zero in on the bugs.