This talk will focus on our experiences with making open source tools for the study of social media platforms (amongst others, DMI-TCAT for Twitter, the YouTube Data Tools, and 4CAT for forum-like platforms such as Reddit and 4chan) in the context of social science and humanities research. We will discuss questions of reliability and reproducibility, but also how tools are taking part in shaping which questions are being asked and how research is done in practice - making open source particularly relevant as a form of methodological transparency. Two aspects have become particularly important for our tool-making practice: the relationship with large platform companies and their Web-APIs as well as concerns about user privacy and legal compliance with regulations such as the GDPR. Our talk will address these in turn, scoping the issue and proposing ways forward.