In this talk, Robert Underwood will share the recent progress of the AuroraGPT Team, how data contributes to the project of building a science-focused LLM with AuroraGPT, and the topics that his team sees as open questions. He will discuss the systems and data quality challenges the team tackles to prepare terabytes of scientific data and text to produce high-quality text and data for training.