In recent years, deep-learning based side-channel attacks have been proven to be very effective and opened the door to automated implementation techniques. Building on this line of work, this talk explores how to take the approach a step further and showcases how to leverage the recent advance in AI explainability to quickly assess which parts of the implementation is responsible for the information. Through a concrete set by step example, we will showcase the promise of this approach, its limitations, and how it can be used today.