For a number of medical diagnostic tasks, deep learning (DL) methods have proven to be quite successful, sometimes even outperforming human experts. The algorithms' black-box nature has, however, limited their therapeutic application. Recent studies on explainability seek to identify the factors most responsible for a model's choice. In the biomedical domain, deep neural networks (DNNs) now represent most successful machine learning (ML) technologies. The various topics of interest in this field include BBMI (study of interface between the brain as well as body's mechanical systems), bioimaging (the study of biological cells and tissues), medical imaging (study of human organs through the creation of visual representations), and public and medical health management (PmHM). This study provides an overview of explainable artificial intelligence (XAI) applied in class activation mapping-based DL medical picture analysis. For the purpose of categorizing DL-based medical image analysis (MIA) techniques, a framework of XAI criteria is presented. The papers are then surveyed and categorized in accordance with framework as well as based on anatomical location for use in MIA.