Breast cancer is one of the most common and serious health issues in the world, affecting millions of lives every year. Its prevalence crosses demographic lines, affecting people from all phases of life, genders, and socioeconomic backgrounds. It is a severe health issue that predominantly affects women and, less frequently, males. Timely detection and categorization of breast cancer are extremely important because it is one of the main causes of female death. Early-stage diagnosis significantly improves patient prognosis in around 90% of these cases, the treatment is successful. This striking statistics drives scientists to come up with an even more precise structure for identifying breast cancer as early as possible. Early detection is crucial for better treatment outcomes and patient survival highlighting the need for more accurate and efficient diagnostic techniques. In this context, technical improvements and the incorporation of cutting-edge approaches like machine learning and deep learning have led to significant innovation in breast cancer detection. Hence, in this dissertation focus is given on developing efficient high performance frameworks for detection of breast cancer using mammogram and ultrasound breast images. For this purpose, eight different breast cancer detection schemes including one machine learning-based scheme, five transfer learning-based schemes, and two sparse learning along with transfer learning schemes are proposed. Among all the proposed schemes, the modified Relation and Margin-based Deep Learning Network outperforms others with best performance measures.