In this study, the possibility of using RCA from building demolition waste in asphaltic mix, in combination with natural aggregate is explored. RCA was considered as 0%, 25%, 50%, and 60% of the weight of dry aggregates. Two grades of asphalt were used in this study namely, viscosity grade-30 (VG 30) and a polymer-modified asphalt binder (PMB-40). The RCA was used as coarse and fine fractions and the filler was used as hydrated lime. The behavior of the asphalt mix composed of different combinations of RCA and natural aggregates was determined using volumetric, indirect tensile strength, moisture damage, and rutting properties. The results showed an increase in optimum asphalt content in the design mixtures with an increase in the percentage of RCA. In addition, the mix prepared with polymer-modified asphalt and combinations of RCA-NA indicates improvement in indirect tensile strength, moisture damage resistance, and rutting resistance properties as compared to an RCA mix prepared with virgin asphalt binders. The inclusion of hydrated lime as filler and polymer-modified binder enhanced the resistance to moisture damage properties of the RCA mix, irrespective of the asphalt binders. Overall study concluded that the optimum percentage of RCA can be considered without pre-treatment up to 50% for better performance of asphalt mix.