The emergence of fifth-generation (5G) networks has introduced complex challenges in managing diverse service requirements, particularly in meeting the strict latency and reliability needs of ultra-reliable low-latency communications (uRLLC) while maintaining the high throughput demands of enhanced mobile broadband (eMBB) under dynamic network conditions. Traditional orthogonal multiple access (OMA) and conventional non-orthogonal multiple access (NOMA) schemes struggle to efficiently allocate resources across these diverse services, often resulting in suboptimal spectral efficiency, increased interference, and degraded quality of service (QoS). To address these challenges, adaptive hybrid NOMA frameworks combined with artificial intelligence (AI)-driven intelligent resource management are developed to enhance 5G network performance. The first part of this research introduces a network slicing based hybrid NOMA approach that utilizes intelligent user pairing techniques - near-far/far-near (NF-FN) and near-near/far-far (NN-FF) - to dynamically allocate resources between uRLLC and eMBB traffic. The NF-FN pairing enhances spectral efficiency by grouping users with different channel conditions, while NN-FF pairing reduces interference and latency for users with similar channel characteristics. This framework demonstrates significant improvements in both throughput and latency compared to traditional OMA, effectively balancing the trade-offs between eMBB and uRLLC performance. To further improve adaptability in dynamic 5G environments, the second part of this work presents a reinforcement learning (RL)-based hybrid NOMA system using a deep Q-network (DQN). This framework includes 1). Dynamic mode switching, where an intelligent DQN agent selects between NOMA and OMA modes based on real-time channel conditions and user mobility 2). Mobility-aware optimization, which reduces decoding complexity while maintaining QoS under high mobility and 3). A multi-objective reward function that optimizes spectral efficiency, fairness, and energy efficiency simultaneously. Experimental evaluations confirm that this approach outperforms conventional NOMA/OMA systems in terms of spectral efficiency, latency, and reliability. The proposed solutions enable self-optimizing, intelligent 5G networks that can efficiently support diverse and evolving service demands, providing both theoretical insights and practical foundations for next generation wireless communication systems.