Rotordynamics is the branch of applied mechanics that deals with various issues related to rotating structures. The shaft or rotor is a rotating module which acts as the major source of vibration in most of the machines. When two or more number of flexible shafts connected through coupling experience misalignment. The machineries incorporating these rotor-bearing systems are subjected to substantial misalignment-induced unbalance force and undergo vibrational instabilities. Failures related to misalignment-induced instabilities and the torsional-flexural coupled effect on the rotor-bearing system are considered to be as single largest cause for rotor failures and hence results in industrial downtime. Misalignment is a phenomenon where driving and driven shaft are not collinear, which causes one of the primary faults in the shaft system. Couplings are inevitable for long transmission shafts and compensates for misalignment. As the coupling is considered a flexible component in the power transmission system, it introduces a certain amount of mass, damping and stiffness to the system. The novelty of present work lies in studying the misalignment related issues, which throws multidimensional research of rotor shaft system. This dissemination first attempts to model a viscoelastic disc coupling, that may be used to join two or more flexible misaligned shafts. For which the flexible disc is segmented into multiple viscoelastic links stating both restoring and dissipating behavior. Operator-based constituted relationship is applied to incorporate the internal damping for coupling material. The flexible shaft is modelled using two techniques, i.e. lumped mass (analytical) and discretized method (finite element). Euler Bernoulli beam theory and operator-based constituted relationship is used to derive the equation of motion of viscoelastic shaft. The coupling parameters are integrated to the shaft with the help of the spring-mass mechanical analogy. A comparative study between analytical and finite element approach is carried out based on various dynamic parameters viz. stability limit of spin speed, Campbell plot and frequency/time response. This comparative study concludes that the lumped mass method is simpler and gives concise results with acceptable accuracy. The above research further extends to study the effect of flexible disc coupling in a misaligned propeller shaft. Considering the Timoshenko beam theory and operator based constitutive relationship, the FE equation of motion of the laminated composite viscoelastic shaft is obtained, and coupling characteristics are integrated. Since joining two or more number of shafts through the coupling produces long slender structure and exhibits nonlinear behaviour due to large deformations. Hence the theoretical study is extended to the geometrically nonlinear rotor dynamics, where loose rotating disc is integrated into the nonlinear imbalance rub pedestal looseness. A comparative study of frequency and time response behaviour between the linear and nonlinear rotor is presented here to identify various resonant conditions. It is evident from the results that the jump phenomenon occurs within specific frequency ranges, which is the distinctive hallmark of the nonlinear rotor system. The carried out research work presents mathematical models that can be applied to any rotor shaft systems subjected to misalignment, misalignment induced instabilities and long shafts joined by coupling exhibiting geometrically nonlinearity.