With the growing demand for designing lightweight, durable, and efficient structures in advanced industries such as aerospace, renewable energy, and automotive engineering, precise analysis of the dynamic behavior of structures like thin-walled beams with variable cross-sections has become increasingly important. This study investigates the vibration and dynamic stress analysis of thin-walled beams with varying cross-sections using the Rayleigh–Ritz method. To accurately model the beam geometry, polynomial functions are employed to define the variation of thickness and diameter along the beam’s length. Utilizing the Rayleigh–Ritz approach, the mass and stiffness matrices are semi-analytically derived, and the governing vibration equations are formulated. Subsequently, the dynamic response of the beam under distributed sinusoidal loading is computed, and the bending stress distribution along the beam is evaluated. For geometry optimization aimed at simultaneously reducing the maximum stress and structural weight, the NSGA-II genetic algorithm is applied, yielding a set of optimal designs on the Pareto front. The results demonstrate that an appropriate design of thickness and diameter functions, assisted by the optimization algorithm, can effectively reduce both dynamic stresses and overall weight. Moreover, comparison of the numerical responses with results obtained from the finite element software ABAQUS shows strong agreement, validating the proposed method. This approach can serve as an effective tool for designing lightweight and robust structures in applications such as aerospace, renewable energy systems, and the automotive industry.