Recently, there has been research on high frequency dissipative mufflers. However, research on shape optimization of hybrid mufflers that reduce broadband noise within a constrained space is sparse. In this paper, a hybrid muffler composed of a dissipative muffler and a reactive muffler within a constrained space is assessed. Using the eigenvalues and eigenfunctions, a coupling wave equation for the perforated dissipative chamber is simplified into a four-pole matrix form. To efficiently find the optimal shape within a constrained space, a four-pole matrix system used to evaluate the acoustical performance of the sound transmission loss (STL) is evaluated using a genetic algorithm (GA). A numerical case for eliminating a broadband venting noise is also introduced. To verify the reliability of a GA optimization, optimal noise abatements for two pure tones (500 Hz and 800 Hz) are exemplified. Before the GA operation can be carried out, the accuracy of the mathematical models has been checked using experimental data. Results indicate that the maximal STL is precisely located at the desired target tone. The optimal result of case studies for eliminating broadband noise also reveals that the overall sound power level (SWL) of the hybrid muffler can be reduced from 138.9 dB(A) to 84.5 dB(A), which is superior to other mufflers (a one-chamber dissipative and a one-chamber reactive muffler). Consequently, a successful approach used for the optimal design of the hybrid mufflers within a constrained space has been demonstrated.