Nonlinear vibration control systems (both passive and active) always involve parameter design and performance optimization tasks. A systematic and novel frequency domain method is established to this aim in this study based on a newly developed concept --- nonlinear characteristic output spectrum (n