Advanced mathematics problem MATLAB solution