MATLAB implementation of number theory problem