Image set matching attracted increasing attention in the field of pattern recognition. Recently, there are a number of effective image set-based matching methods under controlled environment. However in the more complex environment, like multi-view and illumination changed, it is still a challenging