In this paper, a concept of vision-based vibration source localization to extract vibration image regions using pixel-level digital filters in a high-frame-rate (HFR) video is proposed. The method can detect periodic changes in the audio frequency range in image intensities at pixels of vibrating objects. Owing to the acute directivity of the optical image sensor, our HFR-vision-based method can localize a vibration source more accurately than acoustic source localization methods. By applying pixel-level digital filters to clipped region-of-interest (ROI) images, in which the center position of a vibrating object is tracked at a fixed position, our method can reduce the latency effect on a digital filter, which may degrade the localization accuracy in vibration source tracking. Pixel-level digital filters for 128×128 ROI images, which are tracked from 512×512 input images, are implemented on a 1000-frames/s vision platform that can measure vibration distributions at 100 Hz or higher. Our tracking system allows a vibrating object to be tracked in real time at the center of the camera view by controlling a pan-tilt active vision system. We present several experimental tracking results using objects vibrating at high frequencies, which cannot be observed by standard video cameras or the naked human eye, including a flying quadcopter with rotating propellers, and demonstrate its performance in vibration source localization with sub-degree-level angular directivity, which is more acute than a few or more degrees of directivity in acoustic-based source localization.