In this study, we develop a high-speed color-histogram-based tracking system that can be applied to 512x511 pixel images at 2000~fps using the hardware implementation of an improved CamShift algorithm. In the improved algorithm, the size, position, and orientation of an object to be tracked can be extracted using only the hardware implementation of color conversion and the moment feature calculation of 16 binary images quantized by color bins on a high-speed vision platform. We demonstrate its effectiveness by presenting several target tracking results at 2000~fps when color-patterned objects move rapidly under complicated backgrounds.