In this study, we develop a real-time, color histogram-based tracking system for multiple color-patterned objects in a 512×512 image at 2000 fps. Our system can simultaneously extract the positions, areas, orientation angles, and color histograms of multiple objects in an image using the hardware implementation of a multi-object, color histogram extraction circuit module on a high-speed vision platform. It can both label multiple objects in an image consisting of connected components and calculate their moment features and 16-bin hue-based color histograms using cell-based labeling. We demonstrate the performance of our system by showing several experimental results:(1) tracking of multiple color-patterned objects on a plate rotating at 16 rps, and (2) tracking of human hand movement with two color-patterned drinking bottles.