Described is a technology by which video, which may be relatively high-resolution video, is efficiently processed to determine whether the video contains a specified action. The video corresponds to a spatial-temporal volume. The volume is searched with a top-k search that finds a plurality of the most likely sub-volumes simultaneously in a single search round. The score volumes of larger spatial resolution videos may be down-sampled into lower-resolution score volumes prior to searching.