In realtime scenarios, a model must operate within a strict time constraint. For example, in a mobile augmented reality (AR) application, the model must generate predictions faster than the required frame rate to keep pace with incoming images. Traditional batch prediction, which leverages parallel processing, is ineffective here: as soon as the next image arrives, the previous prediction loses its relevance.
