Most of the state-of-the-art Deep Neural Network architectures processing sequence data (e.g., videos) like 3D Convolutional Neural Networks (3D-CNNs) and Transformers were built for offline processing. This means that, instead of processing newly captured input data one at a time, they require the entire sequence to be passed as a single input. Yet, many important real-life applications, such as tasks in Robot Perception and visual monitoring in Smart Cities, need online predictions on a continual input (visual) streams. While 3D-CNNs and Transformers can be applied by re-assembling and passing sequences within a sliding window, this is inefficient due to the redundant intermediary computations from overlapping clips. In this talk, I will present our recently introduced Continual Inference Networks (CINs) which are built to ensure efficient stream data processing. This is achieved by employing an alternative computational ordering that allows sequential computations without the use of sliding window processing. Concrete realizations of CINs are the Continual 3D-CNNs, the Continual Spatio-Temporal Graph Convolutional Networks (ST-GCNs) and the Continual Transformers, which require approximately L x fewer number of computations per prediction compared to sliding window-based inference with non-CINs, where L is the corresponding sequence length of a non-CIN network, while not sacrificing in accuracy.