The orchestration of Machine Learning frameworks with datastreams and GPU acceleration in Kafka-ML: A deep-learning performance comparative

Abstract

Machine Learning (ML) applications need large volumes of data to train their modelsso that they can make high-quality predictions. Given digital revolution enablers suchas the Internet of Things (IoT) and the Industry 4.0, this information is generated inlarge quantities in terms of continuous data streams and not in terms of staticdatasets as it is the case with most AI (Artificial Intelligence) frameworks. Kafka-ML isa novel open-source framework that allows the complete management of ML/AIpipelines through data streams. In this article, we present new features for the Kafka-ML framework, such as the support for the well-known ML/AI framework PyTorch,as well as for GPU acceleration at different points along the pipeline. This pipelinewill be described by taking a real Industry 4.0 use case in the Petrochemical Industry.Finally, a comprehensive evaluation with state-of-the-art deep learning models willbe carried out to demonstrate the feasibility of the platform.