This talk will focus on the emerging era of deep learning and its impact on natural language processing and other industries. In recent years, new neural network architectures, including self-attention models, have transformed the machine learning landscape, challenging many of the fundamentals and techniques that have been traditionally applied. High-capacity models and transfer learning methods now allow us to extract knowledge from supercomputing centers and adapt it to specific industry needs, achieving higher levels of accuracy than traditional machine learning algorithms.

While models like GPT and T5 are already visible examples of this disruption, they are just the beginning. Understanding and transforming these models will be a key factor in the years to come. However, incorporating these neural network architectures into existing systems presents a significant challenge. We must ensure that these models are integrated into companies’ data centers and computing infrastructures in a secure and efficient manner.

In this presentation, we will demonstrate how to incorporate these models into big data infrastructures using tools from the open ecosystem created around these artificial intelligence architectures, such as Apache Spark, Scala, and BigDL. We will describe the process in detail and present a novel implementation of a distributed fine-tuning process for public pre-trained large language models. This implementation allows us to extract these models from public hubs and represent them as new Spark estimators. We will also gain insight into how to integrate these powerful new tools into existing systems and take advantage of the opportunities they provide.