Innovations, Challenges and Emerging Technologies in Data Engineering

Dear Colleagues,

In the age of big data, the field of data engineering has become a cornerstone of technological progress and innovation. The unprecedented growth of data generated from various sources such as digital platforms, social media, IoT devices, and a variety of other digital channels has brought the need for sophisticated data engineering practices to the forefront of modern computational challenges. As we explore this area, we recognize the enormous potential but also the numerous challenges that come with the rapid development of data-driven technologies. These advancements are not only transforming the industry, but are also shaping the future in unprecedented ways. The increasing need for advanced data engineering solutions emphasizes the importance of this area in today’s digital landscape.

This Special Issue, entitled "Innovations, Challenges, and Emerging Technologies in Data Engineering" is dedicated to the latest advances, current challenges, and emerging technologies in data engineering. This topic represents the natural continuation and improvement of the First Edition’s subject “Advanced Web Applications”, in which data engineering has its foundations, and this progression demonstrates a natural and logical evolution of topics, each building upon the other.

Our goal is to provide a comprehensive overview of the current state of data engineering as well as insights into the latest developments and discussions about the challenges that professionals and researchers are facing in this rapidly evolving field.

This Special Issue seeks contributions on the development and innovative application of data engineering technologies, as well as data engineering analysis, design, implementation, evaluation, and training. We welcome high-quality case studies that describe the successful implementation of data engineering projects, as well as research papers that propose new approaches and techniques for solving traditional and emerging data engineering challenges. We welcome submissions that provide insights into innovative data engineering practices as well as solutions and strategies for overcoming complexity and harnessing data's potential in various domains.

The topics covered include, but are not restricted to, the following:

• Data platforms, cloud platforms, building data pipelines.
• Data lake architecture and management.
• Data democratization.
• Data sources, transferring on-premises data to the cloud, fetching from relational or NoSQL databases and public sources, data ingestion.
• Using Pub/Sub agents for message forwarding and data integration, metadata and data provenance.
• Data consolidation and transformation, message schema control, building schema catalogs, and addressing message schema evolution problems.
• Using software containers, managing Kubernetes clusters and Docker containers.
• Creating APIs in the cloud and serving via message forwarding agents.
• Specific technologies including, but not exclusively restricted to, GO and Python programming languages, GIT, Jenkins, Terraform, Swagger, and Avro message formats.
• Applying DevOps concepts in code development.
• Establishing development, testing, staging, and production environments in the cloud.
• Advanced analytics and machine learning in data engineering.
• Real-time analytics and stream processing.
• Case studies on successful data engineering projects.

Dr. Marko Horvat
Prof. Dr. Igor Mekterović
Dr. Ljiljana Brkić
Prof. Dr. Ping-Feng Pai
Dr. Guanfeng Liu
Guest Editors

Manuscript Submission Information

Manuscripts should be submitted online at www.mdpi.com by registering and logging in to this website. Once you are registered, click here to go to the submission form. Manuscripts can be submitted until the deadline. All submissions that pass pre-check are peer-reviewed. Accepted papers will be published continuously in the journal (as soon as accepted) and will be listed together on the special issue website. Research articles, review articles as well as short communications are invited. For planned papers, a title and short abstract (about 100 words) can be sent to the Editorial Office for announcement on this website.

Submitted manuscripts should not have been published previously, nor be under consideration for publication elsewhere (except conference proceedings papers). All manuscripts are thoroughly refereed through a single-blind peer-review process. A guide for authors and other relevant information for submission of manuscripts is available on the Instructions for Authors page. Electronics is an international peer-reviewed open access semimonthly journal published by MDPI.

Please visit the Instructions for Authors page before submitting a manuscript. The Article Processing Charge (APC) for publication in this open access journal is 2400 CHF (Swiss Francs). Submitted papers should be well formatted and use good English. Authors may use MDPI's English editing service prior to publication or during author revisions.

• data engineering
• DevOps
• data platform architecture
• cloud computing
• data pipelines
• data ingestion
• data provenance
• real-time analytics
• stream processing