Mar 26, 2025

I track enterprise software application development & data management. Follow Author Share getty Almost all data moves. Although there is a formalized classification of “data at rest” to denote those information resources that are not on the move between applications, databases, analytics engines and other service components, most data at some point in its life is moved around… and some of it is transmitted in real-time. But data movement is now becoming a technology classification in and of itself. This is in no small part down to the rise of artificial intelligence and the need to feed AI models with information spanning large and small language models, alongside regular systems data that AI needs to operate and interconnect with. Logically then, moving data is risky. This is because data in transport doesn’t necessarily break as such (although connection between different components of a given software system can lead to corruptions), we’re more concerned with making sure data on the move doesn’t end up in the wrong place and isn’t sent to machine entities that don’t have access policy rights. Managing Data Pipelines Airbyte is a company that describes itself as an open source data movement platform specialist. Its technology works to move data at scale for AI and analytics workloads, while ensuring governance so that organizations spend less time managing data pipelines while unlocking value from data. “We make it possible for organizations to protect their data, ensuring that it doesn’t accidentally become accessible outside the organization by consumers of AI models,” said Michel Tricot , co-founder and CEO, Airbyte. “[The technology] MORE FOR YOU we’re delivering helps eliminate data silos and improves data accessibility, while still ensuring security and compliance to maintain data sovereignty without adding operational overhead.” The company is now updating its products designed to provide organizations with additional support for data movement while retaining sovereignty over their own first-party data with enhanced security, speed and enhanced resource management. This includes support for unstructured data and portable data lake formats, which are repositories typically used for the mass storage of raw data streams (think of them more like a holding pen than a dumping ground) before they become managed and structured for onward use. In technology terms this includes support for the Apache Iceberg open standard for moving data into modern lakehouse architectures, which are the backbone for AI workloads with large language models as well as modern analytics. The company is also offering file transfer support for Google Drive, SharePoint and OneDrive for the movement of unstructured data such as PDF, video and image files (along with their metadata and permissions), making all of this data accessible for AI. Tricot and team say that they are making moving data easy and affordable across nearly any source and destination, ensuring enterprises have accurate, timely data for analysis and decision-making. With over 900 contributors and a community of more than 230,000 members, Airbyte supports a large data engineering community that works with an industry-wide open data movement platform. What Is Lightweight Data Transformation? A new Airbyte Mappers feature enables users to perform “lightweight data transformations” (comparatively simple changes to data resources such as deduplicating, formatting, parsing or restructuring to achieve data consistency when different information resources are dovetailed or integrated, or simply for good housekeeping) directly within the Airbyte interface. This transformation can also include hashing (changing all data fields to a fixed length, typically a shorter one), encrypting, renaming fields and filtering rows to help organizations maintain compliance with data privacy regulations like GDPR and HIPAA. Airbyte is also providing an enterprise connector bundle as a complement to its Airbyte Cloud Teams and Self-Managed Enterprise brands. The bundle includes connectors for NetSuite, Oracle database with Change Data Capture, SAP HANA, ServiceNow and Workday. “This bundle of connectors streamlines how the world’s largest organizations access their most valuable financial, operational and human resource data. The Airbyte Enterprise products ensure that organizations can easily and securely extract critical data from complex and sensitive sources with governance controls for data privacy and compliance,” said Tricot. The Data Movement Movement Looking ahead, we may see the data movement movement extend and finesse itself in the same way that house removal businesses do. We can hire a major removal firm for a whole house move, or we can hire a “man with a van” for smaller tasks. We can hire a down and dirty cheap low-end service to lump concrete or broken down wall materials around, or we can hire bespoke white glove services that promise to transport our porcelain tea sets all in one piece and make sure they get to their intended destination. Either way, whether its data or homewares, make sure you lay down a dust sheet.