Fundamentals of Data Engineering
About this book
Reis and Housley describe the data engineering lifecycle, which runs from generation through ingestion, storage, transformation and serving, with governance, security and architecture cutting across all of it. The book is deliberately tool agnostic, so it explains the trade offs between batch and streaming, between different storage abstractions, and between building and buying, rather than teaching a particular vendor's stack. There are chapters on source systems, on queries and data modelling, and on serving data for analytics, machine learning and operational use. It closes with material on security, privacy and where the field appears to be heading. Written for people entering data engineering and for the product and analytics people who depend on them.
Description via Product Digest.
Topics
Related concepts
- Privacy and consent in analyticsWhich facts inside an event property count as personal data, the lawful basis and the four tests a consent has to pass, how long a record may be kept, and the arithmetic showing what a declined consent rate does to every funnel and cohort the product reports.
- The machine learning lifecycleThe stages between a business question and a model in production, why the work does not stop at deployment, and where each cloud vendor's tooling sits along the line.
- Data quality in analyticsThe six failures that quietly stop collected data matching what happened, from an event that never fires to a field renamed in a release, the checks that catch each one, reconciliation against a system of record, and the honest way to report a loss that cannot be recovered.
- Events and propertiesThe event model a product analytics practice is built on, the three kinds of property and why the difference between them decides what a team can ask later, an object action naming convention, and the choice between many specific events and few general ones carrying properties.
- Data for generative AIWhat makes data usable by a model, the difference between structured and unstructured data and between labelled and unlabelled data, and why accessibility decides more projects than volume does.
- The tracking planThe document that holds every event a product records, the columns each row carries, the review a proposed event goes through, how a plan is versioned and how an event is retired, and why a plan that exists only inside the analytics tool has no owner at all.
- The AI supply chainWhat a team is actually depending on when it ships an AI feature, how those dependencies fail, and the provenance work that has to happen while it is still boring.
- Securing an AI systemAccess control, encryption and the shared responsibility model applied to machine learning, plus the attack surfaces that only exist because there is a model in the path.