Concept 5 of 7

Data for generative AI

4 questions test this

Every generative AI project is a data project first. The exam treats data as a business consideration rather than an engineering one, and the questions ask what a given quality problem will cost you.

The two divisions

Structured or unstructured is about schema. Structured data has defined fields with consistent meaning, which covers database tables, transaction records, sensor readings and anything with rows and columns. Unstructured data has none, which covers documents, email, images, audio, video and free text.

The split matters commercially because most of what an organisation knows is unstructured, and it was largely unusable until foundation models arrived. That is where the value in a gen AI project usually sits.

Labelled or unlabelled is about whether an answer is attached. A support ticket tagged with its resolution is labelled. The same ticket with nothing attached is unlabelled. Labels are expensive because people have to produce them, and they are what supervised learning needs.

Generative AI shifted this. A foundation model learned from vast unlabelled text, so many tasks that once needed a labelled dataset now need only a prompt.

What data quality means

The exam names six characteristics and each has a failure attached.

Completeness. Missing values in the fields that matter. A model cannot reason about what is absent, and gaps concentrated in one segment produce a system that works badly for that segment alone.

Consistency. The same thing recorded differently in different places. Dates in three formats, a customer under two identifiers, a product renamed midway through the archive.

Relevance. Whether the data describes the situation you are asking about. Five years of records from a market you have exited is volume without value.

Availability. Whether you can actually reach it. Data locked in a system nobody can export from is not available whatever it contains.

Cost. Acquisition, storage, movement and the licensing of anything bought in. This is a real constraint on retrieval systems, where storage and embedding scale with the corpus.

Format. Whether it arrives in a shape the tooling can use. A decade of scanned paper is a document processing project before it is an AI one.

Accessibility as the real constraint

Google's guide names availability and cost alongside quality, and in practice accessibility is the thing that stops projects. The data exists, and it sits behind a system with no interface, or in a jurisdiction the model cannot run in, or under an agreement that did not anticipate this use.

That last one recurs. Data collected for one purpose and used to train or ground a model is a different purpose, and consent frequently does not travel with it.

The business reading

The useful question before a gen AI project is not what model to use. It is whether the data that would answer the question exists, whether you may use it, whether you can reach it, and whether it is consistent enough to be worth reaching. A team that can answer those four has done the difficult part.

Common misconceptions

More data always produces a better result.

Volume helps only when the data is relevant, consistent and reachable. Ten years of records in a format nobody can query, or covering a market you have left, is a cost rather than an asset.

Generative AI removes the need to prepare data.

It removes the need for labels in many cases and nothing else. Quality, relevance, access rights and format all still decide what the system can do, and a retrieval index built on inconsistent documents returns inconsistent answers.

A spreadsheet counts as unstructured data because it is a file.

Structure is about whether the data has a defined schema, not what it is stored in. Rows and columns with consistent meaning are structured wherever they live. A scanned invoice is unstructured even inside a database.

4 questions test this concept

Which of these collections is unstructured data?

  • AA table of orders with columns for date, customer and amount.
  • BA warehouse of scanned supplier contracts and recorded support calls.
  • CA daily feed of sensor readings with a fixed schema.
  • DA product catalogue held in a relational database.
Check whether it stuck.

One per page, with a worked explanation.

Start the set
Related material
Book
Fundamentals of Data Engineering, On quality, access and the pipeline underneath any model.
Book
Thinking with Data, On asking what a dataset can actually answer.