What is data ingestion?
Data ingestion is how data gets moved from its source systems into somewhere useful, reliably and with proper error handling. It's the first point where your organization becomes responsible for the accuracy, traceability, and security of data it didn't create.
The mechanics are not glamorous, but they are well understood; connect to a source, log in, pull the records or files, handle updates as they come in, remove duplicates, adapt when the source changes format, and store the result somewhere usable. Doing this reliably across dozens of systems is most of what a data engineering team does.
What's changed isn't the mechanics. It's the cost of getting them wrong. When ingestion only fed dashboards, a broken pipeline produced a wrong number, and someone eventually noticed when they checked it. When ingestion feeds AI agents, that same broken pipeline leads to an action taken on a false premise, and nobody catches it, because the agent sounded confident.
Ingestion failures surface far from their cause. This is the property that makes them expensive. A missing partition, a silently dropped record, or a schema change absorbed without complaint produces symptoms in a dashboard, a retrieval result, or an agent's behaviour weeks later and several systems away. Teams then debug the model, the prompt, or the retrieval strategy, none of which is the problem.
What are the main data ingestion patterns?
Five: batch, micro-batch, streaming, change data capture, and hybrid. The choice is driven by how stale the consumer can tolerate the data being, and by what load the source system can carry without being damaged, rather than by architectural preference or by what sounds most modern.
The honest default is batch, because streaming adds state, ordering concerns, and considerably harder testing. Adopt it when latency genuinely requires it rather than because it sounds more modern.
| Pattern | How it works | When it fits | How it fails |
|---|---|---|---|
| Batch | Scheduled bulk movement, typically hourly or daily | Consumers tolerate delay; simplest to operate | Partial reads and writes that look healthy but drop segments |
| Micro-batch | Frequent small batches, minutes rather than hours | A middle ground where streaming is not justified | Cost rises with frequency, often unnoticed |
| CDC | Reads the database transaction log for inserts, updates and deletes | Databases needing freshness without repeated full scans | Replication lag and schema evolution become first-class problems |
| Streaming | Continuous event-by-event delivery | Responses must reflect events seconds old | Ordering, duplicates, and state make testing considerably harder |
| Hybrid | Change capture for freshness plus periodic snapshots for correction | You need both current state and a way to recover from drift | Two paths to reconcile, so discrepancies need a resolution rule |
Change data capture is highlighted because it is the pattern most often under-used and most relevant once agents are involved. Reading the transaction log gives you inserts, updates and, importantly, deletes, without scanning whole tables repeatedly. That last capability matters more than it appears, for reasons the trust-boundary section covers.
How does ingestion change when an agent is the consumer?
Five things get stricter. Freshness has to be tracked per source, not on one fixed schedule. Permissions must travel with the data itself, not get applied later in a warehouse. Document structure has to survive parsing intact. Where the data came from must be preserved. And deletions have to flow through the whole system, something analytics pipelines rarely guarantee at all.
None of these is new as a concept. What is new is that each one moves from being desirable to being load-bearing, because an agent acts on what it retrieves without a person checking first.
| Requirement | Feeding analytics | Feeding an agent |
|---|---|---|
| Freshness | One schedule across sources is usually acceptable | Per-source targets. A six-hour batch means six-hour-old context, however fast the model answers |
| Permissions | Access control applied later, in the warehouse | Access rules must travel with the record, because retrieval happens on a user's behalf at inference time |
| Structure | Rows and columns; layout is irrelevant | Meaning lives in layout, tables, and reading order, so flattening a document destroys it |
| Provenance | Useful for lineage and debugging | Required, because an answer without a source cannot be checked or defended |
| Deletion | Rarely urgent; stale rows are visibly stale | Content removed at source but still indexed is retrieved as current evidence |
The permissions row is the one that most often forces a redesign. Analytics pipelines assume access control is somebody else's problem, applied at query time in the warehouse. An agent retrieving on a user's behalf needs the retrieval layer itself to know who may see each item, which means access metadata has to be captured at ingestion and carried through parsing, chunking, and indexing. Retrofitting that is considerably harder than building it in.
Why is ingestion a trust boundary?
Because it is where content you did not write enters a system that will later treat it as instruction. Two failures follow from that: material gets in which should not, and material stays in which should have been removed. Both present as confident, sourced answers.
This framing is largely absent from ingestion literature, which treats the subject as a quality and freshness problem. For agents it is also a security problem, and the two failures below are mirror images of each other.
What gets in that should not
Every document ingested for retrieval is content an agent may later read as though it were direction. A support ticket, a shared drive file, a scraped page, or a supplier's PDF can carry instructions aimed at the model rather than at a person. Indirect prompt injection arrives through exactly this path, which makes ingestion the earliest place to do anything about it, and the place almost nobody does.
Two consequences worth designing for. Content should carry a trust label from the moment it enters, distinguishing material authored internally under review from material that arrived from outside. And relevance is not trust: a retrieval system ranks by similarity, and an attacker who wants their content retrieved writes content that is highly relevant. See context engineering for what happens to this material once it reaches the window.
What should have been deleted, but was not
This is the quieter failure, and the more common one. Ingestion pipelines are built mainly to handle new and updated records. Deletions are handled last, tested least, and often not carried through the system at all, because in an analytics context, one extra row is a minor annoyance.
For an agent, it's not minor. A document deleted at the source but still sitting in the index gets pulled up and presented as current, sourced fact. That could be a policy that's been replaced, a price that's been withdrawn, a customer record someone asked to have erased, or a contract that's ended. Content that should have been deleted is worse than content that's simply outdated, because outdated data at least shows up in a timestamp. A deletion that never propagated leaves no trace at all.
How do you design ingestion for AI agents?
Six steps, in dependency order: set freshness targets per source, capture permissions and provenance at the boundary, preserve structure through parsing, keep a replayable raw copy, make deletion a first-class path, and instrument the data itself rather than the job that moved it.
The ordering reflects dependency. Anything not captured at the boundary cannot be recovered downstream without re-pulling from source, which is why the middle steps matter disproportionately.
-
Set freshness targets per source, not one schedule
A single interval across all sources produces either unnecessary cost or unacceptable staleness, usually both in different places. High-change operational systems may need change capture in near real time; static document repositories are fine on a daily sync. Write the target down as a service level for each source, so lag becomes measurable rather than assumed.
-
Capture permissions and provenance at the boundary
Record who may see each item, where it came from, when it was last modified, and under what authority it was collected, and carry all of it through parsing, chunking, and indexing. This is the step that cannot be retrofitted, because the metadata exists at the source and stops existing the moment you drop it.
-
Preserve structure through parsing
For documents, meaning lives in layout, reading order, tables, and embedded objects. Flattening a PDF to a wall of text destroys the relationships an agent needs to answer correctly. Normalize to a consistent structured representation across file types rather than accepting whatever each parser produces.
-
Keep a replayable raw copy
Store the original alongside the normalized output with stable identifiers. When your chunking strategy changes, and it will, you reprocess from your own copy rather than re-pulling from every source system. This one decision is the difference between a strategy change taking an afternoon and taking a quarter.
-
Make deletion a first-class path
Design and test propagation of deletes with the same seriousness as inserts, including from the vector index. Change data capture helps because the transaction log carries deletions; document sources usually need explicit reconciliation. Then test it, deliberately and repeatedly.
-
Instrument the data, not just the job
A job that completes successfully can still deliver nothing useful. Monitor freshness lag per dataset, completeness against expected partitions, and schema drift, and alert on the data rather than on the exit code.
How does ingestion fail, and how would you know?
Predictably, and almost always silently. Sources change format, connectors drop records, retries create duplicates, timestamps arrive in mismatched formats, and deletions never make it through. None of this reliably triggers an error. That's why catching it has to be built in on purpose, not assumed to happen on its own.
The failures below are ordinary distributed-systems edge cases. Treating them as exceptional is what causes teams to relearn the same incident every few months.
- Partial success. A batch job completes, having read some of what it should. It reports healthy because it did not crash. Compare expected against observed partitions or the missing data looks like legitimate silence.
- Silent schema drift. A source adds, removes, or retypes a field. Downstream either absorbs it and produces subtly wrong output, or fails somewhere unrelated. Detect and route changes through review rather than adapting automatically.
- Duplicates from retries. A retry re-delivers records already written. In analytics this inflates a count. In a retrieval index it means the same passage is retrieved several times, crowding the context window with one document.
- Lag nobody is watching. Freshness degrades gradually rather than breaking. Without a per-dataset lag metric and a stated target, the first report is a user noticing an agent gave outdated information.
- Deletes that never arrive. Covered above, and worth repeating as a monitoring item rather than only a design one. Reconcile index contents against source periodically, not just on change events.
The common thread is that every one of these is invisible to job-level monitoring and visible in data-level monitoring. If your alerting tells you pipelines ran and not whether they delivered current, complete, correctly shaped data, you will find out about ingestion problems from whoever is using the output. For what happens when that output reaches an agent, see data intelligence and AI agent.