Data engineering for AI
The unglamorous work that decides whether anything above will function. Pipelines, quality, lineage and the schema nobody wrote down.
The problem
Most stalled AI projects are not model problems. They are three systems holding contradictory versions of the same record.
What the work involves
- Mapping what you hold, where it lives and which copy is authoritative
- Pipelines that fail loudly instead of silently writing nulls
- Deduplication and entity resolution across systems
- PII identification and handling before anything reaches a model provider
- Lineage, so you can answer where a number came from
Typical stack
PostgreSQL
dbt
Airflow
Python
Pandas
DuckDB
Indicative rather than fixed. The right choice depends on what you already run, and inheriting a stack your team knows usually beats introducing a better one nobody can maintain.
Got a data engineering problem?
Technical detail welcome. The more concrete the problem, the more useful the first reply.