Verify the source first
The Assemblée nationale documentation pointed to /api/v2/, but the endpoint no longer served the expected data. The durable source was downloadable archives. Before designing transformations, I learned to prove that the source exists, is current, and can be retrieved repeatedly.
Seven production lessons
- Documentation is evidence, not availability. Run a small retrieval test before building around an interface.
- Network round trips dominate naive inserts. At roughly 160 ms per request, inserting 6,000 rows individually took 16 minutes. Batching reduced it to eight seconds.
- Long-lived connections are not automatically reliable. A connection dropped around row 470,000. Short, bounded transactions were easier to retry.
- Local database behavior can hide hosted limits. PostgreSQL WAL growth exceeded a managed 1 GB storage tier. Reproducible ingestion made recovery possible.
- Mechanical fixes need tests. A broad search-and-replace accidentally made a retry wrapper call itself recursively.
- Production networking has its own topology. A direct Supabase connection on port 5432 was blocked while the pooler on 6543 worked.
- Scripts deserve explicit orchestration. Running standalone stages as subprocesses preserved isolation and clearer failure boundaries.
The common cause: I developed locally and assumed production would behave the same. Reliable data systems make environmental assumptions visible and testable.
Why this mattered for MonÉlu
MonÉlu needs traceable parliamentary evidence, not plausible text. The ingestion system therefore treats official records as reproducible inputs, preserves source identifiers, and keeps retrieval separate from answer generation.
Adapted from the original Medium publication.