Source ingestion system
The ingestion layer creates candidates. It does not overwrite trusted festival records automatically.
scripts/ingest_candidates.py writes only to data/candidates/; human review is required before a candidate enters data/festivals.json.
1. Source discovery
Find official organizers, municipal pages, tourism boards, regional calendars and specialist directories.
2. Candidate extraction
Extract only candidate name, town, region, raw date, food theme and source URL. Keep uncertainty visible.
3. Date normalization
Normalize dates only when the source gives enough information. Seasonal clues stay seasonal-pattern.
4. Deduplication
Merge exact duplicates. Flag near-duplicates in validation notes.
5. Confidence assignment
Official organizers: official. Tourism boards: tourism-source. Directories: directory-source. Everything else: unconfirmed.
6. Publishing rules
Publish only when a source URL is present or the record is clearly marked for verification. Never upgrade a directory date without a stronger source.