Campus information at a large university is scattered by design. The official calendar has official events. Departments post their own. Student organisations announce on social. Athletics is a separate system. A student who wants to know what is happening tonight has to check five places, so most check none.
UA Today checks them instead, on a schedule, and shows the result on one page: 6 sources, roughly 700 events at a time. Here is what turned out to be hard, and it was not the parsing.
The easy problem
Fetching and normalising six feeds is a day of work. Different date formats, different notions of what a location is, different ideas about whether a recurring event is one record or forty. Tedious, bounded, done.
The interesting problem is what happens on day ninety.
The real problem: a source that stops without failing
A source that returns an error is easy. You see it, you fix it.
A source that returns an empty list, with a 200 status and a well-formed response, is the one that hurts. Somebody renames a query parameter over winter break. The feed still answers. It just answers with nothing, and your site quietly shows fewer events, and nobody notices because a quiet week on campus looks exactly like a quiet week on campus.
This is the same failure family as almost every real bug I have written about this year: a canonical tag telling Google to skip nine pages, ten agents that were never being invoked while the runner exited zero. Nothing errors. Something is just missing, and absence does not raise.
Floor checks
The fix is to assert a minimum rather than the absence of an exception. Each source declares how many events it should plausibly produce, and returning fewer is a failure even when the request succeeded.
The floor is deliberately loose. It is not there to catch a slow week, it is there to catch a source that has gone to zero or near it while still answering politely. A health endpoint reports each source with its count and its response time, and the morning check reads that endpoint rather than trusting that the job ran.
Choosing the number is a judgement call, and the failure mode of choosing badly is worse than you would expect: a floor set too high fires on every quiet week, people learn to ignore it, and now you have a monitor that is worse than no monitor because you believe you have one.
Two smaller things that mattered more than expected
A moderation queue. Anyone can submit an event and nothing appears without a human looking at it. For a site with a university's name near it, the cost of one bad listing is much higher than the cost of a slower pipeline.
Saying it is unofficial, in the masthead. Not in a footer, not on a terms page. The first screen. It is a student project, it is not affiliated with the university, and event details are aggregated automatically and can be wrong. Being clear about that is the reason it is allowed to exist at all.
What I would do differently
Write the floor checks first. I added them after a source went quiet, which is the usual order and the wrong one. The monitoring for a pipeline is not a thing you add once the pipeline works. It is the part that tells you whether the pipeline works, and building it second means you spend the gap trusting a report you have not tested.
Also: an aggregator's real product is not the aggregation. It is confidence. A student uses it twice, finds something wrong once, and never returns. Everything expensive in the build was in service of the listing being right, not in service of there being more listings.
Common questions
How do you detect when a data source silently stops returning results?
Assert a minimum count rather than the absence of an error. A source that has been renamed or deprecated often still returns a well-formed response with a 200 status and an empty list, which no error handler catches. Give each source a loose floor for how many records it should plausibly produce and treat falling below it as a failure even though the request succeeded.
Why is a monitor that produces false alarms worse than no monitor?
Because people mute it, and a muted monitor leaves you believing you have coverage that you do not. A threshold set too aggressively fires on normal variation, users learn to ignore it within a week, and the real alert arrives into an audience that has already stopped reading.
What is the hardest part of building an event aggregator?
Not the parsing, which is bounded and tedious. It is detecting a source that quietly goes to zero while still answering politely, because the resulting site looks the same as a genuinely quiet week. The second hardest part is trust: an aggregator's product is confidence, and a user who finds one wrong listing does not come back.