A scholarship ingestion pipeline that dedupes across three sources

Most scrapers are a script. This one is a small pipeline with the seams in the right places, because scholarship listings change often and a flaky source should not take the whole run down.

Repo: swadhinbiswas/fundmystudy-scraper

It is a Node.js service that pulls opportunities from public sources and submits them to the FundMyStudy API for moderation. Three sources ship today: the DAAD scholarship database, the Erasmus Mundus catalogue, and Chevening programmes.

Shape of the code

Each source implements one interface with a name, a cron schedule, a URL and an async fetch() that returns a list of raw listings. Adding a source is a new file and one line in the registry, plus its name in the SOURCES environment variable.

scheduler -> runner (per source) -> normalize -> dedupe -> API client

The scheduler is a thin node-cron wrapper. The runner walks each enabled source through the pipeline. The normalizer handles the shared cleanup. The deduper keys on external_id, then URL, then a normalized title plus provider, which is what stops the same scholarship arriving by two routes from becoming two rows. Every stage emits counters, so a run reports what each source produced and what was dropped.

Playwright is in the mix because some of these catalogues render client-side and a plain HTTP fetch returns an empty shell. It downloads Chromium on first install, which is worth knowing before you run it in CI.

Running and testing

npm run typecheck
npm run run:once
npm run test:source -- daad

The test:source script fetches and normalizes one source and prints the result without posting anything, and run-once takes a single source end to end. That split is the part I use most, because debugging a parser against a printed listing is faster than watching a full run.

What I would carry forward

Two habits made this maintainable. Every source is isolated behind the same interface, so a selector change or a 429 is contained to one module and one retry policy. And dedupe keys are explicit and ordered, because "is this the same listing" is a data-model question, not a string comparison you invent at the last minute.

The repository above has the source implementations and the moderation API client.

Reach me at swadhinbiswas.cse@gmail.com or on GitHub and LinkedIn.