There is a list of software companies in Bangladesh that people keep asking for, and every version of it I found was a spreadsheet that aged badly. So I built one where the data is the source of truth and everything else is generated, including the README.
Repo: swadhinbiswas/software-companies-in-bangladesh
The rule
The README says, in a comment at the top, AUTO-GENERATED FILE - DO NOT EDIT. If you want to change a company, you edit a TOML file under data/ and open a pull request. That one rule is what keeps 140 entries consistent: no one can quietly fix a row by hand and drift from the canonical record.
The layout
The source data is a small set of TOML files: companies.toml for the rows, schema.toml for the allowed company types and technologies, and ignore.toml for exclusions. There is an info.json and a job-posts.json feed alongside.
Everything downstream is derived, and it lands in data/gold/ as a set of small JSON files, one per question someone might ask:
companies.json company_tech.json employment_breakdown.json
jobs_per_company.json location_heatmap.json recent_jobs.json
salary_stats.json tech_demand.json stats.json
That split between raw and gold is the whole point. The TOML is curated and reviewed, and the gold files are rebuildable. When a new technology type shows up, I add it to schema.toml and the aggregates follow.
The pipeline
A warehouse/ directory holds schema.sql, a build.py that turns the source into the gold tables, a dataset card template, and an hf_push.py that publishes the result as a Hugging Face dataset. A tools/ directory holds a Rust crate, and there is a crawl.sh for collecting job pages plus a dashboard/ for the public view.
jobs.md is generated from the same data and currently reflects openings from 140 companies. Because the jobs feed and the company list come from one source, they cannot disagree.
Why this is data engineering
Parsing job pages was the easy part. Designing a small schema that stays honest under contribution was the work. Types and technologies are constrained lists rather than free strings, so tech_demand.json and employment_breakdown.json are groupable without a normalization pass. The README and the dashboard are both outputs of the same build, so the thing people read first is never stale relative to the data.
It is the same discipline as any warehouse, at a scale one person can hold in their head. Curated source layer, constrained schema, generated aggregates, and a published artifact with a card that explains what it is.
The repository, the schema and the published dataset are linked above.
Reach me at swadhinbiswas.cse@gmail.com or on GitHub and LinkedIn.
No comments yet.