Rolecall combines job postings from multiple sources into one dataset. It normalizes fields, extracts technical skills, converts salary ranges to annual USD estimates, and merges likely cross-posts while retaining source provenance.
The bundled JSON sources make the pipeline deterministic and runnable offline.
python -m venv .venv
source .venv/bin/activate
pip install -e .
jobintel report examples/board_a.json examples/board_b.jsonExample output:
5 raw postings -> 4 unique (1 cross-posted)
Top skills: Python 4, Docker 3, PostgreSQL 2, Pandas 2, FastAPI 1
Remote share: 50%
The sample data includes the same Acme job under slightly different company and title spellings. Rolecall merges the records and adds the second source to also_seen_on.
source records
-> common schema
-> salary and skill enrichment
-> fuzzy duplicate matching
-> aggregate report
salary.py handles annual, monthly, and hourly compensation. It recognizes examples such as $120k-$150k, £65,000 pa, $60/hour, and R$ 8.000/mês, then converts the range with explicit currency rates.
skills.py uses a curated taxonomy and aliases such as postgres for PostgreSQL and k8s for Kubernetes. Word-boundary matching prevents short terms from appearing inside unrelated words.
dedup.py compares normalized company names and titles with RapidFuzz. A match must clear the configured threshold on both fields. Merged records keep the richer salary and skill data plus the original source references.
pytest -qThe 18 tests cover salary formats, currency conversion, skill aliases, boundary matching, duplicate thresholds, provenance, and the complete pipeline.
Prefer official APIs and feeds. When a site permits scraping, request public job listings at a conservative rate and retain source attribution. Rolecall's schema does not need applicant or recruiter personal data.