Web Scraping for Lead Generation
- Python
- Selenium
- →~5,000 clinic leads collected across India and Southeast Asia
- →Per-site HTML parsing across directories and competitor sites
- →Proof of concept; paused before rollout in a reorg
Context
Sales reps were spending a lot of time just finding clinics to approach, mostly by hand and without much direction. I was asked to build a proof of concept to test whether scraping could produce better-targeted lead lists: relevant clinics in a rep's region, with enough supporting detail to shape the outreach.
Building the scraper
I built it in Python with Selenium, driving Microsoft Edge. Each target site needed its own logic: I worked through the page structure and wrote extraction rules for clinic name, contact details, location, and the dentist's specialisation. Beyond general directories, I pointed it at competitor websites too, so the sales team could see which clinics were already aligned with a rival and either skip them or change their pitch.
Result
The scraper pulled roughly 2,600 leads from India and 2,400 from Southeast Asia, including about 2,000 qualified dentists in tier-one Indian cities and another 2,000 usable leads in Indonesia. It clearly could have scaled, but a reorganisation paused the project before it was formally integrated or measured against actual conversions.
What I'd improve
A scraper like this is brittle by nature: each site can change layout without warning. A production version would need monitoring, a scheduled refresh, and a check on each source's terms of use before it ran unattended.