We are looking for a skilled Python Web Scraping Engineer to develop and maintain robust web-scraping agents that extract structured data from various websites and store it in our PostgreSQL database. The ideal candidate should have experience with Beautiful Soup, Selenium WebDriver, and knowledge of Python scripting. You will be working with a GitHub-based repository, scheduling agents to run automatically or on demand.
Remote, full-time
Working Schedule: flexible Responsibilities: * Develop data-scraping agents using Python, Beautiful Soup, and Selenium WebDriver * Extract structured data from websites and store it in PostgreSQL * Write clean, efficient, and reusable code, following version control best practices (Git/GitHub) * Configure scraping agents to run on schedule (cron or job scheduler) or trigger them on-demand * Monitor, debug, and maintain existing scraping scripts to ensure data accuracy and stability * Handle anti-bot mechanisms (CAPTCHAs, dynamic loading, cookies, user-agents) where applicable * Document data collection processes, endpoints, and script logic clearly * Collaborate with the data and backend teams to ensure smooth data integration
Requirements: * Proven experience with Python 3.x scripting * Strong proficiency in Beautiful Soup for HTML parsing and data extraction * Experience with Selenium WebDriver for dynamic page interaction and automation ( would be a plus) * Hands-on experience with PostgreSQL (writing inserts/updates, handling schema) * Familiarity with Git/GitHub for version control and code collaboration * Understanding of HTML, CSS, and browser DOM structures * Ability to write modular, testable, and well-documented code * Good problem-solving and debugging skills * English: Upper-Intermediate
