Scrapipy is a Python web-scraping project aimed at JavaScript-heavy websites. Selenium handles browser rendering, which helps the scraper access content that may not be present in the initial HTML response. The collected content is then passed through a LangChain-based workflow for AI-assisted extraction and summarisation. A Streamlit dashboard gives users a place to run the process, inspect the output, and export it to CSV or Excel. The project brings together browser automation and language-model tooling, while still leaving room for users to review the extracted information rather than treating generated output as unquestionable fact.
The problem
Traditional scraping can miss information when a website builds its page in the browser with JavaScript. Even after the page is loaded, raw HTML is not always the format people want to work with. Users may need selected fields, a short explanation, or a structured file rather than a large block of page content.
The solution
I used Selenium with a Chromium-based browser driver to render dynamic pages and collect their content. LangChain connects the extracted material to an LLM workflow for turning it into structured information and concise summaries. I used Streamlit to provide an interface for running the workflow, reviewing the output, and exporting data for later analysis.
What it does
- Render dynamic websites using Selenium and a Chromium driver
- Collect page content after browser-side rendering
- Use an LLM workflow for extraction and summarisation
- Provide a Streamlit dashboard for running and reviewing jobs
- Export collected information to CSV and Excel
- Turn page content into more structured output for analysis
What was hard
Dynamic pages do not all load at the same pace, and a browser that starts scraping too early may capture incomplete content. I had to account for page readiness and browser automation before sending the extracted material into the language-model workflow. Another consideration is output quality: an LLM can misinterpret or omit details, so extracted information should be reviewed when accuracy matters. Keeping the scraping step and the interpretation step separate makes the workflow easier to reason about and debug.
Results
Scrapipy connects browser automation with AI-assisted analysis in one workflow. It is intended to reduce the manual effort between visiting a dynamic page and turning the information on it into something that can be reviewed, compared, or reused.
