Scraping a website into Excel without installing anything

A stdlib-only scraper: how to pick a row container, why one wrapper div breaks your selector, and how to do pagination and detail pages correctly.

Every scraping tutorial starts with pip install selenium. On a locked-down machine that is where the project ends. But most list pages are plain server-rendered HTML, and parsing them needs three things: an HTML parser, a selector, and the discipline to check you got rows before writing a file.

1 · Find the row, not the field

The mistake is starting from a field (span.price) and hoping you can line things up later. Start from the container that repeats — one card, one table row, one list item — and extract every field inside it. That is what keeps row 3's price with row 3's name.

--row "div.list > div.card"
--field "sku=@data-sku"           an attribute of the container itself
--field "name=h3.name@text"
--field "price=span.price@text"
--field "link=a.detail@href"

2 · The wrapper that breaks a naive selector

Real pages contain this:

<div class="card">          <!-- page header block, same class -->
  <h3>Catalogue</h3>
</div>
<div class="list">
  <div class="card"> ... 9 real cards ... </div>
</div>

div.card alone returns 10 containers and one of them is a header with no price. div.list > div.card returns exactly the 9. The fix is not "filter empty rows later" — filtering hides the bug, and next month the header gains a price element and starts producing phantom products.

Test the selector in the browser first: document.querySelectorAll('div.list > div.card').length in the console tells you the right answer before your script disagrees with you.

3 · Whitespace ruins exact matching, so collapse it

<h3 class="name">
  Gasket Ring 1
</h3>

The text is "\n Gasket Ring 1\n". Any == "Gasket Ring 1" comparison fails. Collapse whitespace on extraction (re.sub(r"\s+", " ", t).strip()) and do the comparison after, never before.

4 · Pagination: resolve relative URLs, and stop on repeats

next_href = first_match("div.pager > a.next@href")
url = urllib.parse.urljoin(url, next_href)   # "/list/2.html" -> absolute
if url in visited: break                     # a "next" that points back to page 1

That second line is not paranoia. Sites with a "last page" link, or a template that renders the pager wrongly on the final page, will loop forever. A visited set costs one line.

5 · Detail pages: fetch per row, merge, keep the URL

--field "link=a.detail@href" --follow link
--detail-field "desc=p.desc@text"
--detail-field "made_in=table.spec td@text"

Always keep the detail URL in the output. When a field turns out to be wrong three weeks later, the URL is how you re-check the source without re-crawling.

6 · The failure that wastes the most time

no rows extracted — check --row and the field selectors

[!] these fields came back empty every time: ['price'] — selector probably wrong

Two different bugs. The first is the container. The second is one field: rows came back, so the container is right, but that field is empty on every row — a selector typo, or the field is inside an element that only some rows have. Without that check you get a CSV with a blank column and no idea why.

7 · Behave like a guest

• 0.5s delay between requests, at minimum. 10 req/s to a small site is an attack.
• Retries with backoff (2s, 4s, 8s) — but a 404 will never become a 200; fail fast on those.
• A real user agent. Many sites return 403 to Python-urllib/3.x.
• Read robots.txt. It is not enforced by anything except your own judgement, which is the point.

When this approach does not work

If the data is rendered by JavaScript, the HTML you fetch has an empty container in it. That is not a selector problem and no amount of retrying fixes it — you need a headless browser or the site's own API. Recognizing this in ten seconds (view-source, search for your value) saves an hour of debugging a selector that can never match.

The packaged version

WebScrape Kit is this whole pipeline in one dependency-free file: CSS-lite selectors (tag, class, id, attributes, descendant and child combinators), pagination with a visited set, per-row detail pages, incremental runs via a seen-file, backoff and delay, CSV / JSON / xlsx output, and the two error messages above. It also ships an offline fixture site — including the wrapper-div trap — so you can verify it works before pointing it at anything real. 29 USDT.