What I learned scraping public data for a side project

EN data engineeringscrapingdatabasespython

A while back I started building a database of Finnish company data — ownership structures, registered addresses, board members, that kind of thing. The source is the Finnish Business Information System (YTJ / PRH), which is public and free to query. I expected it to be a straightforward data collection project. It mostly was, but the interesting problems were in the details.

Check for a bulk download first

This is the most important lesson, and I learned it late: before writing a single line of scraping code, check whether there’s a bulk export.

YTJ has an API for querying individual companies and a separate page buried in the documentation about a weekly bulk data export — a set of zipped CSV files covering every registered Finnish company with all their basic data. The bulk export is hundreds of megabytes compressed, covers everything, and is updated every week.

If I had found that earlier I would have saved several days of work and my scraper would never have existed. The bulk export gives you everything in one shot; the API is for lookups and incremental updates once you have a baseline.

The general rule: look for bulk exports, open datasets, or data dumps before scraping. Data providers often offer these because they’d rather you download once than hammer their API thousands of times. They’re not always easy to find — look in the documentation, the terms of service, the developer section of the site, and sometimes just the robots.txt file which occasionally mentions endpoints.

Rate limiting and being a good citizen

When you do need to scrape — either because there’s no bulk export or because you need near-real-time data — be conservative with request rates, even when the site doesn’t enforce limits.

My defaults:

  • 1–2 requests per second for most sites
  • Random jitter (±20%) to avoid a perfectly metronomic pattern
  • Exponential backoff on 429/503 responses
  • Respect Retry-After headers when present
  • User-Agent that identifies your project and includes a contact address
import time
import random
import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

session = requests.Session()
session.headers['User-Agent'] = 'MyProjectBot/1.0 (contact@example.com)'

retry = Retry(total=5, backoff_factor=1, status_forcelist=[429, 500, 502, 503, 504])
session.mount('https://', HTTPAdapter(max_retries=retry))

def fetch(url):
    time.sleep(1 + random.uniform(-0.2, 0.2))
    return session.get(url, timeout=10)

The goal is to be indistinguishable from a moderately active human browser in terms of load, while being polite enough that the operator doesn’t have a reason to block you.

Deduplication is harder than it looks

The first time you build a dataset from a public registry, you expect each entity to have a clean unique identifier. Sometimes they do. Often they don’t, or the same real-world entity appears multiple times under different names or identifiers.

With Finnish company data, the Y-tunnus (business ID) is the unique identifier and it’s reliable. But when you try to link companies across data sources — matching a YTJ company to a company mentioned in a news article, or to a company in a contracts database — you’re doing entity resolution, and that’s where it gets messy.

Name matching alone doesn’t work:

  • “Acme Oy” and “ACME OY” are the same
  • “Acme Finland Oy” and “Acme Oy” might be the same or might be different legal entities
  • Companies change names; the old name still appears in old documents
  • Abbreviations, missing words, transliterations for bilingual companies (Finnish/Swedish)

My approach for a first pass:

  1. Normalize names: lowercase, strip punctuation, normalize whitespace, expand common abbreviations (Oy → osakeyhtiö, etc.)
  2. Exact match on normalized name: catches the easy cases
  3. Fuzzy match with a threshold: I use rapidfuzz with a token sort ratio threshold of ~88 for candidates to review
  4. Manual review queue: anything above threshold but below certainty goes into a queue for human confirmation
from rapidfuzz import fuzz

def match_company(name, candidates):
    normalized = normalize(name)
    results = []
    for c in candidates:
        score = fuzz.token_sort_ratio(normalized, normalize(c['name']))
        if score >= 88:
            results.append((score, c))
    return sorted(results, reverse=True)

Don’t try to automate this all the way. The cost of a false positive (linking two different companies) is usually worse than the cost of a false negative (missing a match). Keep a manual review step and log your confidence scores.

Structuring data that changes over time

Once you have the data, the next question is how to store it, and this is where I ran into the temporal data problem I’ve written about separately.

Company ownership changes. Directors change. Addresses change. If you just overwrite the current value every time you refresh your data, you lose the historical record — and for anything involving company data, the historical record is often the whole point.

The short version: add valid_from and valid_to columns to any table that tracks something that can change. When the data changes, close the old record and insert a new one instead of updating. This gives you point-in-time queries: “what was the registered address of this company on this date?”

What I’d do differently

A few things I’d change if I were starting over:

Store the raw response alongside the structured data. I lost several edge cases early on because I only kept the parsed result, not the original HTML or JSON. If you store the raw response, you can re-parse later when you discover a pattern you missed.

Record when you collected it, not just when it was published. There’s a difference between “this information was valid as of 2023-01-01” and “I collected this information on 2023-06-15.” Both timestamps are useful; keep both.

Start with a smaller scope. I initially tried to collect everything at once. Starting with one region or one company type, getting the pipeline solid, then expanding is a better sequence.

If you’re building a data collection pipeline for your business — registry data, contract data, anything involving public records — I’m happy to talk through it.