I pulled every independent business within 2500 metres of the Quad in Tuscaloosa, checked whether each one had a website that answers, and had the numbers in forty minutes. The result was 121 businesses and 27 working sites. You can run the same check on your own town.
It costs nothing, needs no API key or account, and does not scrape anybody's search results.
Where does the business list come from?
OpenStreetMap, queried through Overpass. Volunteers maintain it, it is openly licensed, it permits this use, and it carries exactly the tag that matters: website.
Overpass takes a query language that looks alarming and is mostly mechanical. The shape you need is: for each kind of business, find every node and way within a radius of a point.
[out:json][timeout:90];
(
node["shop"="hairdresser"](around:2500,33.2140,-87.5460);
way ["shop"="hairdresser"](around:2500,33.2140,-87.5460);
node["amenity"="restaurant"](around:2500,33.2140,-87.5460);
way ["amenity"="restaurant"](around:2500,33.2140,-87.5460);
);
out center tags;Those coordinates are the centre of your circle. Get them by right-clicking any point in OpenStreetMap. Repeat the node and way pair for every category you care about: shop=barber, shop=clothes, shop=books, amenity=cafe, and so on.
What out center tags; does in Overpass QL
The last line of the query decides what Overpass sends back. The Overpass QL documentation describes the out statement as a verbosity level plus an optional geometry modifier. Each word in out center tags; does one job.
outis the output statement. It prints the elements the query found. With nothing after it, Overpass usesout body.centeradds one centre coordinate to each way and relation. A shop mapped as a building outline comes back with a single lat and lon, the same as a shop mapped as a point. Overpass takes the centre of the element's bounding box, and the documentation warns that this point can fall outside the shape.tagsprints each element's id and tags only. Ways lose their node list and relations lose their member list, so the response stays small.
Nodes keep their own coordinates. When I ran the minimal query below against Tuscaloosa, every way came back with a center object and its tags, and no node list.
[out:json][timeout:25];
way["amenity"="restaurant"](around:2500,33.2140,-87.5460);
out center tags;One element of the response:
{
"type": "way",
"id": 103489064,
"center": { "lat": 33.2249900, "lon": -87.5419030 },
"tags": { "amenity": "restaurant", "building": "yes", "name": "Cypress Inn" }
}Pick the output by what you plan to do with the result.
out center tagsfor lists, spreadsheets and map pins. You get one point and the tags for every business, which is all an audit needs.out geomwhen you need the full shape, for example to draw building outlines or measure an area. It adds coordinates to every node of every way, so the response grows much larger.out bodywhen you want the raw structure. Ways come back with node ids and relations with member lists, and you fetch the node coordinates yourself with a recursion step such as(._;>;);before the output line.
POST that to overpass-api.de/api/interpreter with the query in a form field called data. Send a User-Agent with a contact address in it. It is a free volunteer service, and the address lets the operators reach you.
Filtering to independents
You do not want chains. Do not try to detect them by name, because you will build a list of brand strings and it will be wrong in both directions.
OpenStreetMap tags them for you. Drop anything with a brand or brand:wikidata tag, which takes one line of code.
How do I check each website?
For each business with a website tag, make a request and see what happens. Both of my mistakes happened at this step.
Mistake one: relying on HEAD requests
I sent HEAD requests, because they are cheap and you only need the status code.
Plenty of live servers do not answer HEAD. Two restaurants with working sites came back as broken, and that number reached a published page before I caught it. I had to correct the post in public.
Fall back to GET whenever HEAD fails:
for method in ("HEAD", "GET"):
try:
r = request(url, method=method)
if r.status < 400:
return "OK"
except HTTPError as e:
if e.code in (403, 405):
return "OK" # refusing the probe is not being down
last = f"HTTP {e.code}"
except Exception:
last = "DEAD"
return lastThe 403 and 405 handling matters too. A server that refuses a probe is up, and marking it down misreports the business.
Mistake two: claiming more than the data shows
The headline I wanted was "93 businesses near campus have no website." The data does not support it.
Volunteers maintain OpenStreetMap. A missing website tag shows only that a machine looking for a website does not find one.
A customer searching on a phone runs into the same gap, so that claim is still worth publishing.
Write down the claim your method supports before you write the headline. If the supported claim is weaker, change the method or the headline, because somebody can disprove an overstated claim in one reply.
What should I do with the result?
Sort the businesses that list a website which no longer answers to the top, because they already decided a site was worth paying for and theirs broke. That is a much shorter conversation than convincing somebody they need one.
Keep only name, category, street, and whether the site answers. Leave out phone numbers and email addresses. Once you collect contact details you have a marketing list, and if you told people you were doing research, you lied to them.
What I am using it for
My script is under two hundred lines. I am using it to write about independent businesses near campus, one at a time. If you run it on your town and find something surprising, I would like to hear it.
Questions
What does out center tags do in Overpass QL?
out center tags tells Overpass to print each element's id and tags, and to add one centre coordinate to every way and relation. Nodes keep their own lat and lon. A business mapped as a building outline then comes back as a single point you can pin on a map, without its full shape. Use out geom when you need the shape, and out body when you need the node ids and member lists.
Part of the guide AI for small business.