WEMBot

How to identify it

Every request sends this exact User-Agent:

WEMBot/1.0 (+https://wem3.ai/bot; price comparison; respects robots.txt)

Anything claiming to be WEMBot that does not link to https://wem3.ai/bot is not us.

What it reads

• Public product pages, and the structured data (schema.orgProduct/Offer) already published on them for search engines.
• From that: the product’s identity, its price, its currency and whether it is in stock. WEM records a price, a link and a barcode.
• Not product photography. A price is a fact; a photograph is someone’s work. WEM does not copy your images.
• No accounts, no carts, no checkout, no pages behind a login, no forms.

How it behaves

• Reads /robots.txt before anything else and honours it, including Crawl-delay. If robots.txt cannot be fetched, it does not crawl.
• Honours W3C TDM reservations — the /.well-known/tdmrep.json file, the TDM-Reservation response header and the equivalent meta tag.
• One request at a time per host, spaced by your crawl delay.
• Stops for the rest of the run on a 403 or 429. It will not keep asking after you have said no.

How to block it

Add this to your robots.txt. It takes effect on the next run, and no further pages are read.

User-agent: WEMBot
Disallow: /

To have prices already held for your shop removed as well, use the opt-out. You do not need to explain why, and it is not conditional on anything.

If you would rather send us the data

A feed is better for both of us: you control what is listed and what it says, it is fresher than anything a crawl can be, and WEM stops fetching your pages. Submit a product feed — CSV, TSV, XML, JSON or Google Shopping.

Why WEM does this

To tell a shopper which shop is cheapest, and to send them to you to buy. WEM never sells your products itself and never takes the payment. Listing is free and is not conditional on an affiliate agreement — a shop with no commercial relationship with WEM is shown the same as one with a contract.