On an eCommerce site, bots are not all the same. Some are useful, some eat resources, and others harvest prices and catalogue data to feed competitors, comparison engines or automated systems.
The problem is that you can't simply block everything. Googlebot has to be able to crawl the pages that matter. Legitimate services have to keep working. Real users must not be met with endless captchas. And the site has to stay fast even when somebody tries to download thousands of products.
Defending the catalogue means telling things apart: useful crawlers, tolerable crawlers, aggressive bots, price scraping, application attacks and human traffic.
Why price scraping is a real problem
Anyone selling online knows that prices get compared. Monitoring competitors is normal. But when scraping turns aggressive, it can cause concrete damage.
- it consumes server resources and slows the site down;
- it downloads catalogue, prices, availability and variants in bulk;
- it feeds competitors' automated repricing systems;
- it inflates fake traffic and fills the logs with noise;
- it can hit filter pages and internal search, generating load;
- it can bypass the cache and hit expensive endpoints;
- it can be mistaken for organic traffic if it isn't classified.
The point isn't to make any data collection impossible. The point is not to hand over the catalogue in a way that is easy, unlimited and damaging to performance and margins.
Don't block Googlebot by mistake
The biggest risk of a badly built defence is hitting the crawlers you actually want.
If you block Googlebot, or serve it incomplete pages, you can damage indexing, price updates, snippets, images and category pages. On an eCommerce site that can cost traffic and sales.
That is why a rule along the lines of "block all bots" is dangerous. You need a classification.
- Googlebot has to be recognised and verified correctly.
- Legitimate SEO crawlers can be rate-limited, not always blocked.
- Aggressive bots should be slowed down or blocked.
- Real users must not pay the price of the protection.
- APIs need different rules from public pages.
Robots.txt is not a security barrier
The robots.txt file is useful for giving instructions to cooperative crawlers, but it does not protect the site from anyone determined to scrape it.
A malicious scraper can ignore it completely. Worse: if you list every sensitive area in robots.txt, you are also leaving an interesting map for anyone who wants a look around.
Use it to manage crawling by legitimate crawlers, not as a firewall.
How to spot a price scraper
In the logs, scrapers often show patterns that differ from real users.
- thousands of requests in a short time;
- sequential visits to product pages;
- requests concentrated on price, availability or JSON endpoints;
- generic or spoofed User-Agents;
- no requests for normal assets such as CSS, JS or images;
- browsing with no basket, no scrolling, no real sessions;
- distributed IPs but identical behaviour;
- requests on implausible filter combinations;
- spikes always at the same times of day.
A single signal is not enough. You have to look at behaviour, frequency, endpoints, IPs, ASNs, User-Agent, session, cookies and the impact on resources.
Intelligent rate limiting
Rate limiting is one of the first defences, but it has to be configured with judgement.
Not all pages cost the same:
- the homepage and static pages can be served from cache;
- internal search and filters can be very heavy;
- price and availability endpoints are sensitive;
- basket and checkout must not be disturbed;
- APIs and feeds need authentication and their own limits.
A good system doesn't block on request count alone. It also weighs the quality of the behaviour.
WAF, cache and rules for sensitive endpoints
A Web Application Firewall can help reduce malicious traffic and suspicious patterns, but it does not replace the architecture.
On an eCommerce site it is worth protecting these specifically:
- internal search;
- filter pages with many combinations;
- price and availability endpoints;
- the catalogue API;
- private feeds;
- login and account area;
- basket and checkout;
- administrative imports and exports.
Cache and CDN can reduce load, but if the scraper hits dynamic endpoints or endless filter combinations, you need application-level control.
Good bots, bad bots and ambiguous bots
The classification has to be graduated.
- Allowed: verified Googlebot, necessary crawlers, internal monitors.
- Limited: SEO crawlers, authorised comparison engines, partners, monitoring systems.
- Suspicious: anomalous traffic, spoofed User-Agents, repetitive requests.
- Blocked: aggressive scrapers, bots that ignore limits, known attacks.
Not everything has to be blocked immediately. Sometimes it is better to slow the traffic down, serve it from cache, ask for a verification step or restrict only certain areas.
Watch out for SEO filter pages
Filter pages can be an enormous SEO advantage, but also a sensitive point for bots and scraping.
If you have thousands of pointless combinations, a scraper can generate load by exploring URLs with parameters or empty combinations. If the filters are instead managed as real, controlled pages, you can protect the structure much better.
This connects to the work on eCommerce filters and SEO and on managing canonicals, parameters and filter combinations.
What to monitor every week
- top IPs and ASNs by request volume;
- the most frequent User-Agents;
- the most heavily downloaded product pages;
- endpoints with the most 403, 404, 429 and 500 errors;
- spikes on internal search and filters;
- verified Googlebot traffic;
- crawl changes in Search Console;
- the impact on server performance and cache hit ratio;
- anomalous requests on price and availability.
Monitoring is there to avoid two opposite mistakes: leaving everything open, or blocking too much.
How we handle it at BitHub
For eCommerce sites we work at several levels: logs, CDN, WAF, application, APIs, cache, Search Console and rules on the most sensitive paths.
The goal isn't to "hide the site", but to protect it without sacrificing SEO, real users and campaigns.
This subject connects to cyber security for websites, custom eCommerce and competitor price monitoring.
Useful sources
To set up SEO-friendly protection we drew on the Google Search Central documentation on Googlebot, the guide to verifying Googlebot and the introduction to robots.txt.
FAQ
Can I block all bots?
No. You would also block useful crawlers and could damage SEO, monitoring and legitimate services. You need to make distinctions.
Does robots.txt stop price scraping?
No. It helps with cooperative crawlers, but a malicious scraper can ignore it. Real protection comes from logs, rate limiting, a WAF and application-level rules.
How do I avoid blocking Googlebot?
Verify Googlebot properly, monitor Search Console and apply different rules to legitimate crawlers than to suspicious bots.
Is price scraping always illegal?
It depends on the context, the method and the jurisdiction. From a technical point of view, though, you can reduce abuse, load and bulk access to the catalogue.
Do I need a WAF?
It helps, but it has to be configured together with cache, logs, application rules and SEO monitoring. A badly configured WAF can create false positives.
Want to protect your catalogue and prices without losing SEO?
We can analyse logs, bot traffic, filter pages, APIs and Search Console to build a balanced protection.