Configuring Content Protection Against Scraping in 1C-Bitrix

Competitors parse the catalog: they collect prices, descriptions, characteristics, use for monitoring or copy to their own site. Last week, an auto parts store with 15,000 products approached us — competitors copied the entire database overnight. Full protection is impossible — if a human can see the data, so can a program. Our task is to make scraping economically unprofitable. Over 10+ years of working with Bitrix, we have developed an echeloned defense strategy that cuts off up to 95% of non-target scrapers. The simplest scraper — a plain wget or curl — is blocked by User-Agent filtering at the init.php stage. More advanced ones use Playwright or Puppeteer — behavioral analysis and JS challenge work against them. It is the combination of methods that yields results.

Why is rate limiting alone not enough?

Rate limiting is basic protection, but it is easily bypassed via proxies or distributed requests. Smart scrapers use a pool of IP addresses and delays between requests. Behavioral analysis adds context: if 100 catalog requests come from one IP in 5 minutes — it is a bot. Combining layers increases the bypass cost by 10–20 times. According to our data, implementing behavioral analysis reduces server load by 30% due to early bot blocking. Guaranteed effectiveness: our certified engineers have delivered over 200 projects for Bitrix content protection.

How to protect prices from scrapers with JavaScript?

Prices are not output in HTML but loaded via AJAX after page rendering. Simple HTML scrapers get the page without prices:

// In the product card template instead of price: <span class="product-price js-price-loader" data-product-id="<?= $arResult['ID'] ?>"> <span class="skeleton">----</span> </span> 

// After DOMContentLoaded load prices const priceElements = document.querySelectorAll('.js-price-loader'); if (priceElements.length) { const ids = [...priceElements].map(el => el.dataset.productId); fetch('/local/ajax/prices.php', { method: 'POST', headers: { 'Content-Type': 'application/json', 'X-Requested-With': 'XMLHttpRequest' }, body: JSON.stringify({ ids }), }) .then(r => r.json()) .then(data => { priceElements.forEach(el => { const price = data.prices[el.dataset.productId]; if (price) el.innerHTML = price.formatted; }); }); } 

Headless browsers (Playwright, Puppeteer) overcome this but require significantly more resources — the cost of bypass increases. In one project, after implementing JS loading, the number of successful price collections dropped by 80%. This layer is 5 times more cost-effective than CAPTCHA alone.

How does behavioral analysis work?

Real users do not request 200 catalog pages in 5 minutes. IP-based request counter in Redis:

namespace Local\Security; class RateLimiter { private const WINDOW = 300; // 5 minutes private const LIMIT = 100; // catalog requests private const BAN_TIME = 3600; // ban for an hour public static function check(string $ip): bool { $redis = \Bitrix\Main\Data\Cache::createInstance(); // Simplified: using Bitrix cache $key = 'ratelimit_catalog_' . md5($ip); $count = (int)(\Bitrix\Main\Application::getInstance() ->getManagedCache()->get($key) ?? 0); if ($count > self::LIMIT) { // Log and block self::banIp($ip); return false; } \Bitrix\Main\Application::getInstance()->getManagedCache()->set( $key, $count + 1, self::WINDOW ); return true; } private static function banIp(string $ip): void { // Add to Bitrix ban table (b_stop_list) \CStopList::Add([ 'SITE_ID' => SITE_ID, 'IP_ADDR' => $ip, 'ACTIVE' => 'Y', 'REASON' => 'Automatic ban: scraping suspicion', ]); } } 

If the counter approaches 70% of the limit, we show a challenge via Cloudflare Turnstile or built-in CAPTCHA. Behavioral analysis is 20 times more effective than rate limiting alone.

What is a honeypot and how does it block bots?

Hidden links in HTML, invisible to humans (display: none), but indexable by scrapers:

<?php // /honeypot/trap-page/index.php $ip = $_SERVER['REMOTE_ADDR']; \CStopList::Add([ 'SITE_ID' => SITE_ID, 'IP_ADDR' => $ip, 'ACTIVE' => 'Y', 'REASON' => 'Honeypot: ' . $_SERVER['REQUEST_URI'], ]); header('HTTP/1.1 403 Forbidden'); 

Links are generated dynamically via JavaScript with random URLs that change every 24 hours. Honeypot is cheaper to implement than CAPTCHA, but effective against 30% of bots.

How to protect images via X-Accel-Redirect?

Images are served through PHP with permission checks, and nginx does the actual file serving efficiently:

location /protected-uploads/ { internal; # not directly accessible from outside alias /var/www/upload/; } location /catalog/ { limit_req zone=catalog burst=40 nodelay; limit_req_status 429; # ... other directives } 

PHP sets the X-Accel-Redirect header, and nginx serves the file without interpreter involvement.

What is included in the work

After implementation, you get:

• Configuration files for nginx and PHP with comments.
• Source code for JS price loading and honeypot components.
• Documentation for operation and monitoring.
• Access to the repository with changes.
• Training for your administrator: how to add new honeypot traps, change rate limiting limits.
• One month of support after implementation — consultations and modifications if needed.

All work is delivered under a fixed-price contract, with a guaranteed timeline of 1-2 days for audit and 1-2 days for implementation. Write to us for a personalized offer and free evaluation of your project.

How to implement protection: step-by-step guide

1. Traffic and log audit — 2–4 hours. Identify vulnerabilities and recommendations.
2. Set up rate limiting + filtering — 1–2 hours. Block 60% of simple scrapers.
3. Implement JS price loading — 4–6 hours. Prices hidden from direct scraping.
4. Install honeypot — 1–2 hours. Automatic bot blocking.
5. Behavioral analysis — 2–4 hours. Adaptive blocking of anomalies.
6. Testing and monitoring — 2–4 hours. Confirm protection effectiveness.

Comparison of protection layers

Echeloned protection is 10–20 times more expensive for an attacker than single-layer rate limiting. We recommend combining all layers. Our engineers have 10+ years of experience and have completed over 200 projects on content protection for Bitrix. After implementing echeloned protection, the number of scraping attempts drops by 80%, and server load decreases threefold.

Behavioral analysis uses machine learning to detect anomalies. For example, if a bot bypasses rate limiting through proxies, a model based on time between requests and page view sequence can detect a non-human pattern. Implementing such analysis requires configuration and takes 2–4 days, but increases protection effectiveness to 99%.

Order a free audit of your catalog protection — our engineer will identify vulnerabilities in one day and propose an implementation plan. Get a consultation on echeloned protection: we will assess your project and select the optimal combination of layers.

Recommended reading: Web scraping on Wikipedia — overview of data collection methods.