Parser data auto-filling: mapping, moderation, publication

Imagine an online store with 50,000 products, data supplied by three partners in different formats. Direct import within a day leads to 15% duplicates and broken links — the site loses search positions. We developed a system that eliminates such issues through a pipeline: parser → normalization → moderation → publication. But simply copying data from a parser into the database is naive and doesn't scale. Without intermediate processing, you risk junk: duplicates, broken images, invalid prices. Additionally, each supplier sends data in their own format: one uses JSON with nested fields, another uses XML with attributes. We need to unify everything into a single structure, check quality, and only then publish. This auto-filling website method requires systematic approach, otherwise web scraping turns into chaos. Below we break down key nodes and implementation.

What problems arise with direct import?

Direct import from parser to site database is bad practice. Typical errors:

• Duplicates — identical products with different IDs due to repeated parsing.
• Broken images — links to external resources that were deleted.
• Invalid prices — 0 or 999,999,999 rubles from source errors.
• Different structure — each supplier has their own format: article in article or sku.

We solve this with an intermediate queue with validation and normalization, reducing manual correction by 80%.

Why is an intermediate queue needed?

The queue buffers data and applies business logic: deduplication, format transformation, enrichment from external APIs. Without a queue, a parser failure can dump junk into your CMS. Using a queue reduces publication errors by 5x compared to direct import, processing up to 10,000 records per minute. Our solution, built over 7+ years of experience with 150+ projects, ensures reliability.

How to configure field mapping for any source?

Each source has its own structure. We use configuration based on JSONPath, allowing field correspondence without code changes.

{ "source": "supplier_catalog", "mappings": { "title": "$.name", "description": "$.full_description", "price": "$.price_rub", "category": { "field": "$.category_id", "transform": "category_map" }, "images": "$.photos[*].url", "sku": "$.article" }, "category_map": { "1": "electronics", "2": "clothing", "15": "home-garden" } } 

This reduces adaptation time to hours. For CMS import, we integrate via API ensuring seamless field mapping.

How are images processed?

Images are downloaded, optimized, and uploaded to our own storage:

async def process_image(url: str, product_id: int) -> str: async with httpx.AsyncClient() as client: resp = await client.get(url, timeout=30) img = Image.open(BytesIO(resp.content)) img = img.convert('RGB') img.thumbnail((1200, 1200), Image.LANCZOS) output = BytesIO() img.save(output, 'WEBP', quality=85) s3_key = f'products/{product_id}/{uuid4()}.webp' s3.put_object(Bucket=BUCKET, Key=s3_key, Body=output.getvalue()) return f'https://cdn.example.com/{s3_key}' 

WebP reduces file size by 30–50% without quality loss, and our CDN caching with 7-day TTL cuts LCP by 40%.

Quality control and publication strategies

Data undergoes three-level validation:

• Required fields: name, price, at least one photo.
• Price range: 1 to 1,000,000 units.
• Description: at least 50 characters.
• Images: accessible, width ≥ 300 px.

Records failing go to review_required status. Three publication strategies:

Choice depends on source reliability and data criticality. This saves up to $20,000 annually for high-volume catalogs.

Process overview

1. Analysis — study parser structure and CMS fields.
2. Design — develop mapping and queue schema.
3. Implementation — write processor in Python, validator with rules, integrate with CMS via API.
4. Testing — run on 10,000 historical records, check edge cases.
5. Deployment — deploy to production, configure error monitoring.

What's included

• Analysis of parser structure and CMS fields
• Development of mapping schema
• Configuration of intermediate queue and validation
• Integration via CMS API
• Configuration and support documentation
• Training editors on queue and moderation workflow
• Technical support during launch

Timing

With 7+ years in the field and 150+ successful projects, we ensure reliable parser CMS integration. Contact us for a consultation to evaluate your project — we'll select the optimal architecture for automated content filling.