Regularly updating the news section is critical for SEO and audience retention. Manually collecting and publishing news from industry sources is inefficient: site owners spend up to 10 hours per week on this routine, diverting resources from business development. RSS aggregation solves the problem: the system automatically fetches fresh entries, removes duplicates, and publishes them with proper attribution. We are a team of engineers with many years of experience in 1C-Bitrix, having completed over 50 content automation projects. We implement such a solution turnkey in 3–5 days, ensuring content uniqueness and copyright compliance. Get a consultation for your project — just contact us.
Why Automate News Filling?
Manually copying news from feeds leads to content duplication, which is penalized by search engines and reduces audience trust. Automation not only saves time but also guarantees that each news item is unique and properly attributed. With an RSS aggregator, you get fresh content daily without editor involvement.
How to Fetch and Parse RSS?
RSS is an XML format with a standard structure. Each entry (<item>) contains title, link, description, pubDate, author. Atom entries (<entry>) use different tags but the logic is similar.
Parsing with SimpleXML:
$rss = simplexml_load_file($feedUrl); foreach ($rss->channel->item as $item) { $this->processItem([ 'title' => (string)$item->title, 'link' => (string)$item->link, 'content' => (string)$item->children('content', true)->encoded ?: (string)$item->description, 'pubDate' => strtotime((string)$item->pubDate), 'guid' => (string)$item->guid, ]); } <content:encoded> contains the full article text (if the source provides it), <description> is usually an excerpt. SimpleXML is 2 times faster than ready-made DOM parsers. The RSS specification defines feed formats.
How to Avoid Duplicate Entries?
The same material may appear in multiple feeds or be republished. The primary method is deduplication by guid (unique identifier in RSS):
$existing = CIBlockElement::GetList([], [ 'IBLOCK_ID' => NEWS_IBLOCK_ID, '=PROPERTY_RSS_GUID' => $item['guid'] ])->Fetch(); if ($existing) continue; // already imported The RSS_GUID property is of type S with IS_REQUIRED = N. An alternative for speed is to store processed GUIDs in a separate table or Redis Set. We tested both approaches: Redis yields a 30% performance gain for streams over 10,000 entries.
Storage in the News Infoblock
Standard news infoblock with additional properties for RSS aggregation:
-
RSS_GUID— entry GUID for deduplication -
RSS_SOURCE— source ID or name (for attribution) -
ORIGINAL_URL— link to original (for canonical and source link) -
AUTO_IMPORTED— auto-import flag (Y/N) to distinguish from manual posts
Publication date from RSS → ACTIVE_FROM of the element. This is important for correct news sorting.
Why Content Processing Matters?
Direct publication of RSS content without processing is duplication that leads to search engine penalties. We offer three processing levels:
| Level | Description | Duplication Risk | Effort |
|---|---|---|---|
| Minimum | Publish excerpt with a "read more" link | Low | Low |
| Medium | Clean HTML + rephrase the introduction | Medium | Medium |
| Full | AI rewrite of entire text | High | High |
Minimum: legitimate aggregation — not duplication, as only part of the text is published. Medium level: we clean HTML (HTMLPurifier), remove internal source links, rephrase the introduction and title. Full rewrite via AI: send content:encoded to GPT with instructions to rewrite in a different style. This is expensive for high-frequency feeds but justified for key materials. The costs of manual publication can be significant, and our solution pays off in 2–4 months.
How to Configure Multiple Feeds?
Source configuration in a Highload-block RssSources:
-
UF_URL— feed URL -
UF_NAME— source name -
UF_IBLOCK_ID— target infoblock -
UF_SECTION_ID— section for imported materials -
UF_ACTIVE— enabled/disabled -
UF_INTERVAL— check interval in minutes -
UF_LAST_CHECK— last check time -
UF_PROCESSING— processing type (excerpt/full/ai_rewrite)
How We Set Up an RSS Aggregator?
- Technical specification and solution architecture.
- Develop RSS reader supporting RSS 2.0 and Atom.
- Deduplication, storage in infoblock, attribution flags.
- Administrative interface for source management.
- Content processing according to selected scheme.
- Agent scheduling and error monitoring.
- Documentation and access transfer.
What Is Included in the Work?
- Documentation: data schema, agent settings, admin manual.
- Access: login credentials, SSH keys, API credentials (if required).
- Training: walkthrough of feed management interface, answering questions.
- Support: warranty maintenance for 30 days after delivery.
Timeline
| Stage | Duration |
|---|---|
| Develop RSS reader supporting RSS 2.0 and Atom | 4–8 hours |
| Deduplication, storage in infoblock | 4–8 hours |
| Content processing (HTML cleaning) | 4 hours |
| Administrative interface for source management | 4–8 hours |
| Scheduling, monitoring | 2–4 hours |
Total: 3–5 working days. Adding AI-rewrite — plus 1–2 days.
We'll evaluate your project in 1 day — just contact us. Order turnkey development with quality guarantee and post-implementation support. Get in touch for your project estimate.