Parsing SEC EDGAR: From Raw HTML to Structured Data
Manual parsing of a 400-page 10-K takes an analyst half a day. Errors in extracting Revenue from tables—every fifth record. We built a system that pulls all key metrics, risks, and trends from PDFs into structured JSON in 5 minutes. And does it at scale—500 reports per hour. Unlike conventional parsers, our pipeline uses an LLM for adaptive extraction, handling reports with unstable markup. For instance, SEC EDGAR changes its HTML structure every quarter, but the model finds the needed sections by context.
Problems We Solve
Rate limiting and unstable structure. EDGAR allows no more than 10 requests per second, and HTML markup changes quarterly. Traditional BeautifulSoup parsers break at the first class-to-id rename. Our pipeline uses an LLM for adaptive extraction—the model determines where Revenue lies by context, even if the table is renamed.
Extracting unstructured sections. MD&A, Risk Factors—these are 20–30 pages of prose. Simply taking the text is not enough. We need to separate trends from facts, find numbers in natural language ("revenue increased by 12% to $5.2B"), and compute sentiment. We employ chain-of-thought prompts with few-shot examples from benchmark reports.
Year-over-year comparison. A risk that was third last year and first today signals trouble. The system automatically builds deltas: extracts Item 1A for current and prior year, computes semantic similarity of paragraphs, and marks what's new, intensified, or diminished. The output is a list of red flags with explanations.
How We Do It: Stack and Case Study
We use a hybrid architecture: Python + LangChain + OpenAI GPT-4 (or local vLLM with LLaMA 3). Vector storage—Qdrant for semantic search over historical reports. Deployment via Kubeflow on a GPU cluster (A10G).
from langchain.chains import create_extraction_chain from langchain.chat_models import ChatOpenAI llm = ChatOpenAI(model="gpt-4", temperature=0.1) schema = { "properties": { "revenue": {"type": "number"}, "net_income": {"type": "number"}, "risk_factors_new": {"type": "array", "items": {"type": "string"}} } } chain = create_extraction_chain(llm, schema) result = chain.run(filing_text) Case study: parsed 5,000 annual reports overnight. A hedge fund client wanted to track retail rotation in 10-Ks. Our system processed all 5,000 documents in 8 hours, extracting the top 10 companies with a sharp rise in mentions of "inflation" and "supply chain." Manual verification showed 97% accuracy on risks. Comparison: three analysts would have spent a week on the same task. Our solution is 10x faster and more accurate. Contact us to discuss a similar project. Get a consultation on your tasks.
How the System Handles EDGAR's Rate Limit?
The SEC limits requests to 10 per second. We implemented adaptive throttling with a queue and retries on 429 errors. For bulk parsing, we use a distributed architecture on Ray—load is spread across instances, enabling up to 500 reports per hour without blocking.
Technical detail: XBRL processing
Many 10-K filings contain iXBRL tags. If you skip them, you lose accuracy. We usearelle to extract axis/concept. This provides structured access to financial metrics from tables. Learn more about XBRL on [Wikipedia](https://en.wikipedia.org/wiki/XBRL).Why Extraction Accuracy Reaches 98.5%
We combine XPath rules for tabular data and a fine-tuned LLM for textual sections. Cross-validation: numeric data is checked against prior periods, text via semantic search against benchmark reports. For financial figures, key metric accuracy is 98.5%.
Approach Comparison: Manual vs. Automated Parsing
| Parameter | Manual Parsing | Our AI Pipeline |
|---|---|---|
| Time per 10-K | 4–6 hours | 5 minutes |
| Revenue extraction accuracy | 80% | 98.5% |
| Throughput | 2 reports/day | 500 reports/hour |
| Cost per report | ~$200 | ~$20 |
Economies of scale: processing 1,000 reports per month cuts costs by 10x, recouping the investment in 2–3 months.
Work Stages
- Analytics—discuss which reports, metrics, and signals are needed. Collect benchmark reports for cross-validation.
- Design—select models (GPT-4 / LLaMA 3 / Mistral), design output schema and augmentation pipeline.
- Implementation—build EDGAR adapter, parsers, LangChain chains, delta module, and dashboard.
- Testing—run on 100 random reports, compare with manual parsing. Achieve recall >95% on key fields.
- Deployment—deploy in your cluster or as SaaS. Set up monitoring (uptime, latency p99, report count).
Timeline and Cost
Timeline: 4 to 12 weeks depending on complexity (basic parsing vs. custom business rules). Cost is calculated individually—depends on report volume, required models, and need for a GPU cluster. On average, investment pays back in 2–3 months by reducing analyst time by 80%.
What's Included
| Module | Description | Timeline (weeks) |
|---|---|---|
| EDGAR adapter | Automated rate-limit handling, queue + retry | 1-2 |
| Extraction pipeline | HTML parsing → normalization → NLP → JSON | 2-4 |
| Delta module | Period comparison with Grafana visualization | 1-2 |
| DWH integration | Snowflake, Redshift, ClickHouse | 1-2 |
| Documentation & training | API, architecture + 2 sessions | 0.5 |
| Support | 3 months | included |
Typical Mistakes and How to Avoid Them
- Ignoring XBRL—many 10-K filings include iXBRL tags. If you skip them, you lose accuracy. We use arelle to extract axis/concept.
- Overly aggressive rate limiting—IP block for a day. We set time.sleep(0.15) between requests and monitor the X-RateLimit-Remaining header.
- Not checking encoding—SEC stores documents in Windows-1252. Without conversion, UTF-8 breaks. We apply chardet and automatically convert.
Our experience—7+ years in NLP, 30+ projects in financial analytics. We provide a guarantee on extraction accuracy (98%+ by contract). Order a pilot parsing of 50 documents—result in 2 days. Get a consultation on your tasks. Learn more about the EDGAR format on Wikipedia.