Data Export from Web Applications: Turnkey Implementation

According to Article 20 of the GDPR, each data subject has the right to receive their personal data in a structured, commonly used, and machine-readable format.

But in practice, exporting 500,000 rows often crashes the server: a 30-second timeout and a 502 Bad Gateway. The cause is synchronous architecture and PHP limits. We solve this at the infrastructure level: asynchronous generation via queues, streaming writes to S3, and temporary signed URLs. Requirements under GDPR (Article 20) mandate providing user personal data in a machine-readable format — we implement export turnkey. Contact us for an assessment of your project.

Problems We Solve

• N+1 queries during data export: when exporting orders with items, each database query spawns new ones — for 10,000 orders that's 10,001 queries. We use lazy loading via lazy(200) and eager loading of relationships.
• Timeouts and memory exhaustion with synchronous exports >100,000 rows: the PHP script hits memory_limit and max_execution_time. Our solution is asynchronous processing via queues with a timeout of 600 seconds.
• Hydration mismatch during SSR: if export data is embedded on the frontend, React/Nuxt can desynchronize. We export everything via API with unique IDs.

Why Asynchronous Export Is Better for Large Data

Synchronous export works only for small volumes (<10k rows). As data grows, it leads to interface blocking and connection drops. An asynchronous scheme with a queue and notifications removes load from the HTTP worker and scales horizontally. When one client generates a report of 200k rows, others continue working without delays.

A typical example: a client requested an export of 150,000 orders to Excel. Initially, the developer wrote a synchronous script — at 40 seconds, PHP crashed due to memory_limit. After switching to a queue, generation took 2 minutes, and the user received an email with a link without any blocking.

[User request] → [Create ExportJob record] → [Queue Worker] ↓ [Generate files] ↓ [Upload to S3 (zip)] ↓ [Email with download link] (signed URL, 7 days TTL) 

How We Do It: Laravel Implementation

We use Laravel with PHP 8.3+ for the backend. The key pattern is the DataExportService, which queues a job. The ExportUserDataJob worker generates profile (JSON), orders (CSV with BOM), and messages (JSON) files, then packages them into a ZIP for S3 upload.

class DataExportService { public function exportUserData(User $user): Export { $export = Export::create([ 'user_id' => $user->id, 'status' => 'pending', 'expires_at' => now()->addDays(7), ]); ExportUserDataJob::dispatch($user, $export); return $export; } } class ExportUserDataJob implements ShouldQueue { public int $timeout = 600; public function __construct(private User $user, private Export $export) {} public function handle(): void { $this->export->update(['status' => 'processing']); $tempDir = sys_get_temp_dir() . '/export_' . $this->export->id; mkdir($tempDir, 0777, true); try { // Профиль file_put_contents( "$tempDir/profile.json", json_encode($this->exportProfile(), JSON_UNESCAPED_UNICODE | JSON_PRETTY_PRINT) ); // Заказы $this->exportOrders("$tempDir/orders.csv"); // Сообщения $this->exportMessages("$tempDir/messages.json"); // Создать ZIP $zipPath = sys_get_temp_dir() . "/export_{$this->export->id}.zip"; $zip = new ZipArchive(); $zip->open($zipPath, ZipArchive::CREATE); foreach (glob("$tempDir/*") as $file) { $zip->addFile($file, basename($file)); } $zip->close(); // Загрузить в S3 $s3Key = "exports/{$this->user->id}/{$this->export->id}/data.zip"; Storage::disk('s3')->put($s3Key, file_get_contents($zipPath)); $this->export->update([ 'status' => 'ready', 's3_key' => $s3Key, ]); $this->user->notify(new ExportReadyNotification($this->export)); } finally { exec("rm -rf {$tempDir}"); if (file_exists($zipPath ?? '')) unlink($zipPath); } } private function exportProfile(): array { return [ 'id' => $this->user->id, 'name' => $this->user->name, 'email' => $this->user->email, 'created_at' => $this->user->created_at->toIso8601String(), 'profile' => $this->user->profile?->toArray(), ]; } private function exportOrders(string $path): void { $handle = fopen($path, 'w'); fprintf($handle, chr(0xEF) . chr(0xBB) . chr(0xBF)); // UTF-8 BOM fputcsv($handle, ['Номер', 'Дата', 'Сумма', 'Статус', 'Позиции'], ';'); $this->user->orders()->with('items')->lazy(200)->each(function (Order $order) use ($handle) { $items = $order->items->map(fn($i) => "{$i->name} x{$i->quantity}")->join(', '); fputcsv($handle, [ $order->number, $order->created_at->format('d.m.Y H:i'), number_format($order->total, 2), $order->status, $items, ], ';'); }); fclose($handle); } private function exportMessages(string $path): void { $messages = $this->user->messages() ->select('id', 'subject', 'body', 'created_at') ->get() ->map(fn($m) => [ 'id' => $m->id, 'subject' => $m->subject, 'body' => strip_tags($m->body), 'date' => $m->created_at->toIso8601String(), ]); file_put_contents($path, json_encode($messages, JSON_UNESCAPED_UNICODE | JSON_PRETTY_PRINT)); } public function failed(\Throwable $e): void { $this->export->update(['status' => 'failed']); Log::error('Export failed', ['export_id' => $this->export->id, 'error' => $e->getMessage()]); } } 

How Signed URLs Work for Download

To avoid storing archives indefinitely, we use temporary S3 links with a 15-minute TTL. The user receives an email with a unique link that points directly to S3 — bypassing our server.

public function download(Export $export): RedirectResponse { $this->authorize('download', $export); if ($export->status !== 'ready' || $export->expires_at->isPast()) { abort(410, 'Экспорт недоступен или устарел'); } $url = Storage::disk('s3')->temporaryUrl($export->s3_key, now()->addMinutes(15)); return redirect()->away($url); } 

Choosing the Right Export Format

The format depends on the purpose: JSON for API and GDPR extraction, CSV with BOM for tables and Excel, XLSX with formatting for accounting, XML for B2B integrations. If multiple files are needed, we use ZIP.

Our Process

1. Analysis — we study the data structure, volumes, and format requirements.
2. Design — select the tech stack (queue, storage), design the export database schema.
3. Development — write services, workers, controllers, and tests.
4. Testing — load testing with real volumes, timeout checks.
5. Deployment — configure queues (Redis, SQS), S3, clean-up crons.
6. Documentation — hand over architectural docs and instructions.

What's Included

• Architectural documentation with data flow descriptions.
• Source code with comments and strict typing.
• Deployment instructions with examples of queue and S3 configuration.
• Training for support staff.
• 3-month warranty for uninterrupted operation.

Timeline and Pricing

Basic user data export (JSON + CSV in ZIP): 2–3 days. With queue, S3, and email notification: 3–4 days. GDPR-compliant export of all personal data: 4–5 days. Pricing is determined individually based on data volumes and number of formats. Our team has over 10 years of experience and has completed 40+ data export projects. Order an export for your project — get a free consultation and estimate within one day.

Contact us to discuss your project and receive an accurate estimate.