A VR application developer gets a task: create a realistic office interior for employee training. Manually modeling every detail takes weeks. Photogrammetry requires on-site visits. The alternative is neural network generation of 3D scenes from text descriptions or a few photos. We implement such pipelines turnkey — 8+ years of experience and over 50 projects in VR/AR. Contact us for a consultation on integrating neural network generation into your project.
The main challenge of neural methods is balancing quality and performance. Many models produce high quality but require minutes per frame. VR demands 72+ FPS. We solve this through automatic LOD optimization and INT8 quantization of neural networks, reducing memory consumption without quality loss.
How we generate 3D scenes from text
We take a text description (e.g., "an abandoned laboratory with fluorescent lamps") and run it through diffusion models — SceneScape or Set-the-Scene. The result is a depth map and semantic segmentation, which are converted into a mesh. An alternative route is PanoGen for 360° panoramas: 30–60 seconds per environment for initial prototyping. To enhance realism, we use few-shot fine-tuning on a small set of reference images.
Step-by-step scene generation process
- Prepare a text description or set of photos (20–100 shots).
- Choose the method: Gaussian Splatting for photos, PanoGen for 360°, Text-to-Scene for text.
- Optimize: quantize the model to INT8, generate LODs, configure occlusion culling.
- Export to the target engine (Unreal, Unity, WebXR).
- Test on the headset (Quest 3, Pico) with FPS and memory metrics.
Why Gaussian Splatting outperforms NeRF for VR
NeRF delivers high detail but requires seconds per frame. Gaussian Splatting renders in real time without quality compromise. We use it for object reconstruction from 20–100 photos. Additionally, we apply INT8 quantization, reducing memory consumption by 2–3 times. Automatic LOD generation using a quadric error metrics simplification algorithm achieves 72 FPS even on mobile headsets. We guarantee compatibility with any target device after tuning.
| Method | Generation time | Quality | Application |
|---|---|---|---|
| Gaussian Splatting (50 photos) | 5–15 min | Photorealistic | Real objects and spaces |
| Text-to-Scene | 2–10 min | Medium-high | Fantasy/sci-fi environments |
| PanoGen (360°) | 30–60 sec | High (for skybox) | Fast prototyping |
| Manual+AI population | 1–3 h | High | Detailed interiors |
Typical errors in 3D scene generation and how to avoid them
When using Gaussian Splatting, artifacts often appear at object boundaries. The solution is to add depth regularization and tune the number of iterations. Text-to-Scene may produce unrealistic proportions — we fix this by adding semantic maps and fine-tuning on an interior dataset. All our pipelines include automatic quality control with PSNR/SSIM metrics. Time savings compared to manual modeling reach 80%, and generation costs are 5–10 times lower.
Technical details of quantization
To reduce memory consumption without quality loss, we apply INT8 quantization with calibration on a representative sample. We use TensorRT and ONNX Runtime libraries for inference optimization. The process is automated in an MLflow-based pipeline.
When does generative 3D generation replace classic modeling?
For mass production of variations of the same environment — for example, 50 different offices for negotiation training — generative methods are indispensable. When time is tight, a text description turns into a draft scene in minutes. For unique high-detail assets, manual modeling remains the primary option. Budget savings on mass generation reach 60% compared to manual work. Contact us for a preliminary assessment of your project — we will select the optimal generation method for your budget and timeline.
What is included in the work
Analysis and design
Audit of input data, selection of architecture (text/photo/geometry). Choice of methods (Gaussian Splatting, NeRF, PanoGen) and pipeline configuration based on the target device.
Implementation and integration
Training and quantization of models, automatic VR optimization. Export to Unreal Engine, Unity, WebXR. Plugin development for your project. All stages are documented.
Testing and support
Performance measurements on target devices (Quest 3, Pico, etc.) with FPS and memory metrics. Training your team, documentation, and one month of support.
Performance metrics (example on Quest 3)
| Method | Triangles | FPS | GPU memory |
|---|---|---|---|
| Gaussian Splatting | 2M Gaussians | 72 | 1.2 GB |
| Text-to-Scene (optimized) | 500K polygons | 90 | 800 MB |
| PanoGen | 10K (skybox) | 144 | 50 MB |
Final scenes are exported in formats: UAsset (Unreal Engine 5), Prefab (Unity), glTF 2.0 (WebXR), OpenXR/GL (Quest 3). Our certified specialists guarantee correct operation on any headset. Request an assessment of your project — we will select the optimal generation method for your budget and timeline.