Audit of the existing AI system: quality, performance, and security
AI systems degrade. Models drift, data changes, threats evolve, and business requirements transform. Regular audits help identify problems before they impact the business.
What is being checked?
Quality Audit:
Model Performance: Current metrics vs. baseline during deployment. Concept drift: whether the data distribution has changed. Performance on subgroups (slicing by segments).
Data Quality: Pipeline integrity — data arrives without transformation errors. Feature distribution drift. Missing values, outliers in production.
Output Quality: for LLM systems - evaluation on golden dataset. Hallucination rate. Relevance scores.
Performance Audit:
Latency percentiles (p50, p95, p99). Throughput under load. Resource utilization (GPU/CPU). Cost per inference. Bottleneck analysis.
Security Audit:
Adversarial Robustness: resistance to adversarial inputs. Prompt injection for LLM systems. Data poisoning vectors.
Model Extraction: risk of model theft via API.
Data Privacy: Training data leaks due to model inversion. PII in logs.
Access Control: who can query the model, with what rate limits, what inputs are filtered.
The audit process
Week 1: Documentation. Existing system documentation, architecture, versions.
Weeks 2–3: Technical assessment. Performance benchmarking. Security tests.
Week 4: Findings report. Prioritized recommendations.
Deliverables
Audit Report: executive summary + technical details. Risk Register with prioritization. Remediation Roadmap with effort assessment. Monitoring Recommendations.
Periodicity
We recommend: quarterly for high-risk systems, semi-annual for medium-risk, annual for low-risk.