Implementing SHAP/LIME for Model Explainability

Model XGBoost gives AUC 0.91 on validation. In production, unexpected predictions appear — high scores for obviously irrelevant objects. Feature importance from boosting itself shows top-10 features but does not explain a specific prediction. That particular object may get score 0.87 for non-obvious reasons — and we as engineers are obliged to answer. We implement SHAP and LIME for model explainability in production, and this is not just an audit — it is part of an end-to-end ML pipeline.

SHAP and LIME answer different versions of the question "why?". It is important to understand when to apply each method and where they break.

How do SHAP and LIME work?

SHAP (SHapley Additive exPlanations, Lundberg & Lee, 2017) is based on Shapley values from cooperative game theory. The idea: the contribution of each feature to the prediction is the average of its marginal influence across all possible feature coalitions. Key property: additivity. The sum of SHAP values of all features plus the base value (average model prediction) equals the specific prediction. This is a mathematically exact decomposition, not an approximation.

LIME (Locally Interpretable Model-agnostic Explanations, Ribeiro et al., 2016) works differently: a random cloud of perturbations is generated around the object, the black-box model predicts each perturbed instance, and then a simple interpretable model (linear regression or decision tree) is trained on that cloud. LIME is stochastic, so in production we fix the seed and use num_samples=5000+.

What problems do SHAP and LIME solve?

1. Failures of feature importance. Built-in feature importance in XGBoost shows a global picture but does not explain a single case. SHAP solves this with deterministic decomposition.
2. Black box for business. Regulators require explanations for each decision. TreeSHAP provides transparency in acceptable time.
3. Model drift without signal. SHAP values logged to ClickHouse allow tracking changes in feature influence earlier than metric drops.

TreeSHAP — why architectural specialization matters

For tree-based models (XGBoost, LightGBM, CatBoost, sklearn RandomForest) there is TreeSHAP — an algorithm with polynomial complexity O(TLD²). This is orders of magnitude faster than naive KernelSHAP.

import shap
import xgboost as xgb

model = xgb.XGBClassifier()
model.fit(X_train, y_train)

explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X_test)

# Waterfall plot for a specific prediction
shap.plots.waterfall(explainer(X_test)[0])

# Summary plot — global importance
shap.summary_plot(shap_values, X_test)

In practice, TreeSHAP on LightGBM with 500 trees processes 10,000 examples in 2–3 seconds on CPU. Quite acceptable for batch inference.

Why LIME is sometimes better than SHAP?

• The model is not supported by TreeSHAP and KernelSHAP is too slow.
• You need an explanation in terms of "superpixels" for images or word highlighting for texts.
• A quick prototype without deep mathematics is required.

But remember: LIME is not deterministic. With different random_state, explanations for the same object can differ. In production we use a fixed seed and num_samples=5000+.

Method comparison

Typical problems and solutions

Integration into production ML pipeline

Explanations are needed not only for audit — they are part of the operational pipeline.

Case study: client — an insurance company, calculating insurance premiums (LightGBM, 120 features). Requirement: an agent must explain over the phone why a premium is high. Solution: TreeSHAP in the inference API. For each prediction, return the top-3 features with the highest SHAP values + an automatic text template: "Your premium is above average due to: vehicle age (+12%), registration region (+8%), claims history (+6%)". Latency overhead: 35ms for TreeSHAP with average inference 18ms — acceptable.

Monitoring: SHAP values are logged to ClickHouse. Once a week we aggregate — drift in SHAP value distribution signals feature drift earlier than AUC drop.

Limitations to be aware of

SHAP ≠ causality. A high SHAP value for a feature means correlation with the prediction, not causation. "Feature X influences the prediction" ≠ "changing X will change the outcome in reality".

Multicollinearity breaks interpretation. If two features are correlated (r > 0.8), SHAP splits their influence arbitrarily. Correlation analysis is needed when interpreting.

For LLMs — both methods give rough estimates. Attention weights are often more informative for generation tasks, but are also not a strict proxy for importance.

What is included in our work

• Model and data analysis: select the appropriate method — TreeSHAP, KernelSHAP, LIME — considering architecture and latency requirements.
• Explainer module development: integration into the existing inference API.
• Report generation: waterfall plots, summary plots, automatic text templates.
• Monitoring: logging SHAP values to ClickHouse, building drift dashboards.
• Team training: documentation, workshop for engineers and business users.

Timelines and cost

Timelines: from 1 week for basic integration of one method to 3–4 weeks for a full pipeline with monitoring and dashboards. Cost is calculated individually for each project. Contact us for a preliminary assessment — we will tell you what results you will get.

Our experience: over 50 projects in explainable AI, 5 years in the market, certified ML engineers. We guarantee transparency and post-implementation support.

Request a consultation — we'll help make your model explainable and compliant with regulatory requirements.