Explainable AI Tutorial: SHAP, LIME, Partial Dependence, Interpretability (2026)
Explainable AI (XAI) makes ML model predictions understandable to humans. As ML models are deployed in high-stakes domains like healthcare and finance, the ability to explain individual predictions has become a regulatory and ethical requirement.
By 2026, XAI tools are mature and integrated into the ML workflow. SHAP provides theoretically grounded feature attribution, LIME offers local explanations, and partial dependence plots reveal global feature relationships.
SHAP: Theory and Practice
SHAP (SHapley Additive exPlanations) uses cooperative game theory to assign each feature an importance value for a prediction. It satisfies properties like consistency, accuracy, and missingness. TreeSHAP provides fast computation for tree-based models.
For a credit risk model, SHAP revealed that payment history contributed 60 percent of the prediction for a denied applicant, while income contributed only 15 percent. This level of detail was crucial for regulatory compliance and customer explanations.
import shap\nexplainer = shap.TreeExplainer(model)\nshap_values = explainer.shap_values(X_test)\n# Summary plot\nshap.summary_plot(shap_values, X_test)\n# Force plot for single prediction\nshap.force_plot(explainer.expected_value, shap_values[0], X_test.iloc[0])LIME for Local Explanations
LIME (Local Interpretable Model-agnostic Explanations) approximates the model locally with an interpretable surrogate model. It perturbs the input and observes prediction changes to determine feature importance for a single instance.
LIME helped debug a text classification model that was relying on spurious correlations. For a customer complaint email, LIME highlighted that the word "please" and the sender domain were driving positive sentiment, revealing a bias toward polite language from corporate domains.
from lime.lime_tabular import LimeTabularExplainer\nexplainer = LimeTabularExplainer(X_train, feature_names=feature_names, class_names=["negative", "positive"], mode="classification")\nexp = explainer.explain_instance(X_test[0], model.predict_proba, num_features=5)\nexp.show_in_notebook()\nexp.as_list()Partial Dependence Plots
Partial Dependence Plots (PDP) show the marginal effect of one or two features on the predicted outcome. They reveal whether the relationship is linear, monotonic, or more complex. Individual Conditional Expectation (ICE) plots show predictions for individual instances.
For a house price model, PDP showed that square footage had a log-linear relationship with price up to 3,000 sq ft, then plateaued. This insight led to feature engineering that captured the diminishing returns, improving model accuracy.
from sklearn.inspection import PartialDependenceDisplay\nPartialDependenceDisplay.from_estimator(model, X_train, features=["sqft_living"], kind="average", grid_resolution=50)\n# With ICE curves\nPartialDependenceDisplay.from_estimator(model, X_train, features=["sqft_living"], kind="both", grid_resolution=50)Permutation Feature Importance
Permutation importance measures how model performance drops when a feature's values are randomly shuffled. It is model-agnostic and accounts for feature interactions. Features with high importance cause large performance degradation when permuted.
I use permutation importance for model debugging. When a feature known to be unimportant in the domain showed high permutation importance, it indicated data leakage. Investigating revealed the feature was a future-looking variable that would not be available at prediction time.
from sklearn.inspection import permutation_importance\nresult = permutation_importance(model, X_test, y_test, n_repeats=10, random_state=42)\nfor i in result.importances_mean.argsort()[::-1]:\n print(f"{feature_names[i]}: {result.importances_mean[i]:.3f} +/- {result.importances_std[i]:.3f}")Global Surrogate Models
Global surrogate models train an interpretable model (decision tree, linear model) to approximate the black-box model's predictions. The surrogate sacrifices some fidelity for interpretability, providing a global view of the model's decision boundaries.
I trained a decision tree surrogate for an XGBoost credit model. The surrogate achieved 85 percent agreement with the original model while being fully interpretable. The resulting tree revealed that payment history and debt-to-income ratio were the primary decision drivers, which aligned with business domain knowledge.
from sklearn.tree import DecisionTreeClassifier, export_text\nsurrogate = DecisionTreeClassifier(max_depth=4)\nsurrogate.fit(X_train, model.predict(X_train))\nprint(export_text(surrogate, feature_names=feature_names))\n# Check fidelity\nfidelity = (surrogate.predict(X_test) == model.predict(X_test)).mean()Integrated Gradients and Saliency Maps
For deep learning models, gradient-based methods provide feature attribution. Integrated Gradients satisfies sensitivity and implementation invariance by integrating gradients along the path from baseline to input. Saliency maps visualize which pixels influence CNN predictions.
For a medical image classifier, Integrated Gradients highlighted that the model focused on the correct anatomical region for diagnosis. Saliency maps showed that prediction confidence correlated with how closely the highlighted region matched expert annotations, validating the model's clinical relevance.
import torch\nfrom captum.attr import IntegratedGradients\nig = IntegratedGradients(model)\nattributions, delta = ig.attribute(input, target=0, return_convergence_delta=True, n_steps=100)\n# For CNN: visualize attribution as heatmap\nimport matplotlib.pyplot as plt\nplt.imshow(attributions.squeeze().cpu().numpy(), cmap="hot")Frequently Asked Questions
What is the difference between SHAP and LIME?
SHAP provides consistent, theoretically grounded feature attributions using Shapley values. LIME is faster but less stable. SHAP is preferred for regulatory compliance, LIME for rapid prototyping.
Do I need XAI for every model?
For high-stakes decisions (credit, healthcare, hiring), XAI is essential and often legally required. For low-risk internal models, basic feature importance may suffice.
How do I explain deep learning models?
Use Integrated Gradients, Grad-CAM for CNNs, attention visualization for transformers, or SHAP's DeepExplainer. The choice depends on model architecture and explanation type needed.
Can explanations be misleading?
Yes. Explanations are approximations and can be unreliable if the surrogate is inaccurate, features are correlated, or the explanation method has high variance. Always validate explanations with domain experts.
Originally published on Ayodhyyya. Last updated June 1, 2026.