Ok, so after last post, I really ought to tackle “but what do you think would be better than SHAP?”. In particular, it seems to me to be an enormous waste to go to all that computing just to produce a single number; could you get more information out of it?
Here is a quick experiment. I’d already run a computation obtaining Shapley values for dropping features from a prediction set. In particular, I used a subset Beijing Housing Data with the task of predicting log totalPrice and looked at Random Forest OOB squared error as my value function when the forest was trained with different subsets of features.
Here I used 100 permutations, and for each feature I attributed to it the change (almost always a decrease) in OOB squared error when it entered the model in each permutation.** After I removed features that were annoying to deal with, we had 18 features so I had a 100-by-18 matrix of permutation-by-SSE change.
The Shapley value estimate is just the average over those 100 matrices, but I think we can see how there might be more information here. In particular, part of the information that I thought was useful was dependence between features, along the lines of “You need one of size and number of rooms, but not both.”, or “Whether there is an elevator doesn’t tell you much unless you know the number of bathrooms” (although I still haven’t worked out what’s going on with that last one). Well here we can see some of this, from the order in which the features enter the model.
In particular, for each feature, I created a new 100-by-17 matrix with the Hadamard indicator of precedence of the other features. That is, when I was looking at the importance of X1, say, I went through each permutation and recorded whether X2 came before or after the X1. If before I recorded this as 1, and -1 if after.
Now I obtained a linear regression of the X1 column of my OOB Change matrix on this new precedence matrix. A positive coefficient for the precedence indicator of X2 indicates that X1 appears to make more difference when X2 is already in the model, a negative coefficient indicating that X1 doesn’t add as much information when X2 is already in the model compared to when it isn’t. In this case, we have the following
Intercept Lat DOM followers square livingR drawingR kitchen bathR buildiType
0.0231 -0.0023 0.0006 -0.0018 -0.0004 -0.0013 -0.0012 -0.0013 0.0009 0.0009
construction renovation Structure ladderRat elevator fiveYear subway district
0.0006 -0.0022 -0.0025 -0.0007 -0.0017 -0.0027 -0.0031 -0.0084
If we did all possible permutations, symmetry arguments give us that the intercept this is model should give us the Shapley value that we calculated. It won’t exactly, but the correlation was 0.987 which isn’t bad; the Shapley value for X1 here is 0.026.
Of course, this analysis only allows for main effects and wouldn’t tell us something like “you need two out of three of X1, X5 and X9”, but we can get an idea of how much a linear model tells us about the importance of X1 in the model.
I’ve summarized all of this information in the graph below. Here, nodes correspond to features and an arrow connecting X1 to X2 indicates the effect that the presence of X1 has on X2. The thickness of the arrow indicates the strength of the effect and the color gives direction (red if negative, blue if positive). I’ve annotated the node name with the R-squared of each linear model.
Clearly there is a lot to do here still. I thresholded effects at a rather arbitrary 0.003, one might instead use a LASSO penalty as in LIME (although see this paper on cautions). And a lot of the R-squared values are fairly low indicating that something might be gained from higher-order interactions, although you’d need a new way of visualizing those. Most importantly, this graph is very unstable — repeat the calculation and you’ll get something that looks rather different. But perhaps there is potential.
Note that I’ve employed this in the service of global interpretation — how important are my features to the model as a whole. I’m not sure that this would be as helpful at local scales and I still find the value function that SHAP uses to be far too abstract to be a useful explanation method. But perhaps someone can produce a convincing use case.
** OK, technically, my value function is total SS minus Error SS, but it works out to the same attribution.
with all the features, and 
with everything except 
?.
. That is, how much does my error rate increase when I remove 
without it?
are nearly interchangeable? Maybe one measures my height and the other is my trouser inseam. I can predict one from the other nearly perfectly, which means that if I remove height from the model to get 
.
and can produce a value 
for any subset 
of the players and we want to find a way of working out a value 
for each individual player 
*****. This maps directly onto the problem above: 
being the error rate for a model learned only with 
.
which integrates out all the features not in the set 
.
is to the value of 
. So here we are breaking out 
into a sum of contributions, one from each feature. There’s some appeal here, but note that this is not a representation as an additive model — each of the 
is itself a rather complex function of all the features. To be actually fair, we should write 
always contributes the same as 
to any coalition, they have the same Shapley value
. That is, we could not say that the bias in our models was small enough to ignore when performing inference. What were we really learning if we couldn’t relate what we got to real-world quantities?
and data 
are given by ![I(X) = [\hat{\theta}(X)-l,\hat{\theta}(X)+u]](https://s0.wp.com/latex.php?latex=I%28X%29+%3D+%5B%5Chat%7B%5Ctheta%7D%28X%29-l%2C%5Chat%7B%5Ctheta%7D%28X%29%2Bu%5D&bg=ffffff&fg=111111&s=0&c=20201002)
where 
calculates an estimator for an unknown 
includes 
95% of the time.
so that 
is in 
is an new set of data independent of 
— you do have to take a distribution over both 
then 
— these are simply 
times as large as standard confidence intervals.
, and form 
.
Of Models and Meanings 