I believe, in GTL at least, the mask handles Reflectivity/Shininess, in a similar way to how the Alpha handles transparency.
They can also be used when items of the body become invisible - so when someone makes an open topped version of an existing closed top, the roof elements etc can be made transparent by the alpha, and made to not reflect anything by the mask. Like wise when stuff is added like extra lights etc.
I don't know a great deal more than that, tbh (and even that is based on how I've seen them used in the past).
The problem we had with the Capri was that when i saved to DDS file you could see the original capri form that model underneath...
i looked at other skins and saw that a mask was used i copped it and my problem went away.
but i did need to use a different line to describe the body
this is how it was wrote for the body 875_FORD_CAPRI_872_BODY1
and the mask was FORD_CAPRI_872_BODY1_MASK
this was the only way to get the skin to show in game..
you can see it in action in the new Capri skin pack