AO for self-shadows
Standard Unity Shader
Playtime painter's terrain Shader
Light attenuation is a value that is calculated in a shader for each fragment (think: each pixel) to tell if the light is falling on its face or not. On both images left side is facing the sun. Now, ambient occlusion should not have any effect on it, and using it may feel like straight up wrong approach. Everyone who worked with shaders is likely to have had a stage, where just fiddling with values and formulas, and obtaining some nice-looking results felt rewarding. But later, it becomes clear that shaders have such a huge range of applications, that it's more valuable to have true command over them. But in this case, using ambient or height mask to change attenuation value actually, has a reasonable explanation:
even though attenuation is obtained from a dot product of normal and light direction, and not from sampling shadow mask, it's effect is the same: blocking directional light. Indented fragments have a greater chance of being shadowed by the surface elevation near. And Ambient occlusion contains information on chances of that happening.
Ambient Occlusion mask is used to block some of the light, which comes from all directions. It's an approximation. Until hardware gets to the point when raytracing in real time is possible, we'll have to cheat and approximate. So there is no discussion about, whether this is a physically accurate way - neither of them is. And it's not really about is it pretty: depends on taste and implementation. But this method can be built upon because it provides a very close approximation to how directional light would be blocked by surface's own relief. The formula used in the screenshots is:
1: float atten = saturate((dot(worldNormal, _WorldSpaceLightPos0.xyz)));
2: atten = saturate(atten - ambientBlock * 4 * (1 - atten ));
The first line is your standard attenuation method. And the second was made by trial and error. Doesn't really have any explanation behind it. Removing that line will produce the following image (left - without AO atten; right - with):
Ambient Occlusion texture represents how much light will reach current fragment from all directions (assumes light comes from all directions in equal amounts). So for light attenuation, it represents the probability of current fragment being shadowed by a texture's own relief next to it. Sometimes using height texture instead of Ambient may give better results. Depends on the time of the surface.
Directional Light only: