Question: HLSL: Final pixel color is different than the calculated return value of the pixel shader
Question created on Sunday February 26, 2017
After reading and trying to apply the advice from How to enable Hardware Percentage Closer Filtering? into my pixel shader, I got some very strange results when debugging - in that all of the pixels had very low RGB values although the shader returned some greater RGB float4 values.
This issue this only manifested itself while using the hardware (GPU) and was behaving properly when forcing WARP (via the DirectX Control Panel). I thought it was a driver problem, but tested on two different computers, one with a mobile nVidia GPU, the other with a desktop AMD card, and behavior is the same with both - it only works as expected with WARP.
Check out the following screenshots of debugging the Pixel Shader which I think are very informative about my encountered problem:
Debugging the pixel shader with a regular hardware run:
Have a look at the return value - and on the right - in the "Graphics Pixel History" at the Pixel Shader RGB values and Result values - how come they are so different? Nearly division by 3 factor.
Now, take a look at the same debugging - on a similar shadowed pixel - when running the same shader - only now using WARP instead of running on the GPU:
As you can see, the values are nearly identical - and I think the small diferences are because of the render target being 8 bit per channel while the float is 4 bytes (32 bits).
I implemented a 16-tap PCF - taking 16 samples with the Sample function instead of using the SampleCmpLevelZero func and doing the comparison with the target value myself, in the shader - for deciding the extent the pixel should be shadowed.
Upon reading the question mentioned above - I modified my shader to use SampleCmpLevelZero and that's when my problems appeared and my shaders only behaved as expected with WARP.
Can someone give me some suggestion on what else to check to further debug this situation?
After enabling the D3D Device Debugging - following the valuable instructions from here - I got the following message - and hence found out the cause for my issue:
D3D11 ERROR: ID3D11DeviceContext::DrawIndexed: The Shader Resource
View in slot 1 of the Pixel Shader unit is using the Format
(R16_FLOAT). This format does not support 'SampleCmp' or
'SampleCmpLevelZero', at least one of which is being used on the
Resource by the shader. This mismatch is invalid if the shader
actually uses the view (e.g. it is not skipped due to shader code
branching). [ EXECUTION ERROR #372:
This was reffering to my depth texture, for which I had set the DXGI_FORMAT_R16_FLOAT option. Changing to DXGI_FORMAT_R32_FLOAT has fixed the issue and allowed me to enable PCF via SampleCmpLevelZero.
I guess the moral is that it's very useful to enable DirectX device debugging to find out if your execution gets in a corrupt or erroneous state.