Just a quick note to anyone deploying SAML (in my case ADFS) for Splunk. I'm running 6.5.0 and although it seemed to be working early on after not too long I started to see the following error on logon:

The 'NotBefore' condition could not be verified successfully. The saml response is not valid.

After a little investigation it seemed likely that Splunk was rejecting the assertion from ADFS as it didn't like the "NotBefore" attribute. A little searching showed up that this may be due to clock skew between Splunk (the SP) and ADFS (the iDP).

There is a particularly good article on medium which explains the issue.

However we checked clocks and noted that they appeared to be in Sync. As time went on it became clear that the issue was intermittent and was probably due to very subtle clock drift between the SP and the iDP.

A bit more searching showed up a good question on Splunk Answers which seemed to be on the same track. Finally a Microsoft question on technet provided the full solution.

It seems Splunk is very aggressive when applying the NotBefore condition and doesn't allow for very much drift at all. As Splunk doesn't seem to have a configurable allowable window for this attribute we need to make a change in ADFS. Unfortunately this change isn't in the GUI but can be done using powershell and applied to the Splunk federation:

Add-PSSnapin Microsoft.Adfs.PowerShell #Load up the ADFS PowerShell plug in
Get-ADFSRelyingPartyTrust –identifier “urn:party:sso” #Just to see what the values were
Set-ADFSRelyingPartyTrust –TargetIdentifier “urn:party:sso” –NotBeforeSkew 2 #Set the skew to 2 minutes

Credit goes to the Technet answer for the above PowerShell goodness.

Applying this change seemed to fix up the issue so if you have the same thing hopefully this helps you.