Sitecore XP 9.1 Scaled: Experience Analytics not working

Currently we are upgrading our Sitecore Solution to Sitecore XP 9.1 (initial release) and we are setting up Scaled environment as :

Read more about (Scaling Scenarios)

(CM/CD/PRC/RPT) -1

(XConnect )– 1

(Solr )- 1

(Proxy) – 1

After the Installation was completed and post installation steps were performed we used below testing matrix to verify the Vanilla 9.1 instance.

#	Checklist	Status
1	Verify Solr is running on the Solr server as Windows service	Passed
2	Verify that Sitecore loads on {url}/sitecore/login	Passed
3	Verify Sitecore cache is working at {url}/sitecore/admin/cache.aspx	Passed
4	Verify Sitecore showconfig page is loading at {url}/sitecore/admin/showconfig.aspx	Passed
5	Verify if you can build the indexes from Sitecore control Panel	Passed
6	Verify if xConnect loads	Passed
7	Verify if Marketing Automation windows service is running	Failed
8	Verify if Processing Engine windows service is running	Passed
9	Verify if Search-IndexWorker windows service is running	Passed
10	Verify if log files are generating properly.	Passed
12	Verify if Sitecore Content Editor loads	Passed
13	Verify the IIS user has access rights to the Web root	Passed
14	Verify if Analytics data is getting updated on Dashboard	Failed

All the above steps got passed except one and that was Sitecore Analytics.

First thing we did was checked the log files for the CM environment and we found errors related to the analytics and it was “The certificate was not found“.

Exception: System.InvalidOperationExceptionMessage: The certificate was not found.Source: Sitecore.Xdb.Common.Web   at Sitecore.Xdb.Common.Web.CertificateWebRequestHandlerModifier.Process(HttpClientHandler handler)

We had already checked that Xconnect was getting loaded successfully.

We checked for the Marketing Automation, Index worker and Processing engine service whether they were running and found out that the Marketing Automation service was not running.

We checked the connection strings from CM and Xconnect servers to be sure that a Valid thumbprint is associated wherever required and found out that in one place for the Xconnect server the Thumbprint was wrong. We updated the thumbprint and the Marketing Automation service got started.

We thought that might solve our issue but No, my thought was wrong. Still same error in logs.

We checked the IIS for the Xconnect server and verified whether a Valid Certificate was attached on the IIS for Xconnect.

We checked IIS app pool user rights were assigned to the Xconnect certificate and it was there.

Next step, We tried to load the Xonnect site from the CM server and it showed me an invalid certificate exception (If you are not able to access Xconnect from CM Server make sure you add host entry for Xconnect on CM). So, my doubt was confirmed that CM server was somehow not able to communicate with the Xconnect server.

Now we imported the Xconnect certificate to the CM server’s Local Computer/Personal/Certificates folder to see if it might solve the issue and to our surprise, the error vanished from the logs.

We tried to the hit the Xconnect site from the CM server again. The certificate exception was gone now and Xconnect loaded with the Valid certificate attached to it.

This was something we figured by try and error. I opened a Sitecore support ticket to understand this scenario and they also mentioned that the certificate should be present on both the Machines (i.e CM and Xconnect) in our case.

Good reads :

https://doc.sitecore.com/developers/91/platform-administration-and-architecture/en/enable-client-certificate-authentication.html

https://doc.sitecore.com/developers/91/platform-administration-and-architecture/en/configure-connections-strings.html

https://doc.sitecore.com/developers/91/sitecore-experience-management/en/the-client-certificate-for-sitecore-deployments.html