Summary

You may notice that the Post-Purchase Survey (PPS) ‘Views’ in the Retention Tab does not match the actual number of survey responses. This discrepancy occurs due to differences in data collection, session tracking, and how percentages are calculated.

Why This Happens

• Not All Orders Trigger a PPS View

  • PPS Views count all order confirmations where the survey was displayed.
  • Some orders may not generate a view if the confirmation page is not loaded (e.g., user closes the page).

• Survey Completion is Optional

  • A PPS view does not guarantee a response—some customers choose not to fill out the survey.

• Differences in Percentage Calculations

  • PPS Percentage = Responses / Views
  • Total Impact (TI) Percentage = Responses / Orders
  • Since not all orders generate PPS views, the two calculations will differ.

• Response Quality and TI Thresholds

  • The Total Impact (TI) model requires a 30% PPS response rate to include data.
  • If responses fall below this threshold, the model may filter out low-quality data.

Example Scenarios

Scenario 1: Confirmation Page Not Loaded

1. A store receives 1,000 orders in a day.
2. Only 800 order confirmations load the PPS, generating 800 views.
3. 200 customers complete the survey.
4. PPS Percentage: 200 / 800 = 25% response rate
5. TI Model Percentage: 200 / 1,000 = 20% response rate

Since not all customers loaded the order confirmation page, there is a discrepancy between the PPS percentage and the TI model percentage.

Scenario 2: Low Response Rate Filtering in TI

1. A store has 500 orders and 450 PPS views.
2. Only 50 customers respond, leading to:
   • PPS response rate: 50 / 450 = 11%
   • TI model response rate: 50 / 500 = 10%
3. Since the response rate is below 30%, the TI model excludes the PPS data from attribution.

The TI model prioritizes statistically significant data. If responses are too low, they may be filtered out, creating an additional discrepancy.

Scenario 3: Orders Processed Through Third-Party Checkout

1. A brand offers checkout through Shopify and an external subscription service.
2. 700 orders are placed through Shopify, generating PPS views.
3. 300 orders are processed through the subscription service, bypassing the PPS survey.
4. TI counts all 1,000 orders, but PPS only reflects the 700 from Shopify, leading to:
   • PPS Percentage: 200 / 700 = 28.5%
   • TI Model Percentage: 200 / 1,000 = 20%

Because not all orders go through Shopify’s checkout, PPS views undercount total orders, creating a gap between PPS and TI data.

How This Affects Reporting

• Lower PPS Response Rate in TI Model

  • Because TI uses orders instead of views, response rates may appear lower in TI compared to the Retention Tab.

• PPS Views Are Not the Same as Orders

  • Orders that do not reach the confirmation page will not be counted as views but are still counted in TI calculations.

• Filtered Data Can Lead to Missing PPS Data in TI

  • If response rates are too low or answers lack statistical significance, TI may exclude the data.

How to Interpret the Data Correctly

• Compare PPS Views and Orders Separately

  • Expect fewer PPS views than total orders due to page loads and customer behavior.

• Check the PPS Response Rate in Retention vs. TI

  • If the TI model shows lower response rates, it's likely due to differences in denominator calculations.

• Ensure Your PPS Has a High Completion Rate

  • A response rate above 30% helps maintain data integrity in TI.

By understanding how PPS views, responses, and TI calculations differ, you can better analyze survey data and avoid misinterpretations.