Demo Workspace
Experiments
Default: English
Experiments
5
In Progress
2
Avg Confidence
75.18%
Avg Progress
32.89%
Experiment Summary FormulasScore formula
These formulas only explain the top experiment summary cards.
In Progress Count
Range: CountIn Progress Count = Count of experiments with status In Progress.
Weights / factors
- - Status comes from workflow, rollout, or task events in production.
Interpretation
Higher count means more recommendations have moved into active execution.
Data source
Experiment tracker status events.
Avg Confidence
Range: 0-100%Avg Confidence = Sum of experiment confidence values / Total experiments.
Weights / factors
- - Confidence should be calculated from evidence coverage, sample readiness, metric consistency, and owner confirmation.
Interpretation
Higher average confidence means the experiment set is more decision-ready.
Data source
AI evidence, sample size, metric backfill, and workflow confirmation.
Avg Progress
Range: 0-100%Avg Progress = Sum of experiment progress values / Total experiments.
Weights / factors
- - Progress is calculated per experiment from baseline, current value, and goal value.
Interpretation
Higher average progress means tracked experiments are closer to their goals.
Data source
Experiment metric windows and latest synced results.
Source Data
Shopify revenue/orders, ad spend, product events, support tickets, uninstall reasons, and CRM ownership are joined by merchant ID.
Experiment Creation
AI proposes hypotheses from root-cause evidence; the business owner confirms scope, metric, deadline, and rollout group.
Status Tracking
Backlog means waiting for owner approval; In Progress means rollout started; Needs Review means data is ready for decision.
Decision Rule
Each action keeps evidence, confidence, target metric, sample size, current result, decision rule, and next owner action.
Status Source Rules
Backlog
AI recommendation exists, but owner approval or start date is still pending.
In Progress
Owner confirmed and rollout has started; metrics are still accumulating.
Needs Review
End date or sample threshold is reached; owner needs to decide whether to scale.
Completed
Review decision is recorded and the action is scaled, stopped, or converted to a product task.
Invalid
Tracking failed, sample is polluted, or metric definition is not reliable.
Experiment Action Tracker
Track each recommendation from evidence to rollout, metric backfill, review decision, and owner follow-up.
Experiment tracker rows connect AI recommendations, rollout ownership, test groups, metric windows, and latest synced results.Data source
Data item
Real source
How to get it
Notes / limits
Hypothesis / evidence
AI recommendation log
Generate from root-cause evidence
The hypothesis should keep source recommendation ID.
Groups / rollout
Experiment, ad or product system
Sync control and treatment setup
Real status should come from workflow, not manual text.
Metric result
Shopify, billing, product events, support
Backfill by control/treatment cohort
Requires sample size and metric window.
Decision status
Task/workflow system
Update from approval, start, end and review events
This explains pending, in-progress, review and completed.
Source systems
AI recommendation log, task/workflow system, ad/product rollout records, Shopify, billing, support, uninstall capture.
Join keys
experiment_id, recommendation_id, merchant_id, campaign_id, owner
Demo scope
Mock rows use realistic source IDs, baseline windows, current windows, sample sizes, status reasons, and last sync timestamps.
Production integration
Create experiments from recommendations, sync rollout/task status, and backfill metrics by control/treatment cohorts from the warehouse.
Experiment Tracker Field FormulasScore formula
Experiment scores explain whether an experiment has enough evidence, enough progress, and a reliable status to support a decision.
Experiment Progress
Range: 0-100%Experiment Progress = (Current value - Baseline value) / (Goal value - Baseline value) x 100%, capped between 0 and 100%.
Weights / factors
- - Progress is capped between 0 and 100 in the Demo.
Interpretation
Higher progress means the experiment is closer to its target metric.
Data source
Baseline value, current value, goal value, and metric sync window.
Experiment Confidence
Range: 0-100%Experiment Confidence = Evidence quality score + Sample readiness score + Metric consistency score + Owner confirmation score.
Weights / factors
- - Current Demo uses prepared confidence values close to realistic workflow outputs.
- - Production should calculate it from evidence coverage, sample size, metric stability, and owner confirmation.
Interpretation
Higher confidence means the experiment result is safer to review or scale.
Data source
AI recommendation evidence, sample size, metric backfill, and workflow confirmation.
Experiment Risk
Range: Low / Medium / HighExperiment Risk = High if tracking is invalid; Medium if review is needed; Low if confidence is at least 75%; otherwise Medium.
Weights / factors
- - This rule comes directly from the current Demo analytics service.
Interpretation
Risk explains whether the experiment can continue, needs review, or should be invalidated.
Data source
Experiment status and confidence.
| Experiment | Source / Evidence | Metric | Result | Groups | Status | Risk | Owner | Next Action |
|---|---|---|---|---|---|---|---|---|
Paid Search ICP Split Splitting paid search by merchant order tier and excluding low-intent keywords will improve 30-day retention without reducing qualified installs. North America merchants with estimated annual GMV above USD 1M / Paid Search | REC-ACQ-202608-01 AI recommendation generated from acquisition ROI, funnel setup drop, and uninstall evidence; confirmed by Growth Lead. Paid Search has high volume but weak retention. / Partner Referral cohorts show better fit at lower volume. / Paid Search payback period is materially longer than partner channels. | Paid Search 30-day retention Metric source: Shopify install events + attribution table + retained merchant cohort Baseline: 2026-07-01 00:00:00 至 2026-07-31 23:59:59 Current: 2026-08-12 00:00:00 至 2026-08-16 23:59:59 | 42.18% → 46.28% Change: 4.09% Goal: 50.00% Progress: 52.36% Confidence: 74.38% | Control: Existing broad Paid Search campaigns Treatment: ICP-filtered campaigns with low-intent keywords excluded 312 / 286 | In Progress Start date has passed and paid campaigns are already split into control and treatment ad groups. Last sync: 2026-08-16 09:40:00 | Medium | Growth Lead | Review treatment cohort CAC and setup completion before adding more budget. Continue if treatment 30-day retention improves by at least 5.00 percentage points without CAC increasing above 15.00%. Early signal is positive, but sample age is not yet long enough for full 30-day retention. |
24-hour Setup Rescue A checklist plus CSM assist within 24 hours after install will improve setup completion and reduce early uninstall. All new installs, excluding enterprise-managed onboarding / All acquisition channels | REC-FUNNEL-202608-02 AI recommendation generated from setup funnel drop, support setup friction, and incomplete setup before uninstall. Setup friction is the top support topic. / Incomplete setup appears before uninstall. / Setup Complete is the highest scored funnel bottleneck. | Setup completion Metric source: Product setup events + lifecycle automation logs + support routing log Baseline: 2026-07-15 00:00:00 至 2026-08-14 23:59:59 Current: Not started | 60.48% → 60.48% Change: 0.00% Goal: 72.00% Progress: 0.00% Confidence: 82.15% | Control: Standard onboarding email only Treatment: Checklist email plus CSM task when setup is incomplete after 24 hours 0 / 0 | Backlog Experiment draft is ready, but lifecycle message copy and CSM routing rule still need owner approval. Last sync: 2026-08-16 09:40:00 | Low | Product Growth | Approve lifecycle copy and CSM owner routing, then move status to In Progress. Launch if CSM owner confirms routing capacity; graduate if setup completion improves by at least 8.00 points. No result yet because the experiment has not started. |
Tier 5 Core Adoption Play Guided adoption of shipment tracking and returns workflows will improve Tier 5 GRR. Tier 5 high-GMV merchants / Mixed acquisition channels | REC-RET-202608-03 AI recommendation generated from GRR/NDR bridge and feature impact analysis; confirmed by Retention Owner. Tier 5 GRR is below benchmark. / Core feature adoption is lower in high-risk accounts. / Shipment Tracking shows a strong GRR lift. | Tier 5 GRR Metric source: Billing MRR bridge + merchant tier table + feature usage events Baseline: 2026-07-01 00:00:00 至 2026-07-31 23:59:59 Current: 2026-08-01 00:00:00 至 2026-08-16 23:59:59 | 72.04% → 73.18% Change: 1.15% Goal: 76.50% Progress: 25.67% Confidence: 78.52% | Control: Tier 5 accounts receiving standard lifecycle nudges Treatment: Tier 5 accounts receiving CSM-led core workflow adoption 42 / 39 | In Progress CSM outreach queue has been created and treatment accounts are receiving guided adoption tasks. Last sync: 2026-08-16 09:40:00 | Low | Retention Owner | Review account-level adoption gaps and assign remaining CSM tasks. Continue if treatment GRR improves by at least 3.00 points or at-risk accounts decline by 10.00%. Early GRR movement is positive, but full renewal impact needs the September billing cycle. |
Uninstall Reason Capture Capturing uninstall reasons at exit will improve recovery prioritization and product roadmap clarity. All merchants / All acquisition channels | REC-CHURN-202608-04 Product + Support created this experiment after reviewing uninstall reason gaps and support-linked uninstall cases. Uninstall reasons are currently mixed across support, survey, email, and manual sources. / Early uninstall is concentrated before setup completion. / Support-linked uninstall cases often lack a clean reason taxonomy. | Uninstall reason coverage Metric source: Shopify uninstall webhook + exit survey + support case linkage Baseline: 2026-07-01 00:00:00 至 2026-07-31 23:59:59 Current: 2026-08-05 00:00:00 至 2026-08-16 23:59:59 | 48.00% → 71.33% Change: 23.33% Goal: 75.00% Progress: 86.39% Confidence: 69.24% | Control: Previous uninstall capture without required reason taxonomy Treatment: Exit survey with standardized reason taxonomy and recovery owner field 64 / 58 | Needs Review End date is approaching and reason coverage improved, but the team must review whether captured reasons are specific enough for routing. Last sync: 2026-08-16 09:40:00 | Medium | Product + Support | Review taxonomy quality and add owner routing for setup, pricing, support, and missing feature reasons. Mark completed if reason coverage reaches 75.00% and at least 80.00% of reasons map to an owner. Coverage improved materially, but reason quality and owner mapping need review. |
High-Urgency Support Fast Lane Routing high-urgency setup and billing tickets to a fast lane will improve first-response SLA and reduce negative sentiment before uninstall. ICP and Tier 5 merchants / Support | REC-SUPPORT-202608-05 AI recommendation generated from support SLA, negative sentiment, and uninstall-linked ticket evidence. Negative support rate is high. / Setup and billing tickets appear before uninstall. / SLA misses create retention risk for high-value merchants. | First-response SLA Metric source: Support ticket system SLA events + merchant health table Baseline: 2026-08-01 00:00:00 至 2026-08-15 23:59:59 Current: Not started | 66.67% → 66.67% Change: 0.00% Goal: 85.00% Progress: 0.00% Confidence: 71.60% | Control: Standard support queue Treatment: Fast lane queue with setup and billing specialists 0 / 0 | Backlog Requires support operations approval and queue configuration before start. Last sync: 2026-08-16 09:40:00 | Medium | Support Lead | Define queue rule and assign support specialists. Launch if support lead approves staffing; continue if SLA improves above 85.00% and negative sentiment falls by 10.00%. No result yet because the experiment has not started. |
Showing 1-5 / 5
1 / 1
Implementation note: in production, status is updated by workflow events, task ownership, rollout start/end dates, and metric sync jobs. Manual edits should only override owner, scope, and final decision notes.