Feature flag impact tells you whether a feature actually moves user behavior in the direction you intended. In PostHog, you can automatically track how users in different flag variants differ in key metrics like conversion rate, retention, or feature adoption—without waiting for a formal experiment or statistical analysis from a separate tool.
Understanding Feature Flag Impact
Feature flag impact measures the statistical relationship between flag exposure and user outcomes. PostHog shows you the difference in metrics between users who see a feature and those who don't, giving you instant visibility into whether your change is actually working.
Enable feature flag impact tracking in PostHog
When you create a feature flag in PostHog, navigate to Feature Flags in the left sidebar and select your flag. Scroll to the Metrics section and toggle on the metrics you want to track. PostHog will automatically capture events for users in each flag variant—control (users without the flag) and treatment (users with it). You don't need to instrument anything new; PostHog correlates your existing events with flag exposure automatically.
// PostHog SDK captures flag exposure automatically
if (posthog.isFeatureEnabled('new_checkout_flow')) {
// Your new checkout experience renders here
} else {
// Original checkout flow
}
// PostHog ties this decision to all subsequent events from that userSelect metrics that align with your feature goal
Choose metrics that would actually change if your feature worked. If you're redesigning checkout, measure conversion rate or cart value. If you're testing a new notification, measure feature adoption or engagement. In the flag's Metrics tab, click Add metric and select from your existing events or create a formula that tracks your core KPI.
// Capture events that represent your success metric
posthog.capture('checkout_completed', {
revenue: cartTotal,
variant: 'new_flow', // optional: help yourself debug later
completion_time_ms: elapsed
});
// PostHog will count these completions and compare variant groupsReading and Interpreting Impact Results
PostHog displays impact as a statistical comparison with confidence intervals. You're looking at whether the difference between control and treatment is real or just random noise.
Find the impact dashboard for your flag
In PostHog, open Feature Flags, select your flag by name, and click the Impact tab. You'll see a table with each metric you're tracking. PostHog shows the control group's average, the variant group's average, the relative uplift (percentage change), and a confidence interval. Green means your variant is winning; red means it's underperforming; gray means you don't have enough data yet.
// Fetch flag impact data programmatically via PostHog's API
const flagId = 'your-flag-id';
const impact = await fetch(
`https://app.posthog.com/api/feature_flags/${flagId}/`,
{ headers: { 'Authorization': `Bearer ${YOUR_POSTHOG_API_KEY}` } }
).then(r => r.json());
console.log(impact.experiment); // contains impact stats if enabledInterpret statistical significance correctly
If the 95% confidence interval doesn't cross zero (or doesn't include both positive and negative values), you have evidence of real impact—not just random variation. This doesn't mean you should ship it immediately; it means the difference is unlikely to be due to chance. Still consider the size of the effect, how long you've run it, and what your business decision threshold is.
// Example impact metrics structure
const metricImpact = {
metric: 'conversion_rate',
control_value: 0.12,
variant_value: 0.15,
relative_uplift: 0.25, // 25% relative uplift
confidence_lower: 0.05,
confidence_upper: 0.45,
is_significant: true // confidence interval doesn't span zero
};
// With is_significant = true, you can act on this differenceCommon Pitfalls When Measuring Flag Impact
These are the mistakes most teams hit when they first start using feature flag impact in PostHog.
Don't run too many flags at once
If you're testing 10 flags simultaneously, random variance will create false positives. At a 95% confidence level, roughly 1 in 20 flags will show impact just by luck. PostHog doesn't automatically correct for this multiple-comparison problem, so be disciplined: run 2–3 flags at a time, or accept that you'll see false positives and use a higher confidence threshold (99% instead of 95%).
// Rough sample size calculation for reliable impact
const minimumEventsPerVariant = 5000;
const variantAllocationPercent = 0.5; // 50/50 split
const minimumTotalEvents = minimumEventsPerVariant / variantAllocationPercent;
// For a variant affecting only 10% of users, you'll need 100x more events
const rareFeatureEvents = minimumEventsPerVariant / 0.1;Watch for biased assignment to control vs. treatment
If your flag rolls out unevenly (e.g., always-on for premium users, off for freemium), the two groups won't be statistically comparable at baseline. Control and treatment must be equivalent in all other ways. PostHog uses hash-based assignment for this reason—it assigns variants pseudo-randomly based on user ID, ensuring each user gets the same variant consistently without introducing selection bias.
// Let PostHog's deterministic hashing handle assignment
const variant = posthog.getFeatureFlag('new_onboarding');
// PostHog hashes distinct_id + flag_key to assign 'control' or 'variant_a'
// This is consistent per user but random across users
// Avoid this pattern: breaks causality
// if (user.isPremium) { variant = 'variant_a'; } // selection bias!Common Pitfalls
- Measuring impact before you have enough users in each variant—aim for at least 5,000 events per variant before trusting the result
- Choosing metrics unrelated to what your feature actually does—if you measure signups for a checkout redesign, you'll see noise instead of insight
- Not accounting for time lag in metrics—retention and lifetime value take days or weeks to fully express, so don't measure impact 24 hours after rollout
- Assuming statistical significance means you should ship—a 2% conversion lift might be statistically significant but too small to justify the maintenance burden
Wrapping Up
Feature flag impact in PostHog is your fastest way to know if a change actually moves user behavior. By letting PostHog automatically compare metrics across variants, you skip the busywork and go straight to decisions. If you want to track flag impact automatically across all your tools and build decision dashboards that pull this data in real-time, Product Analyst can help.