Decoupling Heritability: Quantifying G x E Interactions in Home Cultivar Trials

Every serious home cultivar trial faces the same frustration: a variety that shines one season flops the next, or performs brilliantly in a friend's garden but disappoints in yours. That's the genotype-by-environment (G x E) interaction at work. If you cannot separate genetic potential from environmental noise, your selection decisions are essentially gambling. This guide shows you how to design trials that quantify G x E, letting you estimate true heritability and make confident selections.

Why Home Trials Need to Decouple Heritability

Most home trialists start with a simple comparison: grow a dozen tomato varieties, measure yield, pick the top performer. The problem is that yield is a composite of genetic potential, local soil fertility, weather quirks, pest pressure, and random chance. If you select based on a single season in a single location, you are likely selecting for luck, not genetics.

Heritability in the broad sense is the proportion of phenotypic variance that is genetic. But that ratio depends entirely on the environment in which it was measured. A trait like days to flowering may have high heritability across most conditions, while yield often has moderate to low heritability because it is heavily influenced by G x E. Without quantifying the interaction term, your heritability estimate is meaningless for predicting performance elsewhere.

What usually breaks first in home trials is the assumption that a variety's rank order stays consistent. When crossover interactions occur—where variety A beats B in one environment but loses in another—simple rankings mislead. We've seen a bean variety that produced 30% more pods in a cool, wet spring but succumbed to root rot in a warm, dry one. Selecting it based on the first year would have been a costly mistake.

Decoupling heritability means partitioning variance into three components: genetic (G), environmental (E), and their interaction (G x E). Once you know how much variance comes from interaction, you can decide whether to select for broad adaptation (low G x E) or exploit specific adaptations (high G x E for niche environments). That decision is the core of any serious home breeding program.

The Cost of Ignoring G x E

Without accounting for interaction, you overestimate heritability and select genotypes that are actually environment-specific winners. The next season, or a different bed, reveals the deception. Many home trialists quit after two seasons of inconsistent results, blaming the seed rather than their trial design. Quantifying G x E is not just academic—it saves years of wasted effort.

Prerequisites: What You Need Before Starting

Before you can quantify G x E, you need a trial structure that generates the necessary data. That means at least two environments (locations or years) and a minimum of three replications per environment. One environment cannot produce an interaction term—you need variance across environments to estimate the crossover.

Define your environments carefully. An environment is any distinct set of conditions that could affect trait expression: different garden beds, soil types, planting dates, irrigation regimes, or years. The key is to replicate genotypes within each environment and also replicate environments (or sample them representatively). For a home trial, three environments is a practical minimum; five or more gives much cleaner estimates.

Choose traits that are likely to show heritable variation. Days to emergence, flower color, and plant height often have high heritability and low G x E. Yield, fruit size, and disease resistance typically show moderate to high interaction. Focus on traits where you suspect environment sensitivity, because those are the ones where decoupling matters most.

You also need a statistical framework. The simplest is a two-way ANOVA with genotype and environment as fixed effects and replication nested within environment. More advanced options include AMMI (additive main effects and multiplicative interaction) and GGE biplots, which visualize which genotypes win in which environments. For home trialists, a spreadsheet with ANOVA calculations is enough to start; we'll walk through the process.

Minimum Data Requirements

For each genotype-environment combination, you need at least two measurements (replicates) to estimate within-environment variance. Three or four replicates per environment are better. Missing data can be handled with balanced designs, but plan for complete blocks to keep analysis simple. If you use a randomized complete block design (RCBD) in each environment, you can pool error across environments for more power.

The Core Workflow: From Plot to Interaction Estimate

Step one is to lay out your trial. Within each environment, use RCBD with at least three blocks. Randomize genotypes within each block. This controls for within-environment heterogeneity (e.g., shade gradient, soil variation). Across environments, keep the same number of blocks and the same planting density if possible.

Step two: collect data. Measure every plant in every block for your trait of interest. Record any environmental covariates like soil moisture, temperature, or pest incidence—they help explain interaction later. Enter data into a spreadsheet with columns: environment, block, genotype, trait value.

Step three: run a two-way ANOVA. Most spreadsheet programs can do this with the Data Analysis Toolpak (Excel) or built-in functions (Google Sheets with add-ons). The ANOVA table partitions variance into sources: environment, genotype, environment x genotype, and error. The F-test for the interaction term tells you whether G x E is significant. A significant interaction means rank changes exist; you need to explore them.

Step four: estimate variance components. From the mean squares, you can calculate the variance due to genotype (σ²G), environment (σ²E), interaction (σ²GE), and error (σ²). Broad-sense heritability on an entry-mean basis is H² = σ²G / (σ²G + σ²GE/nE + σ²/(nE * nR)), where nE is number of environments and nR is number of replicates. If σ²GE is large relative to σ²G, heritability drops sharply.

Step five: visualize interaction. A simple interaction plot (genotype means per environment) shows crossover patterns. If lines cross, there is crossover interaction. For deeper insight, run an AMMI analysis—free online tools exist (e.g., R package 'agricolae'). The AMMI biplot shows which genotypes are stable (near the origin) and which environments are discriminative.

Example Calculation

Suppose you have 5 tomato genotypes tested in 3 environments with 4 blocks each. Your ANOVA gives MSgenotype = 120, MSinteraction = 80, and MSerror = 40. With 3 environments and 4 replicates, σ²G = (120 - 80) / (3*4) = 3.33; σ²GE = (80 - 40) / 4 = 10; σ²error = 40. Heritability H² = 3.33 / (3.33 + 10/3 + 40/12) = 3.33 / (3.33 + 3.33 + 3.33) = 0.33. So only 33% of phenotypic variation is genetic—the rest is interaction and error. That's a wake-up call to increase replication or environments.

Tools, Setup, and Environmental Realities

You don't need expensive software. A spreadsheet with ANOVA capabilities is sufficient for basic partitioning. For AMMI and GGE biplots, use R (free) or online web apps like PBtools or the 'metan' R package. The learning curve for R is steep, but the payoff is large: you can generate biplots and stability measures with a few lines of code.

In the field, the biggest challenge is creating true environment replicates. Home gardeners often have only one garden bed. Solutions: use different planting dates (early vs late season) as environments, or split the bed into sections with different soil amendments, or collaborate with neighboring gardeners to share trials. Even two distinct microclimates within a yard (sunny vs partly shaded) can serve as separate environments if they differ enough.

Another reality: environmental variance within a block can swamp genotype effects if blocks are too large. Keep blocks compact and uniform. If your garden has a slope, orient blocks perpendicular to the slope. Use border rows to reduce edge effects. Measure and record environmental covariates—they can be used as covariates in ANCOVA to reduce error.

For perennials, year-to-year variation is the primary environment axis. You need at least three years of data to estimate G x E reliably. Plan for attrition: some plants will die. Increase initial replication to account for losses.

When to Use AMMI vs. Factorial ANOVA

Factorial ANOVA with fixed effects is fine when you have few environments (2–4) and want to test significance. For many environments (5+), AMMI is superior because it models interaction patterns and reduces noise. Start with ANOVA to get variance components, then use AMMI for visualization if interaction is significant.

Variations for Different Constraints

If you are limited to a single location across years, treat each year as an environment. This captures year-to-year weather variation but confounds location effects. You cannot generalize to other sites, but you can select for broad adaptation across seasons. Use at least three years; two years only gives one degree of freedom for interaction, which is statistically weak.

If you have multiple locations but only one season, you can estimate spatial G x E. This is useful for selecting varieties for different soil types or climates. However, you miss year effects, so recommendations are specific to that season's weather. Combine multi-location trials over multiple years if possible.

For very small trials (fewer than 10 genotypes), use augmented designs with repeated check varieties. The checks provide a baseline to estimate block effects and environment effects. Then genotype effects are estimated relative to checks. This sacrifices some precision but still allows interaction estimation if checks are included in every environment.

If you lack statistical software, use the rank-sum method: rank genotypes within each environment, then compute the variance of ranks across environments. A high variance indicates that the genotype's rank changes—i.e., interaction. This non-parametric approach is quick and requires no ANOVA, but it gives no formal heritability estimate.

When to Abandon Broad Selection

If your analysis shows that σ²GE is larger than σ²G, broad selection (choosing one variety for all environments) will be inefficient. Instead, consider selecting different genotypes for each environment or grouping environments into mega-environments. GGE biplots help identify which environments cluster together—those can be treated as one selection zone.

Pitfalls, Debugging, and What to Check When It Fails

The most common failure is non-significant interaction but still poor heritability. That often means error variance is too high—your blocks are too variable or measurement error is large. Check your blocking: are the blocks within an environment really uniform? If not, use a more refined blocking scheme (e.g., incomplete blocks). Also check for missing data: unbalanced designs inflate error.

Another pitfall: using too few environments. With two environments, interaction is confounded with environment-specific error. You need at least three environments to separate interaction from error. Always test for significance—if interaction is not significant, you can pool it with error to increase power for genotype comparisons.

Confounding block effects with environment effects happens when blocks are treated as environments. For example, if you have two blocks in one garden and treat each as an environment, you are actually measuring within-garden variation, not real environmental differences. Genuine environments must differ in factors that affect trait expression (soil, climate, management).

If your heritability estimate is negative (which can happen with ANOVA variance components), set it to zero. Negative estimates indicate that the true component is very small and sampling error pushed it negative. This is a signal that your trial lacks power—increase replication or environments.

Finally, avoid over-interpreting biplots. AMMI biplots are sensitive to scaling; always check the proportion of variance explained by the first two interaction principal components. If it's below 60%, the biplot may be misleading. Use the numeric stability measures (e.g., regression coefficient from Finlay-Wilkinson) alongside biplots.

Debugging Checklist

• Is the interaction term significant at p<0.10? If not, consider pooling.
• Are error variances homogeneous across environments? Use Levene's test.
• Do you have at least 3 replications per environment? If not, results are unreliable.
• Are there outliers? Examine residuals; remove extreme values if justified.
• Did you randomize properly? Systematic planting order can bias results.

FAQ: Common Questions from Home Trialists

How many genotypes do I need?

At least 5 for meaningful variance estimation; 10–20 is better. Fewer than 5 and you cannot estimate genotype variance reliably.

Can I use different numbers of replicates per environment?

Yes, but balanced designs are easier to analyze. If unbalanced, use restricted maximum likelihood (REML) estimation, available in R or SAS.

What if I only have one season of data?

You cannot estimate G x E across years. Use multiple locations instead, or treat different planting dates as environments. But remember: one season's results are not generalizable.

Should I select for high heritability or low G x E?

It depends on your goal. For a stable variety for many gardens, select for low G x E (broad adaptation). For a niche variety optimized for your specific conditions, high G x E is acceptable if you only recommend it for that environment.

How do I handle perennial crops that take years to fruit?

Record data annually and treat year as environment. Use repeated measures analysis if you have the same plants across years. Consider using the first year as a covariate to adjust for plant size differences.

Start applying these methods in your next trial. Even a simple two-environment, three-replicate design will reveal whether your favorite variety is truly superior or just lucky. Over multiple seasons, you will build a dataset that lets you select with confidence—and that is the difference between a hobby and a real breeding program.