Ancestry-weighted mismatch baselines inherit the uncertainty of the panels and proportions they are built from

Question: How do methods for estimating relatedness from low-coverage ancient DNA handle reference panel choice and ancestry proportion uncertainty when assigning pairwise baseline mismatch rates in admixed individuals?

Answer

Taken together, the retrieved papers support a two-part answer. First, the field has largely converged on replacing a single global mismatch baseline with a pair-specific baseline computed as an ancestry-weighted mixture of within-population diversity and between-population divergence, using the ancestry proportions of both individuals in the pair — and this is demonstrably better than homogeneous-background approaches in admixed and heterogeneous scenarios [doi:10.64898/2026.10.13.758828]. Second, no retrieved method actually handles the uncertainty part of the question: ancestry proportions enter the baseline as point estimates drawn from an admixture model whose likelihood is often flat and whose maximum-likelihood estimate is non-unique [doi:10.1101/2024.10.18.619150], while the reference panel choice that defines source divergence and supplies per-site allele frequencies propagates errors of comparable magnitude to the kinship signal being estimated [doi:10.1101/2024.07.01.601500] [doi:10.64898/2026.07.30.741148]. Across these papers, ancestry-aware normalization is a practical improvement, but it relocates rather than resolves the reference-panel and proportion-estimation problems: uncertainty is minimized by better upstream estimates, not propagated into the pairwise baseline.

A single global mismatch baseline is biased by anything that varies between the two genomes being compared

Approaches that assume a homogeneous background of relatedness bias kinship estimates whenever individuals have distinct ancestry, which is why READv2 and similar single-baseline tools mis-score admixed pairs [doi:10.64898/2026.10.13.758828]. The assumption is worse than merely convenient: the "standard admixture" model treats admixed source populations as unrelated, whereas a generalized super-admixture model shows they are substantially related and yields pair-level relatedness estimates at kinship-model resolution [doi:10.1101/2024.12.29.630632]. Ancestry is not the only factor that displaces the baseline in the same direction: residual post-mortem damage shifts estimates, with uncorrected non-UDG data moving TKGWV2 toward second-degree relationships while corrected comparisons supported first-degree relatedness, and no single damage-correction method performing best across all measures [doi:10.64898/2026.09.07.749785]. Multiple contributors to a single low-coverage or sedimentary sample are the main obstacle to accurate kinship inference in that setting, and require an explicit heterozygosity-based screen rather than a baseline adjustment [doi:10.64898/2026.02.02.703194].

Ancestry-weighted pair-specific baselines outperform homogeneous assumptions down to 0.03x coverage

mixKIN estimates a pair-specific baseline by weighting within-population diversity and between-population divergence according to the ancestry proportions of both individuals in each pair, rather than applying one mismatch baseline to all pairs [doi:10.64898/2026.10.13.758828]. Benchmarked against correctKin and RelateAdmix — which account for population structure and admixture — and against READv2 across distant-population admixture, intra-population heterogeneity and genetic outliers at coverages from 1x to 0.03x, it achieved higher average classification performance in admixed and heterogeneous scenarios while remaining comparable in homogeneous populations, with relationships evaluated to third degree [doi:10.64898/2026.10.13.758828]. In an empirical medieval English cemetery it recovered relationships involving an individual of West African and European ancestry without the manual adjustment for admixture that the homogeneous-baseline workflow required [doi:10.64898/2026.10.13.758828].

Reference panel choice is not a nuisance parameter: it fixes both the source definitions and the per-site frequencies the baseline is built from

Any descent measure is defined relative to a reference set of allele pairs, and estimators that preserve descent-measure rankings across different reference sets — allele-sharing estimators among them — are preferable to those whose rankings shift with the panel [doi:10.64898/2026.01.26.701864]. That paper goes further and recommends not estimating allele frequencies at all, since doing so confounds descent estimates for a target pair with the status of all pairs in the reference set [doi:10.64898/2026.01.26.701864] — a direct tension with ancestry-weighted baselines, which require allele-frequency-derived divergence parameters. The magnitude at stake is measurable: in pseudohaploid aDNA simulations with realistic damage, mapping bias alone produces differences in estimated admixture proportion of up to 4% depending on the reference population, while method choice produces differences of 10%, and per-site allele-frequency noise from low sequencing depth remains uncorrected even after genotype-likelihood adjustment [doi:10.1101/2024.07.01.601500]. Reference panel size is far from saturated for ancestry inference generally, and the statistic summarizing reference matching plus panel size matter more than model architecture [doi:10.64898/2026.07.30.741148]. Poorly matched references are common enough that local ancestry methods originally requiring a close match per ancestry have been explicitly extended to clustered, unmatched panels [doi:10.1101/2025.10.13.681993], and aDNA is specifically characterized by heterogeneous geographic and temporal sampling and by variable divergence between ancient source proxies and the admixing populations [doi:10.64898/2026.05.19.725905]. Imputation, the route by which such panels reach aDNA data, depends strongly on panel composition and genetic divergence, motivating region-specific enriched panels [doi:10.64898/2026.08.13.744432].

Ancestry proportions enter the baseline as point estimates from a model that does not identify them uniquely

The likelihood in the admixture model is typically flat in some direction around the maximum-likelihood estimate, the estimator is non-unique, and there is a complete spectrum of equally likely admixture proportions; for inter-European comparisons this can be a large set, particularly with highly differentiated markers and concentrated proportions [doi:10.1101/2024.10.18.619150]. Empirical benchmarking of eight local ancestry methods on 176 imputed Neolithic genomes found individual-level ancestry estimates highly correlated across methods while inferred tract lengths and admixture times varied by an order of magnitude [doi:10.64898/2026.04.23.720248]. The practical implication for a pairwise baseline is that global ancestry proportions — the natural input to a pair-level weight — appear robust across methods, whereas local ancestry, if used, is not, and the reliability of all these methods in imputed ancient genomes remains formally unresolved because they assume large present-day reference panels [doi:10.64898/2026.05.19.725905]. Method-driven differences in estimated proportions of around 10%, with qpAdm performing best but remaining mapping-bias-sensitive, set the scale of error that a baseline inherits when proportions are treated as known [doi:10.1101/2024.07.01.601500].

Segment-based and frequency-free estimators avoid estimating a baseline, at the cost of a different panel dependence

Allele-sharing estimators that do not estimate allele frequencies sidestep the reference-set confounding problem entirely and preserve relatedness rankings across panels [doi:10.64898/2026.01.26.701864]. Identity-by-descent segment detection is the other bypass: clusIBD detects IBD from unphased data with high genotype errors and outperformed IBIS, TRUFFLE and IBDseq on simulated, artificially degraded and ancient DNA material [doi:10.1101/2025.03.13.643019], and ancIBD inferred the same widespread sharing of one IBD chromosome copy across all damage-correction strategies tested [doi:10.64898/2026.09.07.749785]. The substitution is not free: these methods consume imputed or phased genotypes, and imputation accuracy at 0.25x ultra-low-pass coverage is achievable with modern reference panels but degrades for assays and ancestries where the panel is poorly matched [doi:10.64898/2026.05.18.726085], while imputation performance in aDNA is bounded by panel composition and divergence [doi:10.64898/2026.08.13.744432]. Downstream pedigree reconstruction consumes these pairwise classifications as input and is evaluated under varying error rates and missingness, so misclassified admixed pairs propagate directly into topology [doi:10.1101/2025.08.21.671608].

Conflicts and gaps

The central unresolved conflict is methodological philosophy. Weir and Goudet argue that allele frequencies need not and should not be estimated, because doing so confounds the target pair's descent estimate with the whole reference set [doi:10.64898/2026.01.26.701864]; ancestry-aware normalization does exactly the opposite, making the baseline an explicit function of ancestry proportions and between-population divergence [doi:10.64898/2026.10.13.758828]. No retrieved paper tests these two prescriptions against each other on the same low-coverage admixed data.

Second, mixKIN reports an advantage over correctKin and RelateAdmix, both of which "account for population structure and admixture" [doi:10.64898/2026.10.13.758828]; the abstract does not quantify that advantage, does not state whether the ancestry proportions used are global or local, and evaluates relationships only to third degree — precisely the range where the baseline error is largest relative to signal.

Third, the super-admixture result that admixed source populations are themselves substantially related [doi:10.1101/2024.12.29.630632] undercuts the unrelated-sources assumption implicit in mixture-weighted baselines, but no retrieved study quantifies whether that coancestry matters at the pairwise baseline level for aDNA-scale samples.

Fourth, no retrieved method propagates ancestry-proportion uncertainty into the baseline. Point estimates from a flat, non-unique likelihood [doi:10.1101/2024.10.18.619150], with method spreads up to 10% [doi:10.1101/2024.07.01.601500] and order-of-magnitude variation in local ancestry tracts [doi:10.64898/2026.04.23.720248], are converted into a single number per pair with no interval. How much a 4% admixture-proportion error moves the baseline relative to a third-degree kinship signal is unquantified in these papers.

Fifth, no retrieved study benchmarks the mismatch baseline itself across reference panels, as distinct from benchmarking imputation or local ancestry accuracy across panels [doi:10.64898/2026.08.13.744432] [doi:10.64898/2026.05.19.725905].

What would change the answer

A factorial simulation would settle most of this: admixed pedigrees at 1x–0.03x, crossed with (i) reference panels ranging from matched present-day to impoverished or region-specific enriched panels, (ii) ancestry-proportion error injected at the magnitudes reported for mapping bias and method choice [doi:10.1101/2024.07.01.601500] and across the flat-likelihood spectrum [doi:10.1101/2024.10.18.619150], and (iii) damage and contamination strata [doi:10.64898/2026.09.07.749785] [doi:10.64898/2026.02.02.703194], reporting classification error for ancestry-weighted baselines, for correctKin/RelateAdmix-style corrections and for frequency-free allele-sharing estimators [doi:10.64898/2026.01.26.701864] on identical sites. Reporting kinship as a posterior or interval rather than a point classification would follow directly from such a design.

Two concrete empirical levers exist. First-degree pairs across ancestry extremes give an independent check on any baseline: IBS0 approaches zero for parent-offspring regardless of ancestry weighting [doi:10.64898/2026.09.07.749785], so baseline miscalibration should show up as spurious IBS0 elevation in unrelated cross-ancestry pairs drawn from the same cemetery — the design mixKIN applied to its medieval English individual [doi:10.64898/2026.10.13.758828]. Second, ancestry-specific rather than aggregate reference frequencies would reduce the input error directly: in admixed American and African/African American groups, 78.5% and 85.1% of variants respectively show at least a twofold difference between ancestry-specific frequencies [doi:10.1101/2024.10.30.620961], which means aggregate panel frequencies can misstate a pair's expected mismatch rate by more than the correction being applied. Substituting LAI-derived ancestry-specific frequencies into the pair-specific baseline is the most direct test of whether panel choice or proportion uncertainty dominates the residual bias.