Maximum Likelihood Criterion in Phylogenetic Tree Reconstruction

The maximum-likelihood (ML) criterion is one of the most important and statistically rigorous approaches used for phylogenetic tree reconstruction, first proposed by Felsenstein in 1981. Given a particular model of sequence evolution and a proposed tree structure, it is possible to calculate the likelihood, that is, the probability, that the observed sequences would have evolved along that particular tree.

The maximum-likelihood criterion selects, among all the candidate trees considered, the one tree that maximizes this likelihood value. Since this approach explicitly incorporates a statistical model of how sequences evolve, it allows for a more rigorous and often more biologically realistic comparison of alternative tree topologies than simpler distance-based methods.

Parameters Required for Likelihood Calculation

Three qualitatively different types of parameters must be specified in order to calculate the likelihood of a given set of sequences on a particular tree:

1. Tree topology – the branching pattern connecting the species.
2. Branch lengths – the amount of evolutionary change occurring along each branch of the tree.
3. Rate-matrix parameters – values describing the process of sequence evolution itself, such as base frequencies and the transition/transversion ratio.

It is possible to optimize all three of these categories of parameters simultaneously. Specialized programs exist that search through tree space while simultaneously adjusting the topology, the branch lengths, and the rate-matrix parameters, in an attempt to locate the overall maximum-likelihood solution. Alternatively, it is also possible to fix certain parameters at specified values while optimizing only the others. For example, one might choose to search for the ML tree topology and its branch lengths while keeping the rate-matrix parameters fixed at particular predetermined numerical values.

Comparing a Specified Set of Tree Topologies

The ML criterion can also be applied in a more restricted way, to distinguish between a limited set of tree topologies that have been specified in advance, rather than searching the whole of tree space. This targeted approach is particularly useful when bootstrapping analysis (using a distance-matrix method) has already indicated where the main uncertainties in a tree lie; in such cases, it becomes sensible to test a defined set of alternative trees that differ from one another specifically in their branching order at these uncertain points.

• For each of the user-specified trees, the value of the likelihood can be estimated by allowing the branch lengths and rate parameters to vary, while keeping the topology itself fixed.
• This procedure produces a ranking of the specified trees according to their likelihood values.
• In practice, the likelihoods of several different candidate trees will often differ from one another by only a very small amount, making it important to determine whether one tree is genuinely significantly better than the alternatives, rather than merely appearing better due to chance or estimation error.

The Kishino–Hasegawa Test

To assess whether the difference in likelihood between two trees is statistically meaningful, a test proposed by Kishino and Hasegawa (1989) is used.

• The total log likelihood of a tree is calculated as the sum of the log-likelihood values from each individual sitein the sequence alignment.
• If it is assumed that different sites evolve independently of one another, then the error in the total log likelihood can be estimated by calculating the standard deviation of the log-likelihood values across sites.
• Using this estimated error, two different trees can be compared by asking whether the observed difference in their log likelihoods is statistically significant relative to the estimated error in that difference.
• This test allows a researcher to determine whether one tree topology can be considered genuinely better supported than another, or whether the observed difference in likelihood could plausibly have arisen simply through chance variation among sites.

Application of ML to the Primate Phylogeny Example

Restricting the Search Using Pre-Defined Clades

In applying the ML criterion to the primate phylogeny example, a full, unrestricted search of tree space was not necessary, because several closely related groups of species were already known with reasonable confidence from prior analysis. The primary interest instead lay in resolving the early branch points of the tree, which determine how these already well-established, closely related groups are related to one another. For this purpose, the following six clades were specified in advance as fixed rooted subtrees, based on groupings already supported by the Neighbor-Joining tree:

1. (Mouse, Guinea Pig)
2. (Bushbaby, Lemur)
3. (Northern Tree Shrew, Large Tree Shrew)
4. (Western Tarsier, Philippine Tarsier)
5. (Saki Monkey, (Capuchin, ((Tamarin, Marmoset), (Howler Monkey, Spider Monkey))))
6. ((Proboscis Monkey, (Baboon, Rhesus Monkey)), (Gibbon, (Orangutan, (Gorilla, (Human, (Chimpanzee, Pygmy Chimpanzee))))))

By fixing the internal branching order within each of these six clades and allowing only the connections between the clades to vary, the total number of distinct unrooted tree topologies that needed to be considered was reduced to just 105.

Search Procedure

• An ML algorithm was used to optimize the branch lengths and rate parameters separately for each of the 105 possible topologies, and to identify the overall ML topology among them.
• This optimization included adjustment of branch lengths within each of the six clades, as well as adjustment of the internal branches connecting the clades to one another.
• Rearrangement of species within a clade was not permitted, since the internal structure of each clade was treated as fixed and already known.
• Because this algorithm performed an exhaustive search of all 105 possible topologies, it was guaranteed to find the true ML solution, provided that the six clades themselves had been correctly defined at the outset.
• This restricted approach also had the practical advantage of being computationally rapid, since the total number of topologies requiring evaluation was kept relatively small by fixing the well-supported clades in advance.
• An alternative approach would have been to use a standard, unrestricted ML search program that does not require clades to be specified beforehand; however, such a search would have required substantially more computer time, and would not have provided the same guarantee that every relevant topology had been fully searched.

Result Using the Simple JC Model

When this algorithm was run using the simple Jukes–Cantor (JC) evolutionary model, the resulting ML tree topology was found to be identical to the Neighbor-Joining tree obtained earlier, although the estimated branch lengths differed slightly between the two methods. However, several other topologies among the 105 examined had likelihood values only very slightly lower than this best topology, and statistical testing showed that these alternative topologies were not significantly worse than the ML topology. This meant that, at this stage, the maximum-likelihood approach had not resolved the problematic relationships that had already been identified as unreliable in the NJ tree (such as the placement of the tree shrews).

• One important reason for this lack of improvement was that the analysis was still using a very simple model of sequence evolution.
• The JC model makes the restrictive assumption that all four nucleotide bases occur with equal frequency.
• In the actual sequence data being studied, however, the true base frequencies were found to be 37.5% A, 24.7% C, 12.6% G, and 25.2% U — clearly far from equal, indicating that the JC model was not an appropriate representation of the true evolutionary process for this data set.
• It was also considered likely that the ratio of transition to transversion rates would differ significantly from the value of 1 assumed under the simplest models.

Result Using a More Realistic Model (HKY with Rate Variation)

To address these shortcomings, the ML analysis was repeated using the HKY model, which allows for unequal base frequencies and for differing transition and transversion rates. In addition, since the sequence alignment showed that some sites were essentially invariant across all species while others varied much more rapidly than average, a model incorporating variable rates across sites was also used. This model included a certain proportion of invariable sites, together with six categories of variable sites following a gamma distribution of rates. The same 105 topologies, based on the same six pre-defined clades, were then re-evaluated using this more realistic combined model.

• The resulting ML tree, shown in Fig. 8.12, differed from the earlier NJ tree in one key respect: the tree shrews were now positioned as outgroups to the primates, rather than being grouped incorrectly with the tarsiers as in the earlier NJ analysis.
• This demonstrates that the use of a more realistic evolutionary model was able to correct the main problem that had been present in the tree obtained under the overly simplistic JC model.

The Trifurcation Among Lemurs, Tarsiers, and Monkeys/Apes

An important additional feature of the resulting ML tree was the presence of a trifurcation — a single node giving rise to three branches simultaneously — among the lemurs, the tarsiers, and the combined monkey/ape group.

• There are three possible fully bifurcating topologies consistent with resolving this trifurcation, corresponding to each of the three groups (lemurs, tarsiers, or monkeys/apes) in turn being treated as the earliest-diverging outgroup relative to the other two.
• All three of these alternative bifurcating topologies were found to give exactly the same maximum-likelihood value, because the internal branch length separating them shrinks to zero, which is precisely what produces the observed trifurcation in the tree.
• Independent evidence from morphological studies suggests that the lemurs are, in fact, the earliest-diverging of these three groups.
• However, the single gene sequence used in this particular molecular analysis did not contain sufficient information to distinguish confidently between these three possible arrangements.
• Among the full set of 105 topologies examined, several other topologies also had likelihood values lower than the one shown in Fig. 8.12, but statistical tests indicated that these were not significantly less likely, reflecting the general pattern that many candidate topologies can remain statistically indistinguishable from one another even under a more sophisticated model.

Conclusion

The maximum-likelihood criterion provides a statistically grounded method for phylogenetic tree reconstruction by selecting the tree topology that maximizes the probability of observing the given sequence data, based on an explicit model of sequence evolution, tree topology, branch lengths, and rate-matrix parameters. When applied to a restricted, pre-defined set of candidate topologies, together with a statistical test such as that of Kishino and Hasegawa, it becomes possible to rigorously assess whether one tree is genuinely better supported than its close competitors.

The application of ML to the primate phylogeny example demonstrates the critical importance of the underlying evolutionary model: while the simple Jukes–Cantor model failed to resolve the same problematic relationships present in the earlier distance-based tree, the use of a more realistic model incorporating unequal base frequencies, differing transition/transversion rates, and among-site rate variation (the HKY model with invariant sites and gamma-distributed rate categories) successfully corrected the misplacement of the tree shrews, correctly positioning them as an outgroup to the primates. The persistence of an unresolved trifurcation among the lemurs, tarsiers, and monkeys/apes further illustrates an important general limitation of single-gene phylogenetic analysis: even a well-chosen evolutionary model applied to a single gene sequence may not always contain sufficient information to resolve every branching relationship with statistical confidence.