Evaluating Alphafold predictions

AF gives quality scores for each prediction. A great FAQ can be found at https://alphafold.ebi.ac.uk/faq.

Another great resource is this YouTube video by EMBL-EBI:

Also, have a look at this lecture by John Jumper, the research lead of AlphaFold:

Here are just some examples from our HSV-1 predictions:

The first measure is a depiction of the Multiple Sequence Alignment (MSA) that is used as input for the network.

The MSA above from HSV-1 UL55 shows ok coverage of both similar and less similar sequences as well as good coverage for the C-terminus also with less similar sequences.

Compare the MSA of UL56. It is much less well populated and it does not incorporate many less similar sequences.

Now, let’s look at the resulting structure predictions:

In both cases, the predictions are colored by the pLDDT which is a confidence measure of how well AlphaFold "thinks" its prediction is.

Here is an excerpt from the EBI FAQ:
AlphaFold produces a per-residue estimate of its confidence on a scale from 0 – 100 . This confidence measure is called pLDDT and corresponds to the model’s predicted score on the lDDT-Cα metric. It is stored in the B-factor fields of the mmCIF and PDB files available for download (although unlike a B-factor, higher pLDDT is better). pLDDT is also used to colour-code the residues of the model in the 3D structure viewer. The following rules of thumb provide guidance on the expected reliability of a given region:

• Regions with pLDDT > 90 are expected to be modelled to high accuracy. These should be suitable for any application that benefits from high accuracy (e.g. characterising binding sites).
• Regions with pLDDT between 70 and 90 are expected to be modelled well (a generally good backbone prediction).
• Regions with pLDDT between 50 and 70 are low confidence and should be treated with caution.
• The 3D coordinates of regions with pLDDT < 50 often have a ribbon-like appearance and should not be interpreted. We show in our paper that pLDDT < 50 is a reasonably strong predictor of disorder, i.e. it suggests such a region is either unstructured in physiological conditions or only structured as part of a complex.
• Structured domains with many inter-residue contacts are likely to be more reliable than extended linkers or isolated long helices.
• Unphysical bond lengths and clashes do not usually appear in confident regions. Any part of a structure with several of these should be disregarded.

Note that the PDB and mmCIF files contain coordinates for all regions, regardless of their pLDDT score. It is up to the user to interpret the model judiciously, in accordance with the guidance above. The pLDDT per position is also given as a plot for the five models made in every run and gives a simpler overview:

Note the high overall scores for UL55 and low ones for UL56. The overall low scores for the UL56 prediction should make us cautious.

Finally, the Predicted Alignment Error (PAE) gives an estimate of the relative position of domains. Again an excerpt from the EBI FAQ:

Independent of the 3D structure, AlphaFold produces an output called “Predicted Aligned Error”. This is shown at the bottom of structure pages as an interactive 2D plot.

• The colour at (x, y) indicates AlphaFold’s expected position error at residue x if the predicted and true structures were aligned on residue y.
• If the predicted aligned error is generally low for residue pairs x, y from two different domains, it indicates that AlphaFold predicts well-defined relative positions for them.
• If the predicted aligned error is generally high for residue pairs x, y from two different domains, then the relative positions of these domains in the 3D structure is uncertain and should not be interpreted.

Let's look at the PAE Plots for both UL55 and UL56:

You can immediately see, that the position of most amino acids to each other is unclear in all predictions. The relative position of the predicted alpha-helices should be therefore taken with more than a grain of salt.