SNV accuracy "not there yet"

[kpalin]

Me and my student have done some testing for longcallD (v.0.0.4) during the summer and here are some of the things we've noticed.

While we're very happy to see indel calls from nanopore reads, we've seen quite clearly that both the precision and recall for SNV:s, compared to Illumina WGS, is lower for longcallD than for longshot (Using "longshot --max_cov 75 --max_snvs 1 --density_params 30:850:17 --ts_tv_ratio 0.15") This holds for all kinds of genome regions. Our data is mostly old ~20x coverage LSK109 so neither software gives good SNV concordance with Illumina.

Also, the phase sets seem to be much shorter with longcallD.

And the variant qualities are all fixed at 60. Should I consider the genotype qualities as variant quality?

[lh3]

Hi Kimmo, long time no see! LSK109 is compatible with both R9 and R10. Is this R9 or R10 data? LongcallD is optimized for HiFi and the latest LSK114+R10.4. We know it doesn't work well with R9.

[yangao07]

Hi Kimmo, thanks for using/testing longcallD.
In our in-house testing, we also noiced significant decrease of accuracy for low-coverage ONT R10 data, i.e., 15X/20X.

I have fixed the major issue causing that in longcallD-v0.0.4.
Right now, using HG002 GIAB T2T v1.1 benchmark set, the SNP calling of longcallD is comparable to other tools, while INDEL calling, especially homopolymer indel calling, is worse than deepvariant/clair3.
We believe that there is some room for longcallD to improve small variant calling, but we accept that deep-learning method is doing better than us.
The new version will be released soon.

Here is the recall/precision for HG002 ONT R10 downsamping data using T2T v1.1 benchmark set:

Tool	Type	Coverage	Recall	Precision	F1_Score	1-Recall	1-Precision
clair3	INDEL	15X	61.91%	64.42%	63.14%	38.09%	35.58%
clair3	SNP	15X	96.22%	95.30%	95.76%	3.78%	4.70%
deepvariant	INDEL	15X	58.03%	83.02%	68.31%	41.97%	16.98%
deepvariant	SNP	15X	95.71%	96.98%	96.34%	4.29%	3.02%
longcallD	INDEL	15X	60.59%	58.77%	59.67%	39.41%	41.23%
longcallD	SNP	15X	95.01%	96.61%	95.80%	4.99%	3.39%
longshot	SNP	15X	95.06%	96.68%	95.86%	4.94%	3.32%
clair3	INDEL	20X	64.28%	70.75%	67.36%	35.72%	29.25%
clair3	SNP	20X	96.58%	96.15%	96.36%	3.42%	3.85%
deepvariant	INDEL	20X	61.66%	83.60%	70.97%	38.34%	16.40%
deepvariant	SNP	20X	96.30%	97.11%	96.70%	3.70%	2.89%
longcallD	INDEL	20X	63.43%	62.63%	63.03%	36.57%	37.37%
longcallD	SNP	20X	96.15%	96.74%	96.44%	3.85%	3.26%
longshot	SNP	20X	95.06%	96.68%	95.86%	4.94%	3.32%
clair3	INDEL	35X	67.81%	78.49%	72.76%	32.19%	21.51%
clair3	SNP	35X	96.70%	96.93%	96.81%	3.30%	3.07%
deepvariant	INDEL	35X	67.48%	85.36%	75.37%	32.52%	14.64%
deepvariant	SNP	35X	96.50%	97.25%	96.87%	3.50%	2.75%
longcallD	INDEL	35X	67.04%	69.19%	68.10%	32.96%	30.81%
longcallD	SNP	35X	96.77%	96.91%	96.84%	3.23%	3.09%
longshot	SNP	35X	95.06%	96.68%	95.86%	4.94%	3.32%

[kpalin]

Thanks for the benchmarks @yangao07 That is fairly close to what we've seen. The 'global' difference between longcallD and longshot is only few percentage points. (See table below for comparison with Illumina calls with GATK/ https://github.com/nf-core/sarek). Samples 59443 and 30289 have median coverage 32x, while a5387 and c30a5 have 19x. The data is from 4khz R10.

Any hints how to force more sensitivity for low coverage samples? Or more agressive phasing (longer phase sets?)

(And just to be clear: I think longcallD the first realistic improvement/alternative to longshot for our data.)

Caller	sample_id	Filter	Precision	Recall
LongcallD	59443	ALL	0.963809	0.965054
LongcallD	30289	ALL	0.965545	0.963261
LongcallD	a5387	ALL	0.94523	0.876321
LongcallD	c30a5	ALL	0.945227	0.873715
Longshot	59443	ALL	0.95312	0.982047
Longshot	30289	ALL	0.957742	0.981653
Longshot	a5387	ALL	0.943735	0.973706
Longshot	c30a5	ALL	0.951893	0.971866
Longshot	59443	PASS	0.972655	0.975735
Longshot	30289	PASS	0.974557	0.976006
Longshot	a5387	PASS	0.973998	0.953676
Longshot	c30a5	PASS	0.976114	0.949952

[lh3]

I think longcallD the first realistic improvement/alternative to longshot for our data

You may try the latest deepvariant. Older deepvariant could only achieve high accuracy with the DV-whatshap-DV pipeline. Recent versions work in one pass.