Compare commits

..
65 Commits
Author SHA1 Message Date
Heng Li b6ff332de1 Release minimap2-2.18 (r1015) 2021-04-09 13:33:34 -04:00
Heng Li 77abafaaf3 prepare for release 2021-04-09 13:18:56 -04:00
Heng Li 507d39af15 r1013: changed to a more accurate similarity est
Based on DOI:10.1101/2021.01.15.426881. One minimap2 reviewer suggested the
right formula to me but I thought the difference would be insignificant. I was
wrong.
2021-04-08 13:57:53 -04:00
Heng Li 827ca4b461 r1012: fixed an off-by-one bug; resolves #489 2021-04-07 23:31:31 -04:00
Marcus Stoiber d3dde2fdd4 Convert from spaces to tabs. 2021-04-05 11:55:10 -04:00
Marcus Stoiber 7db2e8d21a Convert python install from build_ext to setuptools setup_requires. 2021-04-05 11:55:10 -04:00
Heng Li 0b41dd26a2 r1009: fixed a compiler warning 2021-04-05 11:43:13 -04:00
Heng Li 2b47846cd6 r1008: don't parse space in BED 2021-04-05 11:41:00 -04:00
Heng Li 67dd906a80 bump travis python version to 3.9 2021-03-23 09:12:49 -04:00
Heng Li 1b0bb7b0ba require overlap ratio when considering centromere 2021-03-11 19:11:44 -05:00
Heng Li 1c4b7e8a48 explained --junc-bed in README 2021-03-06 19:44:24 -05:00
Heng Li ecbc399fa2 improved sveval 2021-03-06 19:44:13 -05:00
Heng Li 4dfd495cc2 added sveval 2021-02-15 14:49:36 -05:00
Heng Li 194b457e79 option to print errors only 2021-02-07 12:58:03 -05:00
Heng Li 75c8933511 evaluate large-scale misjoins 2021-02-07 12:34:13 -05:00
Heng Li 1025993469 print number of errors on each read 2021-01-29 14:08:21 -05:00
Heng Li a3253d1a6b added a command for simple VCF statistics 2021-01-14 12:24:38 -05:00
Heng Li 2da649d1d7 Merge remote-tracking branch 'origin/master' 2020-11-15 18:47:15 -05:00
Heng Li f995f55610 added --mask-len for #659 2020-08-21 11:12:50 -04:00
Armin Töpfer c9874e2dc5 Initialize r->p if ez->zdropped 2020-06-12 09:22:18 -04:00
Heng Li ccb0f7b05d added a new Makefile for simde 2020-04-25 22:43:29 -04:00
mbrcic 66db9da7d8 changed preprocessor conditionals for SIMDe 2020-04-22 19:50:59 +02:00
mbrcic 2b3403f094 fix for Neon after test. 2020-04-21 02:08:21 +02:00
mbrcic 3e16e4e39d Added documentation entry for added functionality, simde and no_simd. 2020-04-21 01:36:19 +02:00
mbrcic f47e8a525e SIMDe made optional. Include paths changes for SIMDe. 2020-04-21 01:08:32 +02:00
mbrcic c172df7d2d fix for Neon 2020-04-20 21:14:15 +02:00
mbrcic 9e6fdd376b Changed sse2neon with SIMDe. Added building non-SIMD version. 2020-04-20 18:28:06 +02:00
Heng Li 29f67a1666 r982: more accurate sum; output errors 2020-04-14 16:18:53 -04:00
Heng Li adde608a42 Merge remote-tracking branch 'refs/remotes/origin/master' 2020-04-14 15:52:57 -04:00
Heng Li f10dff78dc r981: asmgene to check duplicate genes 2020-04-14 15:52:36 -04:00
Jun Aruga d97bba9f27 travis: added arm64 test. 2020-04-13 08:33:03 -04:00
Heng Li 50775362bb r980: support auNGA 2020-04-10 21:36:59 -04:00
Heng Li 0a5e386359 r979: fixed asmgene wrong report. Resolves #581. 2020-04-06 19:57:15 -04:00
Heng Li cb56fb762a Merge branch 'master' of github.com:lh3/minimap2 2020-03-22 19:17:01 -04:00
Heng Li e2451e497a r975: asmstat without CIGAR/NM 2020-03-22 19:16:43 -04:00
Jared Simpson d2de282d21 remove second definition of kstring 2020-03-02 13:18:37 -05:00
Jared Simpson 48cb80ea94 change kstring_t integer storage size
This is for compatibility with kstring_t in htslib.
2020-02-28 09:35:51 -05:00
Heng Li 6a4b9f9082 r974: more informative msg on wrong FASTQ records
Resolves #510
2020-01-21 10:56:59 -05:00
Heng Li a7a01fe5bd r973: fixed compiling errors caused 2020-01-21 10:43:31 -05:00
Heng Li 9dceae59a0 r972: renamed --alt-diff to --alt-drop 2020-01-21 10:33:39 -05:00
Heng Li 20a3987082 Merge branch 'master' into alt 2020-01-21 09:17:50 -05:00
Heng Li eb3ed6993d support ALT mapping 2020-01-21 09:17:50 -05:00
Heng Li 7996f04008 r972: fixed negative de:f caused by ambiguous base 2020-01-21 09:14:37 -05:00
Heng Li d2e14705e7 r968: allow large mini_batch; resolves #491 2020-01-18 12:24:44 -05:00
Heng Li 24f50f38e8 r967: no duplicated @SQ lines with --split-prefix
resolves #527 and #400
2020-01-18 12:01:28 -05:00
Heng Li 04e015d803 r966: minimap2 returns 1 on file failure (#532) 2020-01-18 10:58:59 -05:00
Heng Li 040f74102c r965: added --chain-gap-scale for #540 2020-01-18 10:29:33 -05:00
Heng Li cdb7857841 r963: --junc-bonus not working; resolves #513 2020-01-06 22:03:50 -05:00
Heng Li 3c0d05d272 r962: abort given wrong RG line; resolves #541 2020-01-06 21:53:21 -05:00
Heng Li 47b646acbf r961: print indexed length 2020-01-06 21:13:33 -05:00
Heng Li a79cb3e991 Merge remote-tracking branch 'origin/master' 2019-12-23 17:33:56 -05:00
Heng Li 367aed4271 added the asan and tsan targets to Makefile 2019-12-23 17:33:10 -05:00
xdudiagnoa 081df6ac7d Fix example.c seq read logic
for every idx should map all input seqs
2019-11-11 00:46:07 -05:00
Torsten Seemann a3e7a575fb Add splice:hq to --help 2019-11-11 00:45:13 -05:00
Heng Li d90583b83c r954: fixed two potential undef behaviors (#443) 2019-07-18 09:17:08 -04:00
Heng Li 7fc03b0c32 r953: krealloc is buggy
Its use in minimap2 didn't trigger the bug, so the older minimap2 is still ok.
2019-07-18 09:13:30 -04:00
John Marshall 20c104ce8d Report errno on file opening failures and I/O errors
Add the underlying operating system error (usually "No such file" or
"Out of space" respectively, but highly informative when it is not)
to these error messages.
2019-07-17 09:04:02 -04:00
Marcus Stoiber 238b6bb3ea Fix memory leak in mappy.aligner.map. 2019-07-08 09:50:54 -04:00
Heng Li e026e18439 added the description of "SA" tag. Closes #438 2019-07-01 09:18:33 -04:00
Heng Li 58c2251b18 compatibility with GenBank GTP (resolves $422) 2019-06-11 09:16:03 -04:00
Heng Li 03dc8d5d97 test if index is built for #413 2019-06-07 09:11:11 -04:00
Heng Li 5cb61f8ee6 added FAQ 2019-06-06 10:47:33 -04:00
Heng Li c16a1742a3 Er... Tavis doesn't have python 3.7. 2019-05-11 20:06:48 -04:00
Heng Li 4bd5a018c2 test python 3.7 instead of 3.6 2019-05-11 20:05:06 -04:00
Heng Li 05974c80f1 r943: allow long ref name for --split-index
Resolved #394.
2019-05-10 15:39:41 -04:00
37 changed files with 990 additions and 188 deletions
+3
View File
@@ -0,0 +1,3 @@
[submodule "lib/simde"]
path = lib/simde
url = https://github.com/nemequ/simde.git
+5 -1
View File
@@ -6,6 +6,10 @@ matrix:
- language: c
compiler: clang
script: make
- arch: arm64
language: c
compiler: gcc
script: make arm_neon=1 aarch64=1
- language: python
python: "2.7"
before_install: pip install cython
@@ -15,6 +19,6 @@ matrix:
before_install: pip install cython
script: python setup.py build_ext
- language: python
python: "3.6"
python: "3.9"
before_install: pip install cython
script: python setup.py build_ext
+46
View File
@@ -0,0 +1,46 @@
#### 1. Alignment different with option `-a` or `-c`?
Without `-a`, `-c` or `--cs`, minimap2 only finds *approximate* mapping
locations without detailed base alignment. In particular, the start and end
positions of the alignment are impricise. With one of those options, minimap2
will perform base alignment, which is generally more accurate but is much
slower.
#### 2. How to map Illumina short reads to noisy long reads?
No good solutions. The better approach is to assemble short reads into contigs
and then map noisy reads to contigs.
#### 3. The output SAM doesn't have a header.
By default, minimap2 indexes 4 billion reference bases (4Gb) in a batch and map
all reads against each reference batch. Given a reference longer than 4Gb,
minimap2 is unable to see all the sequences and thus can't produce a correct
SAM header. In this case, minimap2 doesn't output any SAM header. There are two
solutions to this issue. First, you may increase option `-I` to, for example,
`-I8g` to index more reference bases in a batch. This is preferred if your
machine has enough memory. Second, if your machines doesn't have enough memory
to hold the reference index, you can use the `--split-prefix` option in a
command line like:
```sh
minimap2 -ax map-ont --split-prefix=tmp ref.fa reads.fq
```
This second approach uses less memory, but it is slower and requires temporary
disk space.
#### 4. The output SAM is malformatted.
This typically happens when you use nohup to wrap a minimap2 command line.
Nohup is discouraged as it breaks piping. If you have to use nohup, please
specify an output file with option `-o`.
#### 5. How to output one alignment per read?
You can use `--secondary=no` to suppress secondary alignments (aka multiple
mappings), but you can't suppress supplementary alignment (aka split or
chimeric alignment) this way. You can use samtools to filter out these
alignments:
```sh
minimap2 -ax map-out ref.fa reads.fq | samtools view -F0x900
```
However, this is discouraged as supplementary alignment is informative.
+10
View File
@@ -22,6 +22,16 @@ else #if aarch64 is defined
endif
endif
ifneq ($(asan),)
CFLAGS+=-fsanitize=address
LIBS+=-fsanitize=address
endif
ifneq ($(tsan),)
CFLAGS+=-fsanitize=thread
LIBS+=-fsanitize=thread
endif
.PHONY:all extra clean depend
.SUFFIXES:.c .o
+97
View File
@@ -0,0 +1,97 @@
CFLAGS= -g -Wall -O2 -Wc++-compat #-Wextra
CPPFLAGS= -DHAVE_KALLOC -DUSE_SIMDE -DSIMDE_ENABLE_NATIVE_ALIASES
INCLUDES= -Ilib/simde
OBJS= kthread.o kalloc.o misc.o bseq.o sketch.o sdust.o options.o index.o chain.o align.o hit.o map.o format.o pe.o esterr.o splitidx.o \
ksw2_extz2_simde.o ksw2_extd2_simde.o ksw2_exts2_simde.o ksw2_ll_simde.o
PROG= minimap2
PROG_EXTRA= sdust minimap2-lite
LIBS= -lm -lz -lpthread
ifneq ($(arm_neon),) # if arm_neon is defined
ifeq ($(aarch64),) #if aarch64 is not defined
CFLAGS+=-D_FILE_OFFSET_BITS=64 -mfpu=neon -fsigned-char
else #if aarch64 is defined
CFLAGS+=-D_FILE_OFFSET_BITS=64 -fsigned-char
endif
endif
ifneq ($(asan),)
CFLAGS+=-fsanitize=address
LIBS+=-fsanitize=address
endif
ifneq ($(tsan),)
CFLAGS+=-fsanitize=thread
LIBS+=-fsanitize=thread
endif
.PHONY:all extra clean depend
.SUFFIXES:.c .o
.c.o:
$(CC) -c $(CFLAGS) $(CPPFLAGS) $(INCLUDES) $< -o $@
all:$(PROG)
extra:all $(PROG_EXTRA)
minimap2:main.o libminimap2.a
$(CC) $(CFLAGS) main.o -o $@ -L. -lminimap2 $(LIBS)
minimap2-lite:example.o libminimap2.a
$(CC) $(CFLAGS) $< -o $@ -L. -lminimap2 $(LIBS)
libminimap2.a:$(OBJS)
$(AR) -csru $@ $(OBJS)
sdust:sdust.c kalloc.o kalloc.h kdq.h kvec.h kseq.h ketopt.h sdust.h
$(CC) -D_SDUST_MAIN $(CFLAGS) $< kalloc.o -o $@ -lz
ksw2_ll_simde.o:ksw2_ll_sse.c ksw2.h kalloc.h
$(CC) -c $(CFLAGS) -msse2 $(CPPFLAGS) $(INCLUDES) $< -o $@
ksw2_extz2_simde.o:ksw2_extz2_sse.c ksw2.h kalloc.h
$(CC) -c $(CFLAGS) -msse4.1 $(CPPFLAGS) $(INCLUDES) $< -o $@
ksw2_extd2_simde.o:ksw2_extd2_sse.c ksw2.h kalloc.h
$(CC) -c $(CFLAGS) -msse4.1 $(CPPFLAGS) $(INCLUDES) $< -o $@
ksw2_exts2_simde.o:ksw2_exts2_sse.c ksw2.h kalloc.h
$(CC) -c $(CFLAGS) -msse4.1 $(CPPFLAGS) $(INCLUDES) $< -o $@
# other non-file targets
clean:
rm -fr gmon.out *.o a.out $(PROG) $(PROG_EXTRA) *~ *.a *.dSYM build dist mappy*.so mappy.c python/mappy.c mappy.egg*
depend:
(LC_ALL=C; export LC_ALL; makedepend -Y -- $(CFLAGS) $(CPPFLAGS) -- *.c)
# DO NOT DELETE
align.o: minimap.h mmpriv.h bseq.h kseq.h ksw2.h kalloc.h
bseq.o: bseq.h kvec.h kalloc.h kseq.h
chain.o: minimap.h mmpriv.h bseq.h kseq.h kalloc.h
esterr.o: mmpriv.h minimap.h bseq.h kseq.h
example.o: minimap.h kseq.h
format.o: kalloc.h mmpriv.h minimap.h bseq.h kseq.h
hit.o: mmpriv.h minimap.h bseq.h kseq.h kalloc.h khash.h
index.o: kthread.h bseq.h minimap.h mmpriv.h kseq.h kvec.h kalloc.h khash.h
index.o: ksort.h
kalloc.o: kalloc.h
ksw2_extd2_sse.o: ksw2.h kalloc.h
ksw2_exts2_sse.o: ksw2.h kalloc.h
ksw2_extz2_sse.o: ksw2.h kalloc.h
ksw2_ll_sse.o: ksw2.h kalloc.h
kthread.o: kthread.h
main.o: bseq.h minimap.h mmpriv.h kseq.h ketopt.h
map.o: kthread.h kvec.h kalloc.h sdust.h mmpriv.h minimap.h bseq.h kseq.h
map.o: khash.h ksort.h
misc.o: mmpriv.h minimap.h bseq.h kseq.h ksort.h
options.o: mmpriv.h minimap.h bseq.h kseq.h
pe.o: mmpriv.h minimap.h bseq.h kseq.h kvec.h kalloc.h ksort.h
sdust.o: kalloc.h kdq.h kvec.h sdust.h
self-chain.o: minimap.h kseq.h
sketch.o: kvec.h kalloc.h mmpriv.h minimap.h bseq.h kseq.h
splitidx.o: mmpriv.h minimap.h bseq.h kseq.h
+86
View File
@@ -1,3 +1,89 @@
Release 2.18-r1015 (9 April 2021)
---------------------------------
This release fixes multiple rare bugs in minimap2 and adds additional
functionality to paftools.js.
Changes to minimap2:
* Bugfix: a rare segfault caused by an off-by-one error (#489)
* Bugfix: minimap2 segfaulted due to an uninitilized variable (#622 and #625).
* Bugfix: minimap2 parsed spaces as field separators in BED (#721). This led
to issues when the BED name column contains spaces.
* Bugfix: minimap2 `--split-prefix` did not work with long reference names
(#394).
* Bugfix: option `--junc-bonus` didn't work (#513)
* Bugfix: minimap2 didn't return 1 on I/O errors (#532)
* Bugfix: the `de:f` tag (sequence divergence) could be negative if there were
ambiguous bases
* Bugfix: fixed two undefined behaviors caused by calling memcpy() on
zero-length blocks (#443)
* Bugfix: there were duplicated SAM @SQ lines if option `--split-prefix` is in
use (#400 and #527)
* Bugfix: option -K had to be smaller than 2 billion (#491). This was caused
by a 32-bit integer overflow.
* Improvement: optionally compile against SIMDe (#597). Minimap2 should work
with IBM POWER CPUs, though this has not been tested. To compile with SIMDe,
please use `make -f Makefile.simde`.
* Improvement: more informative error message for I/O errors (#454) and for
FASTQ parsing errors (#510)
* Improvement: abort given malformatted RG line (#541)
* Improvement: better formula to estimate the `dv:f` tag (approximate sequence
divergence). See DOI:10.1101/2021.01.15.426881.
* New feature: added the `--mask-len` option to fine control the removal of
redundant hits (#659). The default behavior is unchanged.
Changes to mappy:
* Bugfix: mappy caused segmentation fault if the reference index is not
present (#413).
* Bugfix: fixed a memory leak via 238b6bb3
* Change: always require Cython to compile the mappy module (#723). Older
mappy packages at PyPI bundled the C source code generated by Cython such
that end users did not need to install Cython to compile mappy. However, as
Python 3.9 is breaking backward compatibility, older mappy does not work
with Python 3.9 anymore. We have to add this Cython dependency as a
workaround.
Changes to paftools.js:
* Bugfix: the "part10-" line from asmgene was wrong (#581)
* Improvement: compatibility with GTF files from GenBank (#422)
* New feature: asmgene also checks missing multi-copy genes
* New feature: added the misjoin command to evaluate large-scale misjoins and
megabase-long inversions.
Although given the many bug fixes and minor improvements, the core algorithm
stays the same. This version of minimap2 produces nearly identical alignments
to v2.17 except very rare corner cases.
Now unimap is recommended over minimap2 for aligning long contigs against a
reference genome. It often takes less wall-clock time and is much more
sensitive to long insertions and deletions.
(2.18: 9 April 2021, r1015)
Release 2.17-r941 (4 May 2019)
------------------------------
+34 -6
View File
@@ -19,12 +19,17 @@ cd minimap2 && make
./minimap2 -ax splice ref.fa rna-reads.fa > aln.sam # spliced long reads (strand unknown)
./minimap2 -ax splice -uf -k14 ref.fa reads.fa > aln.sam # noisy Nanopore Direct RNA-seq
./minimap2 -ax splice:hq -uf ref.fa query.fa > aln.sam # Final PacBio Iso-seq or traditional cDNA
./minimap2 -ax splice --junc-bed anno.bed12 ref.fa query.fa > aln.sam # prioritize on annotated junctions
./minimap2 -cx asm5 asm1.fa asm2.fa > aln.paf # intra-species asm-to-asm alignment
./minimap2 -x ava-pb reads.fa reads.fa > overlaps.paf # PacBio read overlap
./minimap2 -x ava-ont reads.fa reads.fa > overlaps.paf # Nanopore read overlap
# man page for detailed command line options
man ./minimap2.1
```
[Unimap][unimap] is recommended for aligning long contigs against a reference
genome. It often takes less wall-clock time and is much more sensitive to long
insertions and deletions.
## Table of Contents
- [Getting Started](#started)
@@ -71,8 +76,8 @@ Detailed evaluations are available from the [minimap2 paper][doi] or the
Minimap2 is optimized for x86-64 CPUs. You can acquire precompiled binaries from
the [release page][release] with:
```sh
curl -L https://github.com/lh3/minimap2/releases/download/v2.17/minimap2-2.17_x64-linux.tar.bz2 | tar -jxvf -
./minimap2-2.17_x64-linux/minimap2
curl -L https://github.com/lh3/minimap2/releases/download/v2.18/minimap2-2.18_x64-linux.tar.bz2 | tar -jxvf -
./minimap2-2.18_x64-linux/minimap2
```
If you want to compile from the source, you need to have a C compiler, GNU make
and zlib development files installed. Then type `make` in the source code
@@ -80,7 +85,14 @@ directory to compile. If you see compilation errors, try `make sse2only=1`
to disable SSE4 code, which will make minimap2 slightly slower.
Minimap2 also works with ARM CPUs supporting the NEON instruction sets. To
compile for 32 bit ARM architectures (such as ARMv7), use `make arm_neon=1`. To compile for for 64 bit ARM architectures (such as ARMv8), use `make arm_neon=1 aarch64=1`.
compile for 32 bit ARM architectures (such as ARMv7), use `make arm_neon=1`. To
compile for for 64 bit ARM architectures (such as ARMv8), use `make arm_neon=1
aarch64=1`.
Minimap2 can use [SIMD Everywhere (SIMDe)][simde] library for porting
implementation to the different SIMD instruction sets. To compile using SIMDe,
use `make -f Makefile.simde`. To compile for ARM CPUs, use `Makefile.simde`
with the ARM related command lines given above.
### <a name="general"></a>General usage
@@ -178,6 +190,19 @@ This is because SIRV does not honor the evolutionarily conservative splicing
signal. If you are studying SIRV, you may apply `--splice-flank=no` to let
minimap2 only model GT..AG, ignoring the additional base.
Since v2.17, minimap2 can optionally take annotated genes as input and
prioritize on annotated splice junctions. To use this feature, you can
```sh
paftools.js gff2bed anno.gff > anno.bed
minimap2 -ax splice --junc-bed anno.bed ref.fa query.fa > aln.sam
```
Here, `anno.gff` is the gene annotation in the GTF or GFF3 format (`gff2bed`
automatically tests the format). The output of `gff2bed` is in the 12-column
BED format, or the BED12 format. With the `--junc-bed` option, minimap2 adds a
bonus score (tuned by `--junc-bonus`) if an aligned junction matches a junction
in the annotation. Option `--junc-bed` also takes 5-column BED, including the
strand field. In this case, each line indicates an oriented junction.
#### <a name="long-overlap"></a>Find overlaps between long reads
```sh
@@ -315,9 +340,10 @@ highlighted in bold. The description may help to tune minimap2 parameters.
### <a name="help"></a>Getting help
Manpage [minimap2.1][manpage] provides detailed description of minimap2
command line options and optional tags. If you encounter bugs or have further
questions or requests, you can raise an issue at the [issue page][issue].
There is not a specific mailing list for the time being.
command line options and optional tags. The [FAQ](FAQ.md) page answers several
frequently asked questions. If you encounter bugs or have further questions or
requests, you can raise an issue at the [issue page][issue]. There is not a
specific mailing list for the time being.
### <a name="cite"></a>Citing minimap2
@@ -375,3 +401,5 @@ mappy` or [from BioConda][mappyconda] via `conda install -c bioconda mappy`.
[manpage]: https://lh3.github.io/minimap2/minimap2.html
[manpage-cs]: https://lh3.github.io/minimap2/minimap2.html#10
[doi]: https://doi.org/10.1093/bioinformatics/bty191
[smide]: https://github.com/nemequ/simde
[unimap]: https://github.com/lh3/unimap
+10 -3
View File
@@ -38,8 +38,8 @@ static inline void update_max_zdrop(int32_t score, int i, int j, int32_t *max, i
int z = *max - score - diff * e;
if (z > *max_zdrop) {
*max_zdrop = z;
pos[0][0] = *max_i, pos[0][1] = i + 1;
pos[1][0] = *max_j, pos[1][1] = j + 1;
pos[0][0] = *max_i, pos[0][1] = i;
pos[1][0] = *max_j, pos[1][1] = j;
}
} else *max = score, *max_i = i, *max_j = j;
}
@@ -739,6 +739,13 @@ static void mm_align1(void *km, const mm_mapopt_t *opt, const mm_idx_t *mi, int
if (ez->n_cigar > 0)
mm_append_cigar(r, ez->n_cigar, ez->cigar);
if (ez->zdropped) { // truncated by Z-drop; TODO: sometimes Z-drop kicks in because the next seed placement is wrong. This can be fixed in principle.
if (!r->p) {
assert(ez->n_cigar == 0);
uint32_t capacity = sizeof(mm_extra_t)/4;
kroundup32(capacity);
r->p = (mm_extra_t*)calloc(capacity, 4);
r->p->capacity = capacity;
}
for (j = i - 1; j >= 0; --j)
if ((int32_t)a[as1 + j].x <= rs + ez->max_t)
break;
@@ -908,6 +915,6 @@ mm_reg1_t *mm_align_skeleton(void *km, const mm_mapopt_t *opt, const mm_idx_t *m
kfree(km, qseq0[0]);
kfree(km, ez.cigar);
mm_filter_regs(opt, qlen, n_regs_, regs);
mm_hit_sort(km, n_regs_, regs);
mm_hit_sort(km, n_regs_, regs, opt->alt_drop);
return regs;
}
+10 -8
View File
@@ -77,7 +77,7 @@ static inline void kseq2bseq(kseq_t *ks, mm_bseq1_t *s, int with_qual, int with_
s->l_seq = ks->seq.l;
}
mm_bseq1_t *mm_bseq_read3(mm_bseq_file_t *fp, int chunk_size, int with_qual, int with_comment, int frag_mode, int *n_)
mm_bseq1_t *mm_bseq_read3(mm_bseq_file_t *fp, int64_t chunk_size, int with_qual, int with_comment, int frag_mode, int *n_)
{
int64_t size = 0;
int ret;
@@ -99,7 +99,7 @@ mm_bseq1_t *mm_bseq_read3(mm_bseq_file_t *fp, int chunk_size, int with_qual, int
size += s->l_seq;
if (size >= chunk_size) {
if (frag_mode && a.a[a.n-1].l_seq < CHECK_PAIR_THRES) {
while (kseq_read(ks) >= 0) {
while ((ret = kseq_read(ks)) >= 0) {
kseq2bseq(ks, &fp->s, with_qual, with_comment);
if (mm_qname_same(fp->s.name, a.a[a.n-1].name)) {
kv_push(mm_bseq1_t, 0, a, fp->s);
@@ -110,23 +110,25 @@ mm_bseq1_t *mm_bseq_read3(mm_bseq_file_t *fp, int chunk_size, int with_qual, int
break;
}
}
if (ret < -1)
fprintf(stderr, "[WARNING]\033[1;31m wrong FASTA/FASTQ record. Continue anyway.\033[0m\n");
if (ret < -1) {
if (a.n) fprintf(stderr, "[WARNING]\033[1;31m failed to parse the FASTA/FASTQ record next to '%s'. Continue anyway.\033[0m\n", a.a[a.n-1].name);
else fprintf(stderr, "[WARNING]\033[1;31m failed to parse the first FASTA/FASTQ record. Continue anyway.\033[0m\n");
}
*n_ = a.n;
return a.a;
}
mm_bseq1_t *mm_bseq_read2(mm_bseq_file_t *fp, int chunk_size, int with_qual, int frag_mode, int *n_)
mm_bseq1_t *mm_bseq_read2(mm_bseq_file_t *fp, int64_t chunk_size, int with_qual, int frag_mode, int *n_)
{
return mm_bseq_read3(fp, chunk_size, with_qual, 0, frag_mode, n_);
}
mm_bseq1_t *mm_bseq_read(mm_bseq_file_t *fp, int chunk_size, int with_qual, int *n_)
mm_bseq1_t *mm_bseq_read(mm_bseq_file_t *fp, int64_t chunk_size, int with_qual, int *n_)
{
return mm_bseq_read2(fp, chunk_size, with_qual, 0, n_);
}
mm_bseq1_t *mm_bseq_read_frag2(int n_fp, mm_bseq_file_t **fp, int chunk_size, int with_qual, int with_comment, int *n_)
mm_bseq1_t *mm_bseq_read_frag2(int n_fp, mm_bseq_file_t **fp, int64_t chunk_size, int with_qual, int with_comment, int *n_)
{
int i;
int64_t size = 0;
@@ -156,7 +158,7 @@ mm_bseq1_t *mm_bseq_read_frag2(int n_fp, mm_bseq_file_t **fp, int chunk_size, in
return a.a;
}
mm_bseq1_t *mm_bseq_read_frag(int n_fp, mm_bseq_file_t **fp, int chunk_size, int with_qual, int *n_)
mm_bseq1_t *mm_bseq_read_frag(int n_fp, mm_bseq_file_t **fp, int64_t chunk_size, int with_qual, int *n_)
{
return mm_bseq_read_frag2(n_fp, fp, chunk_size, with_qual, 0, n_);
}
+5 -5
View File
@@ -18,11 +18,11 @@ typedef struct {
mm_bseq_file_t *mm_bseq_open(const char *fn);
void mm_bseq_close(mm_bseq_file_t *fp);
mm_bseq1_t *mm_bseq_read3(mm_bseq_file_t *fp, int chunk_size, int with_qual, int with_comment, int frag_mode, int *n_);
mm_bseq1_t *mm_bseq_read2(mm_bseq_file_t *fp, int chunk_size, int with_qual, int frag_mode, int *n_);
mm_bseq1_t *mm_bseq_read(mm_bseq_file_t *fp, int chunk_size, int with_qual, int *n_);
mm_bseq1_t *mm_bseq_read_frag2(int n_fp, mm_bseq_file_t **fp, int chunk_size, int with_qual, int with_comment, int *n_);
mm_bseq1_t *mm_bseq_read_frag(int n_fp, mm_bseq_file_t **fp, int chunk_size, int with_qual, int *n_);
mm_bseq1_t *mm_bseq_read3(mm_bseq_file_t *fp, int64_t chunk_size, int with_qual, int with_comment, int frag_mode, int *n_);
mm_bseq1_t *mm_bseq_read2(mm_bseq_file_t *fp, int64_t chunk_size, int with_qual, int frag_mode, int *n_);
mm_bseq1_t *mm_bseq_read(mm_bseq_file_t *fp, int64_t chunk_size, int with_qual, int *n_);
mm_bseq1_t *mm_bseq_read_frag2(int n_fp, mm_bseq_file_t **fp, int64_t chunk_size, int with_qual, int with_comment, int *n_);
mm_bseq1_t *mm_bseq_read_frag(int n_fp, mm_bseq_file_t **fp, int64_t chunk_size, int with_qual, int *n_);
int mm_bseq_eof(mm_bseq_file_t *fp);
extern unsigned char seq_nt4_table[256];
+9 -7
View File
@@ -19,7 +19,7 @@ static inline int ilog2_32(uint32_t v)
return (t = v>>8) ? 8 + LogTable256[t] : LogTable256[v];
}
mm128_t *mm_chain_dp(int max_dist_x, int max_dist_y, int bw, int max_skip, int max_iter, int min_cnt, int min_sc, int is_cdna, int n_segs, int64_t n, mm128_t *a, int *n_u_, uint64_t **_u, void *km)
mm128_t *mm_chain_dp(int max_dist_x, int max_dist_y, int bw, int max_skip, int max_iter, int min_cnt, int min_sc, float gap_scale, int is_cdna, int n_segs, int64_t n, mm128_t *a, int *n_u_, uint64_t **_u, void *km)
{ // TODO: make sure this works when n has more than 32 bits
int32_t k, *f, *p, *t, *v, n_u, n_v;
int64_t i, j, st = 0;
@@ -52,7 +52,7 @@ mm128_t *mm_chain_dp(int max_dist_x, int max_dist_y, int bw, int max_skip, int m
if (i - st > max_iter) st = i - max_iter;
for (j = i - 1; j >= st; --j) {
int64_t dr = ri - a[j].x;
int32_t dq = qi - (int32_t)a[j].y, dd, sc, log_dd;
int32_t dq = qi - (int32_t)a[j].y, dd, sc, log_dd, gap_cost;
int32_t sidj = (a[j].y & MM_SEED_SEG_MASK) >> MM_SEED_SEG_SHIFT;
if ((sidi == sidj && dr == 0) || dq <= 0) continue; // don't skip if an anchor is used by multiple segments; see below
if ((sidi == sidj && dq > max_dist_y) || dq > max_dist_x) continue;
@@ -62,14 +62,16 @@ mm128_t *mm_chain_dp(int max_dist_x, int max_dist_y, int bw, int max_skip, int m
min_d = dq < dr? dq : dr;
sc = min_d > q_span? q_span : dq < dr? dq : dr;
log_dd = dd? ilog2_32(dd) : 0;
gap_cost = 0;
if (is_cdna || sidi != sidj) {
int c_log, c_lin;
c_lin = (int)(dd * .01 * avg_qspan);
c_log = log_dd;
if (sidi != sidj && dr == 0) ++sc; // possibly due to overlapping paired ends; give a minor bonus
else if (dr > dq || sidi != sidj) sc -= c_lin < c_log? c_lin : c_log;
else sc -= c_lin + (c_log>>1);
} else sc -= (int)(dd * .01 * avg_qspan) + (log_dd>>1);
else if (dr > dq || sidi != sidj) gap_cost = c_lin < c_log? c_lin : c_log;
else gap_cost = c_lin + (c_log>>1);
} else gap_cost = (int)(dd * .01 * avg_qspan) + (log_dd>>1);
sc -= (int)((double)gap_cost * gap_scale + .499);
sc += f[j];
if (sc > max_f) {
max_f = sc, max_j = j;
@@ -155,8 +157,8 @@ mm128_t *mm_chain_dp(int max_dist_x, int max_dist_y, int bw, int max_skip, int m
memcpy(&a[k], &b[w[i].y>>32], n * sizeof(mm128_t));
k += n;
}
memcpy(u, u2, n_u * 8);
memcpy(b, a, k * sizeof(mm128_t)); // write _a_ to _b_ and deallocate _a_ because _a_ is oversized, sometimes a lot
if (n_u) memcpy(u, u2, n_u * 8);
if (k) memcpy(b, a, k * sizeof(mm128_t)); // write _a_ to _b_ and deallocate _a_ because _a_ is oversized, sometimes a lot
kfree(km, a); kfree(km, w); kfree(km, u2);
return b;
}
+2 -2
View File
@@ -31,8 +31,8 @@ To acquire the data used in this cookbook and to install minimap2 and paftools,
please follow the command lines below:
```sh
# install minimap2 executables
curl -L https://github.com/lh3/minimap2/releases/download/v2.17/minimap2-2.17_x64-linux.tar.bz2 | tar jxf -
cp minimap2-2.17_x64-linux/{minimap2,k8,paftools.js} . # copy executables
curl -L https://github.com/lh3/minimap2/releases/download/v2.18/minimap2-2.18_x64-linux.tar.bz2 | tar jxf -
cp minimap2-2.18_x64-linux/{minimap2,k8,paftools.js} . # copy executables
export PATH="$PATH:"`pwd` # put the current directory on PATH
# download example datasets
curl -L https://github.com/lh3/minimap2/releases/download/v2.10/cookbook-data.tgz | tar zxf -
+1 -1
View File
@@ -59,6 +59,6 @@ void mm_est_err(const mm_idx_t *mi, int qlen, int n_regs, mm_reg1_t *regs, const
n_tot = en - st + 1;
if (r->qs > avg_k && r->rs > avg_k) ++n_tot;
if (qlen - r->qs > avg_k && l_ref - r->re > avg_k) ++n_tot;
r->div = logf((float)n_tot / n_match) / avg_k;
r->div = n_match >= n_tot? 0.0f : (float)(1.0 - pow((double)n_match / n_tot, 1.0 / avg_k));
}
}
+2
View File
@@ -35,6 +35,8 @@ int main(int argc, char *argv[])
while ((mi = mm_idx_reader_read(r, n_threads)) != 0) { // traverse each part of the index
mm_mapopt_update(&mopt, mi); // this sets the maximum minimizer occurrence; TODO: set a better default in mm_mapopt_init()!
mm_tbuf_t *tbuf = mm_tbuf_init(); // thread buffer; for multi-threading, allocate one tbuf for each thread
gzrewind(f);
kseq_rewind(ks);
while (kseq_read(ks) >= 0) { // each kseq_read() call reads one query sequence
mm_reg1_t *reg;
int j, i, n_reg;
+11 -7
View File
@@ -79,11 +79,11 @@ static char *mm_escape(char *s)
return s;
}
static void sam_write_rg_line(kstring_t *str, const char *s)
static int sam_write_rg_line(kstring_t *str, const char *s)
{
char *p, *q, *r, *rg_line = 0;
memset(mm_rg_id, 0, 256);
if (s == 0) return;
if (s == 0) return 0;
if (strstr(s, "@RG") != s) {
if (mm_verbose >= 1) fprintf(stderr, "[ERROR] the read group line is not started with @RG\n");
goto err_set_rg;
@@ -108,20 +108,23 @@ static void sam_write_rg_line(kstring_t *str, const char *s)
for (q = p, r = mm_rg_id; *q && *q != '\t' && *q != '\n'; ++q)
*r++ = *q;
mm_sprintf_lite(str, "%s\n", rg_line);
return 0;
err_set_rg:
free(rg_line);
return -1;
}
void mm_write_sam_hdr(const mm_idx_t *idx, const char *rg, const char *ver, int argc, char *argv[])
int mm_write_sam_hdr(const mm_idx_t *idx, const char *rg, const char *ver, int argc, char *argv[])
{
kstring_t str = {0,0,0};
int ret = 0;
if (idx) {
uint32_t i;
for (i = 0; i < idx->n_seq; ++i)
mm_sprintf_lite(&str, "@SQ\tSN:%s\tLN:%d\n", idx->seq[i].name, idx->seq[i].len);
}
if (rg) sam_write_rg_line(&str, rg);
if (rg) ret = sam_write_rg_line(&str, rg);
mm_sprintf_lite(&str, "@PG\tID:minimap2\tPN:minimap2");
if (ver) mm_sprintf_lite(&str, "\tVN:%s", ver);
if (argc > 1) {
@@ -132,6 +135,7 @@ void mm_write_sam_hdr(const mm_idx_t *idx, const char *rg, const char *ver, int
}
mm_err_puts(str.s);
free(str.s);
return ret;
}
static void write_cs_core(kstring_t *s, const uint8_t *tseq, const uint8_t *qseq, const mm_reg1_t *r, char *tmp, int no_iden, int write_tag)
@@ -270,7 +274,7 @@ double mm_event_identity(const mm_reg1_t *r)
if (op == 1 || op == 2)
++n_gapo, n_gap += len;
}
return (double)r->mlen / (r->blen - n_gap + n_gapo);
return (double)r->mlen / (r->blen + r->p->n_ambi - n_gap + n_gapo);
}
static inline void write_tags(kstring_t *s, const mm_reg1_t *r)
@@ -388,7 +392,7 @@ void mm_write_sam3(kstring_t *s, const mm_idx_t *mi, const mm_bseq1_t *t, int se
{
const int max_bam_cigar_op = 65535;
int flag, n_regs = n_regss[seg_idx], cigar_in_tag = 0;
int this_rid = -1, this_pos = -1, this_rev = 0;
int this_rid = -1, this_pos = -1;
const mm_reg1_t *regs = regss[seg_idx], *r_prev = NULL, *r_next;
const mm_reg1_t *r = n_regs > 0 && reg_idx < n_regs && reg_idx >= 0? &regs[reg_idx] : NULL;
@@ -437,7 +441,7 @@ void mm_write_sam3(kstring_t *s, const mm_idx_t *mi, const mm_bseq1_t *t, int se
mm_sprintf_lite(s, "\t%s\t%d\t0\t*", mi->seq[this_rid].name, this_pos+1);
} else mm_sprintf_lite(s, "\t*\t0\t0\t*");
} else {
this_rid = r->rid, this_pos = r->rs, this_rev = r->rev;
this_rid = r->rid, this_pos = r->rs;
mm_sprintf_lite(s, "\t%s\t%d\t%d\t", mi->seq[r->rid].name, r->rs+1, r->mapq);
if ((opt_flag & MM_F_LONG_CIGAR) && r->p && r->p->n_cigar > max_bam_cigar_op - 2) {
int n_cigar = r->p->n_cigar;
+30 -13
View File
@@ -87,6 +87,22 @@ mm_reg1_t *mm_gen_regs(void *km, uint32_t hash, int qlen, int n_u, uint64_t *u,
return r;
}
void mm_mark_alt(const mm_idx_t *mi, int n, mm_reg1_t *r)
{
int i;
if (mi->n_alt == 0) return;
for (i = 0; i < n; ++i)
if (mi->seq[r[i].rid].is_alt)
r[i].is_alt = 1;
}
static inline int mm_alt_score(int score, float alt_diff_frac)
{
if (score < 0) return score;
score = (int)(score * (1.0 - alt_diff_frac) + .499);
return score > 0? score : 1;
}
void mm_split_reg(mm_reg1_t *r, mm_reg1_t *r2, int n, int qlen, mm128_t *a)
{
if (n <= 0 || n >= r->cnt) return;
@@ -106,7 +122,7 @@ void mm_split_reg(mm_reg1_t *r, mm_reg1_t *r2, int n, int qlen, mm128_t *a)
r->split |= 1, r2->split |= 2;
}
void mm_set_parent(void *km, float mask_level, int n, mm_reg1_t *r, int sub_diff, int hard_mask_level) // and compute mm_reg1_t::subsc
void mm_set_parent(void *km, float mask_level, int mask_len, int n, mm_reg1_t *r, int sub_diff, int hard_mask_level, float alt_diff_frac) // and compute mm_reg1_t::subsc
{
int i, j, k, *w;
uint64_t *cov;
@@ -146,13 +162,16 @@ skip_uncov:
min = ej - sj < ei - si? ej - sj : ei - si;
max = ej - sj > ei - si? ej - sj : ei - si;
ol = si < sj? (ei < sj? 0 : ei < ej? ei - sj : ej - sj) : (ej < si? 0 : ej < ei? ej - si : ei - si); // overlap length; TODO: this can be simplified
if ((float)ol / min - (float)uncov_len / max > mask_level) {
int cnt_sub = 0;
if ((float)ol / min - (float)uncov_len / max > mask_level && uncov_len <= mask_len) { // then this is a secondary hit
int cnt_sub = 0, sci = ri->score;
ri->parent = rp->parent;
rp->subsc = rp->subsc > ri->score? rp->subsc : ri->score;
if (!rp->is_alt && ri->is_alt) sci = mm_alt_score(sci, alt_diff_frac);
rp->subsc = rp->subsc > sci? rp->subsc : sci;
if (ri->cnt >= rp->cnt) cnt_sub = 1;
if (rp->p && ri->p && (rp->rid != ri->rid || rp->rs != ri->rs || rp->re != ri->re || ol != min)) { // the last condition excludes identical hits after DP
rp->p->dp_max2 = rp->p->dp_max2 > ri->p->dp_max? rp->p->dp_max2 : ri->p->dp_max;
sci = ri->p->dp_max;
if (!rp->is_alt && ri->is_alt) sci = mm_alt_score(sci, alt_diff_frac);
rp->p->dp_max2 = rp->p->dp_max2 > sci? rp->p->dp_max2 : sci;
if (rp->p->dp_max - ri->p->dp_max <= sub_diff) cnt_sub = 1;
}
if (cnt_sub) ++rp->n_sub;
@@ -166,7 +185,7 @@ set_parent_test:
kfree(km, w);
}
void mm_hit_sort(void *km, int *n_regs, mm_reg1_t *r)
void mm_hit_sort(void *km, int *n_regs, mm_reg1_t *r, float alt_diff_frac)
{
int32_t i, n_aux, n = *n_regs, has_cigar = 0, no_cigar = 0;
mm128_t *aux;
@@ -177,13 +196,11 @@ void mm_hit_sort(void *km, int *n_regs, mm_reg1_t *r)
t = (mm_reg1_t*)kmalloc(km, n * sizeof(mm_reg1_t));
for (i = n_aux = 0; i < n; ++i) {
if (r[i].inv || r[i].cnt > 0) { // squeeze out elements with cnt==0 (soft deleted)
if (r[i].p) {
aux[n_aux].x = (uint64_t)r[i].p->dp_max << 32 | r[i].hash;
has_cigar = 1;
} else {
aux[n_aux].x = (uint64_t)r[i].score << 32 | r[i].hash;
no_cigar = 1;
}
int score;
if (r[i].p) score = r[i].p->dp_max, has_cigar = 1;
else score = r[i].score, no_cigar = 1;
if (r[i].is_alt) score = mm_alt_score(score, alt_diff_frac);
aux[n_aux].x = (uint64_t)score << 32 | r[i].hash;
aux[n_aux++].y = i;
} else if (r[i].p) {
free(r[i].p);
+30 -3
View File
@@ -117,8 +117,8 @@ void mm_idx_stat(const mm_idx_t *mi)
if (kh_key(h, k)&1) ++n1;
}
}
fprintf(stderr, "[M::%s::%.3f*%.2f] distinct minimizers: %d (%.2f%% are singletons); average occurrences: %.3lf; average spacing: %.3lf\n",
__func__, realtime() - mm_realtime0, cputime() / (realtime() - mm_realtime0), n, 100.0*n1/n, (double)sum / n, (double)len / sum);
fprintf(stderr, "[M::%s::%.3f*%.2f] distinct minimizers: %d (%.2f%% are singletons); average occurrences: %.3lf; average spacing: %.3lf; total length: %ld\n",
__func__, realtime() - mm_realtime0, cputime() / (realtime() - mm_realtime0), n, 100.0*n1/n, (double)sum / n, (double)len / sum, (long)len);
}
int mm_idx_index_name(mm_idx_t *mi)
@@ -316,6 +316,7 @@ static void *worker_pipeline(void *shared, int step, void *in)
} else seq->name = 0;
seq->len = s->seq[i].l_seq;
seq->offset = p->sum_len;
seq->is_alt = 0;
// copy the sequence
if (!(p->mi->flag & MM_I_NO_SEQ)) {
for (j = 0; j < seq->len; ++j) { // TODO: this is not the fastest way, but let's first see if speed matters here
@@ -414,6 +415,7 @@ mm_idx_t *mm_idx_str(int w, int k, int is_hpc, int bucket_bits, int n, const cha
}
p->offset = sum_len;
p->len = strlen(s);
p->is_alt = 0;
for (j = 0; j < p->len; ++j) {
int c = seq_nt4_table[(uint8_t)s[j]];
uint64_t o = sum_len + j;
@@ -500,6 +502,7 @@ mm_idx_t *mm_idx_load(FILE *fp)
}
fread(&s->len, 4, 1, fp);
s->offset = sum_len;
s->is_alt = 0;
sum_len += s->len;
}
for (i = 0; i < 1<<mi->b; ++i) {
@@ -607,6 +610,30 @@ int mm_idx_reader_eof(const mm_idx_reader_t *r) // TODO: in extremely rare cases
#include "kseq.h"
KSTREAM_DECLARE(gzFile, gzread)
int mm_idx_alt_read(mm_idx_t *mi, const char *fn)
{
int n_alt = 0;
gzFile fp;
kstream_t *ks;
kstring_t str = {0,0,0};
fp = fn && strcmp(fn, "-")? gzopen(fn, "r") : gzdopen(fileno(stdin), "r");
if (fp == 0) return -1;
ks = ks_init(fp);
if (mi->h == 0) mm_idx_index_name(mi);
while (ks_getuntil(ks, KS_SEP_LINE, &str, 0) >= 0) {
char *p;
int id;
for (p = str.s; *p && !isspace(*p); ++p) { }
*p = 0;
id = mm_idx_name2id(mi, str.s);
if (id >= 0) mi->seq[id].is_alt = 1, ++n_alt;
}
mi->n_alt = n_alt;
if (mm_verbose >= 3)
fprintf(stderr, "[M::%s] found %d ALT contigs\n", __func__, n_alt);
return n_alt;
}
#define sort_key_bed(a) ((a).st)
KRADIX_SORT_INIT(bed, mm_idx_intv1_t, sort_key_bed, 4)
@@ -627,7 +654,7 @@ mm_idx_intv_t *mm_idx_read_bed(const mm_idx_t *mi, const char *fn, int read_junc
char *p, *q, *bl, *bs;
int32_t i, id = -1, n_blk = 0;
for (p = q = str.s, i = 0;; ++p) {
if (*p == 0 || isspace(*p)) {
if (*p == 0 || *p == '\t') {
int32_t c = *p;
*p = 0;
if (i == 0) { // chr
+21 -14
View File
@@ -18,15 +18,14 @@
* | | | |
* p=p->ptr->ptr->ptr->ptr p->ptr p->ptr->ptr p->ptr->ptr->ptr
*/
#define MIN_CORE_SIZE 0x80000
typedef struct header_t {
size_t size;
struct header_t *ptr;
} header_t;
typedef struct {
void *par;
size_t min_core_size;
header_t base, *loop_head, *core_head; /* base is a zero-sized block always kept in the loop */
} kmem_t;
@@ -36,31 +35,39 @@ static void panic(const char *s)
abort();
}
void *km_init(void)
void *km_init2(void *km_par, size_t min_core_size)
{
return calloc(1, sizeof(kmem_t));
kmem_t *km;
km = (kmem_t*)kcalloc(km_par, 1, sizeof(kmem_t));
km->par = km_par;
km->min_core_size = min_core_size > 0? min_core_size : 0x80000;
return (void*)km;
}
void *km_init(void) { return km_init2(0, 0); }
void km_destroy(void *_km)
{
kmem_t *km = (kmem_t*)_km;
void *km_par;
header_t *p, *q;
if (km == NULL) return;
km_par = km->par;
for (p = km->core_head; p != NULL;) {
q = p->ptr;
free(p);
kfree(km_par, p);
p = q;
}
free(km);
kfree(km_par, km);
}
static header_t *morecore(kmem_t *km, size_t nu)
{
header_t *q;
size_t bytes, *p;
nu = (nu + 1 + (MIN_CORE_SIZE - 1)) / MIN_CORE_SIZE * MIN_CORE_SIZE; /* the first +1 for core header */
nu = (nu + 1 + (km->min_core_size - 1)) / km->min_core_size * km->min_core_size; /* the first +1 for core header */
bytes = nu * sizeof(header_t);
q = (header_t*)malloc(bytes);
q = (header_t*)kmalloc(km->par, bytes);
if (!q) panic("[morecore] insufficient memory");
q->ptr = km->core_head, q->size = nu, km->core_head = q;
p = (size_t*)(q + 1);
@@ -125,7 +132,7 @@ void *kmalloc(void *_km, size_t n_bytes)
if (n_bytes == 0) return 0;
if (km == NULL) return malloc(n_bytes);
n_units = (n_bytes + sizeof(size_t) + sizeof(header_t) - 1) / sizeof(header_t) + 1;
n_units = (n_bytes + sizeof(size_t) + sizeof(header_t) - 1) / sizeof(header_t); /* header+n_bytes requires at least this number of units */
if (!(q = km->loop_head)) /* the first time when kmalloc() is called, intialize it */
q = km->loop_head = km->base.ptr = &km->base;
@@ -160,18 +167,18 @@ void *kcalloc(void *_km, size_t count, size_t size)
void *krealloc(void *_km, void *ap, size_t n_bytes) // TODO: this can be made more efficient in principle
{
kmem_t *km = (kmem_t*)_km;
size_t n_units, *p, *q;
size_t cap, *p, *q;
if (n_bytes == 0) {
kfree(km, ap); return 0;
}
if (km == NULL) return realloc(ap, n_bytes);
if (ap == NULL) return kmalloc(km, n_bytes);
n_units = (n_bytes + sizeof(size_t) + sizeof(header_t) - 1) / sizeof(header_t);
p = (size_t*)ap - 1;
if (*p >= n_units) return ap; /* TODO: this prevents shrinking */
cap = (*p) * sizeof(header_t) - sizeof(size_t);
if (cap >= n_bytes) return ap; /* TODO: this prevents shrinking */
q = (size_t*)kmalloc(km, n_bytes);
memcpy(q, ap, (*p - 1) * sizeof(header_t));
memcpy(q, ap, cap);
kfree(km, ap);
return q;
}
+10
View File
@@ -17,6 +17,7 @@ void *kcalloc(void *km, size_t count, size_t size);
void kfree(void *km, void *ptr);
void *km_init(void);
void *km_init2(void *km_par, size_t min_core_size);
void km_destroy(void *km);
void km_stat(const void *_km, km_stat_t *s);
@@ -24,4 +25,13 @@ void km_stat(const void *_km, km_stat_t *s);
}
#endif
#define KMALLOC(km, ptr, len) ((ptr) = (__typeof__(ptr))kmalloc((km), (len) * sizeof(*(ptr))))
#define KCALLOC(km, ptr, len) ((ptr) = (__typeof__(ptr))kcalloc((km), (len), sizeof(*(ptr))))
#define KREALLOC(km, ptr, len) ((ptr) = (__typeof__(ptr))krealloc((km), (ptr), (len) * sizeof(*(ptr))))
#define KEXPAND(km, a, m) do { \
(m) = (m) >= 4? (m) + ((m)>>1) : 16; \
KREALLOC((km), (a), (m)); \
} while (0)
#endif
+1 -1
View File
@@ -89,7 +89,7 @@
#ifndef KSTRING_T
#define KSTRING_T kstring_t
typedef struct __kstring_t {
unsigned l, m;
size_t l, m;
char *s;
} kstring_t;
#endif
+8
View File
@@ -4,15 +4,23 @@
#include "ksw2.h"
#ifdef __SSE2__
#ifdef USE_SIMDE
#include <simde/x86/sse2.h>
#else
#include <emmintrin.h>
#endif
#ifdef KSW_SSE2_ONLY
#undef __SSE4_1__
#endif
#ifdef __SSE4_1__
#ifdef USE_SIMDE
#include <simde/x86/sse4.1.h>
#else
#include <smmintrin.h>
#endif
#endif
#ifdef KSW_CPU_DISPATCH
#ifdef __SSE4_1__
+8 -1
View File
@@ -4,15 +4,22 @@
#include "ksw2.h"
#ifdef __SSE2__
#ifdef USE_SIMDE
#include <simde/x86/sse2.h>
#else
#include <emmintrin.h>
#endif
#ifdef KSW_SSE2_ONLY
#undef __SSE4_1__
#endif
#ifdef __SSE4_1__
#ifdef USE_SIMDE
#include <simde/x86/sse4.1.h>
#else
#include <smmintrin.h>
#endif
#endif
#ifdef KSW_CPU_DISPATCH
#ifdef __SSE4_1__
+8
View File
@@ -3,15 +3,23 @@
#include "ksw2.h"
#ifdef __SSE2__
#ifdef USE_SIMDE
#include <simde/x86/sse2.h>
#else
#include <emmintrin.h>
#endif
#ifdef KSW_SSE2_ONLY
#undef __SSE4_1__
#endif
#ifdef __SSE4_1__
#ifdef USE_SIMDE
#include <simde/x86/sse4.1.h>
#else
#include <smmintrin.h>
#endif
#endif
#ifdef KSW_CPU_DISPATCH
#ifdef __SSE4_1__
+6 -1
View File
@@ -1,9 +1,14 @@
#include <stdlib.h>
#include <stdint.h>
#include <string.h>
#include <emmintrin.h>
#include "ksw2.h"
#ifdef USE_SIMDE
#include <simde/x86/sse2.h>
#else
#include <emmintrin.h>
#endif
#ifdef __GNUC__
#define LIKELY(x) __builtin_expect((x),1)
#define UNLIKELY(x) __builtin_expect((x),0)
Submodule
+1
Submodule lib/simde added at b30129b3b4
+43 -16
View File
@@ -1,12 +1,13 @@
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <errno.h>
#include "bseq.h"
#include "minimap.h"
#include "mmpriv.h"
#include "ketopt.h"
#define MM_VERSION "2.17-r941"
#define MM_VERSION "2.18-r1015"
#ifdef __linux__
#include <sys/resource.h>
@@ -66,6 +67,10 @@ static ko_longopt_t long_options[] = {
{ "junc-bed", ko_required_argument, 340 },
{ "junc-bonus", ko_required_argument, 341 },
{ "sam-hit-only", ko_no_argument, 342 },
{ "chain-gap-scale",ko_required_argument, 343 },
{ "alt", ko_required_argument, 344 },
{ "alt-drop", ko_required_argument, 345 },
{ "mask-len", ko_required_argument, 346 },
{ "help", ko_no_argument, 'h' },
{ "max-intron-len", ko_required_argument, 'G' },
{ "version", ko_no_argument, 'V' },
@@ -108,7 +113,7 @@ int main(int argc, char *argv[])
mm_mapopt_t opt;
mm_idxopt_t ipt;
int i, c, n_threads = 3, n_parts, old_best_n = -1;
char *fnw = 0, *rg = 0, *junc_bed = 0, *s;
char *fnw = 0, *rg = 0, *junc_bed = 0, *s, *alt_list = 0;
FILE *fp_help = stderr;
mm_idx_reader_t *idx_rdr;
mm_idx_t *mi;
@@ -165,14 +170,14 @@ int main(int argc, char *argv[])
else if (c == 's') opt.min_dp_max = atoi(o.arg);
else if (c == 'C') opt.noncan = atoi(o.arg);
else if (c == 'I') ipt.batch_size = mm_parse_num(o.arg);
else if (c == 'K') opt.mini_batch_size = (int)mm_parse_num(o.arg);
else if (c == 'K') opt.mini_batch_size = mm_parse_num(o.arg);
else if (c == 'R') rg = o.arg;
else if (c == 'h') fp_help = stdout;
else if (c == '2') opt.flag |= MM_F_2_IO_THREADS;
else if (c == 'o') {
if (strcmp(o.arg, "-") != 0) {
if (freopen(o.arg, "wb", stdout) == NULL) {
fprintf(stderr, "[ERROR]\033[1;31m failed to write the output to file '%s'\033[0m\n", o.arg);
fprintf(stderr, "[ERROR]\033[1;31m failed to write the output to file '%s'\033[0m: %s\n", o.arg, strerror(errno));
exit(1);
}
}
@@ -208,7 +213,12 @@ int main(int argc, char *argv[])
else if (c == 337) opt.max_sw_mat = mm_parse_num(o.arg); // --cap-sw-mat
else if (c == 338) opt.max_qlen = mm_parse_num(o.arg); // --max-qlen
else if (c == 340) junc_bed = o.arg; // --junc-bed
else if (c == 341) opt.junc_bonus = atoi(o.arg); // --junc-bonus
else if (c == 342) opt.flag |= MM_F_SAM_HIT_ONLY; // --sam-hit-only
else if (c == 343) opt.chain_gap_scale = atof(o.arg); // --chain-gap-scale
else if (c == 344) alt_list = o.arg; // --alt
else if (c == 345) opt.alt_drop = atof(o.arg); // --alt-drop
else if (c == 346) opt.mask_len = mm_parse_num(o.arg); // --mask-len
else if (c == 314) { // --frag
yes_or_no(&opt, MM_F_FRAG_MODE, o.longidx, o.arg, 1);
} else if (c == 315) { // --secondary
@@ -322,11 +332,11 @@ int main(int argc, char *argv[])
fprintf(fp_help, " --version show version number\n");
fprintf(fp_help, " Preset:\n");
fprintf(fp_help, " -x STR preset (always applied before other options; see minimap2.1 for details) []\n");
fprintf(fp_help, " - map-pb/map-ont: PacBio/Nanopore vs reference mapping\n");
fprintf(fp_help, " - ava-pb/ava-ont: PacBio/Nanopore read overlap\n");
fprintf(fp_help, " - asm5/asm10/asm20: asm-to-ref mapping, for ~0.1/1/5%% sequence divergence\n");
fprintf(fp_help, " - splice: long-read spliced alignment\n");
fprintf(fp_help, " - sr: genomic short-read mapping\n");
fprintf(fp_help, " - map-pb/map-ont - PacBio/Nanopore vs reference mapping\n");
fprintf(fp_help, " - ava-pb/ava-ont - PacBio/Nanopore read overlap\n");
fprintf(fp_help, " - asm5/asm10/asm20 - asm-to-ref mapping, for ~0.1/1/5%% sequence divergence\n");
fprintf(fp_help, " - splice/splice:hq - long-read/Pacbio-CCS spliced alignment\n");
fprintf(fp_help, " - sr - genomic short-read mapping\n");
fprintf(fp_help, "\nSee `man ./minimap2.1' for detailed description of these and other advanced command-line options.\n");
return fp_help == stdout? 0 : 1;
}
@@ -337,7 +347,7 @@ int main(int argc, char *argv[])
}
idx_rdr = mm_idx_reader_open(argv[o.ind], &ipt, fnw);
if (idx_rdr == 0) {
fprintf(stderr, "[ERROR] failed to open file '%s'\n", argv[o.ind]);
fprintf(stderr, "[ERROR] failed to open file '%s': %s\n", argv[o.ind], strerror(errno));
return 1;
}
if (!idx_rdr->is_idx && fnw == 0 && argc - o.ind < 2) {
@@ -348,6 +358,7 @@ int main(int argc, char *argv[])
if (opt.best_n == 0 && (opt.flag&MM_F_CIGAR) && mm_verbose >= 2)
fprintf(stderr, "[WARNING]\033[1;31m `-N 0' reduces alignment accuracy. Please use --secondary=no to suppress secondary alignments.\033[0m\n");
while ((mi = mm_idx_reader_read(idx_rdr, n_threads)) != 0) {
int ret;
if ((opt.flag & MM_F_CIGAR) && (mi->flag & MM_I_NO_SEQ)) {
fprintf(stderr, "[ERROR] the prebuilt index doesn't contain sequences.\n");
mm_idx_destroy(mi);
@@ -356,12 +367,20 @@ int main(int argc, char *argv[])
}
if ((opt.flag & MM_F_OUT_SAM) && idx_rdr->n_parts == 1) {
if (mm_idx_reader_eof(idx_rdr)) {
mm_write_sam_hdr(mi, rg, MM_VERSION, argc, argv);
if (opt.split_prefix == 0)
ret = mm_write_sam_hdr(mi, rg, MM_VERSION, argc, argv);
else
ret = mm_write_sam_hdr(0, rg, MM_VERSION, argc, argv);
} else {
mm_write_sam_hdr(0, rg, MM_VERSION, argc, argv);
ret = mm_write_sam_hdr(0, rg, MM_VERSION, argc, argv);
if (opt.split_prefix == 0 && mm_verbose >= 2)
fprintf(stderr, "[WARNING]\033[1;31m For a multi-part index, no @SQ lines will be outputted. Please use --split-prefix.\033[0m\n");
}
if (ret != 0) {
mm_idx_destroy(mi);
mm_idx_reader_close(idx_rdr);
return 1;
}
}
if (mm_verbose >= 3)
fprintf(stderr, "[M::%s::%.3f*%.2f] loaded/built the index for %d target sequence(s)\n",
@@ -369,13 +388,21 @@ int main(int argc, char *argv[])
if (argc != o.ind + 1) mm_mapopt_update(&opt, mi);
if (mm_verbose >= 3) mm_idx_stat(mi);
if (junc_bed) mm_idx_bed_read(mi, junc_bed, 1);
if (alt_list) mm_idx_alt_read(mi, alt_list);
ret = 0;
if (!(opt.flag & MM_F_FRAG_MODE)) {
for (i = o.ind + 1; i < argc; ++i)
mm_map_file(mi, argv[i], &opt, n_threads);
for (i = o.ind + 1; i < argc; ++i) {
ret = mm_map_file(mi, argv[i], &opt, n_threads);
if (ret < 0) break;
}
} else {
mm_map_file_frag(mi, argc - (o.ind + 1), (const char**)&argv[o.ind + 1], &opt, n_threads);
ret = mm_map_file_frag(mi, argc - (o.ind + 1), (const char**)&argv[o.ind + 1], &opt, n_threads);
}
mm_idx_destroy(mi);
if (ret < 0) {
fprintf(stderr, "ERROR: failed to map the query file\n");
exit(EXIT_FAILURE);
}
}
n_parts = idx_rdr->n_parts;
mm_idx_reader_close(idx_rdr);
@@ -384,7 +411,7 @@ int main(int argc, char *argv[])
mm_split_merge(argc - (o.ind + 1), (const char**)&argv[o.ind + 1], &opt, n_parts);
if (fflush(stdout) == EOF) {
fprintf(stderr, "[ERROR] failed to write the results\n");
perror("[ERROR] failed to write the results");
exit(EXIT_FAILURE);
}
+15 -9
View File
@@ -1,6 +1,7 @@
#include <stdlib.h>
#include <string.h>
#include <assert.h>
#include <errno.h>
#include "kthread.h"
#include "kvec.h"
#include "kalloc.h"
@@ -248,7 +249,7 @@ static mm128_t *collect_seed_hits(void *km, const mm_mapopt_t *opt, int max_occ,
static void chain_post(const mm_mapopt_t *opt, int max_chain_gap_ref, const mm_idx_t *mi, void *km, int qlen, int n_segs, const int *qlens, int *n_regs, mm_reg1_t *regs, mm128_t *a)
{
if (!(opt->flag & MM_F_ALL_CHAINS)) { // don't choose primary mapping(s)
mm_set_parent(km, opt->mask_level, *n_regs, regs, opt->a * 2 + opt->b, opt->flag&MM_F_HARD_MLEVEL);
mm_set_parent(km, opt->mask_level, opt->mask_len, *n_regs, regs, opt->a * 2 + opt->b, opt->flag&MM_F_HARD_MLEVEL, opt->alt_drop);
if (n_segs <= 1) mm_select_sub(km, opt->pri_ratio, mi->k*2, opt->best_n, n_regs, regs);
else mm_select_sub_multi(km, opt->pri_ratio, 0.2f, 0.7f, max_chain_gap_ref, mi->k*2, opt->best_n, n_segs, qlens, n_regs, regs);
if (!(opt->flag & (MM_F_SPLICE|MM_F_SR|MM_F_NO_LJOIN))) // long join not working well without primary chains
@@ -261,7 +262,7 @@ static mm_reg1_t *align_regs(const mm_mapopt_t *opt, const mm_idx_t *mi, void *k
if (!(opt->flag & MM_F_CIGAR)) return regs;
regs = mm_align_skeleton(km, opt, mi, qlen, seq, n_regs, regs, a); // this calls mm_filter_regs()
if (!(opt->flag & MM_F_ALL_CHAINS)) { // don't choose primary mapping(s)
mm_set_parent(km, opt->mask_level, *n_regs, regs, opt->a * 2 + opt->b, opt->flag&MM_F_HARD_MLEVEL);
mm_set_parent(km, opt->mask_level, opt->mask_len, *n_regs, regs, opt->a * 2 + opt->b, opt->flag&MM_F_HARD_MLEVEL, opt->alt_drop);
mm_select_sub(km, opt->pri_ratio, mi->k*2, opt->best_n, n_regs, regs);
mm_set_sam_pri(*n_regs, regs);
}
@@ -312,7 +313,7 @@ void mm_map_frag(const mm_idx_t *mi, int n_segs, const int *qlens, const char **
if (max_chain_gap_ref < opt->max_gap) max_chain_gap_ref = opt->max_gap;
} else max_chain_gap_ref = opt->max_gap;
a = mm_chain_dp(max_chain_gap_ref, max_chain_gap_qry, opt->bw, opt->max_chain_skip, opt->max_chain_iter, opt->min_cnt, opt->min_chain_score, is_splice, n_segs, n_a, a, &n_regs0, &u, b->km);
a = mm_chain_dp(max_chain_gap_ref, max_chain_gap_qry, opt->bw, opt->max_chain_skip, opt->max_chain_iter, opt->min_cnt, opt->min_chain_score, opt->chain_gap_scale, is_splice, n_segs, n_a, a, &n_regs0, &u, b->km);
if (opt->max_occ > opt->mid_occ && rep_len > 0) {
int rechain = 0;
@@ -334,13 +335,17 @@ void mm_map_frag(const mm_idx_t *mi, int n_segs, const int *qlens, const char **
kfree(b->km, mini_pos);
if (opt->flag & MM_F_HEAP_SORT) a = collect_seed_hits_heap(b->km, opt, opt->max_occ, mi, qname, &mv, qlen_sum, &n_a, &rep_len, &n_mini_pos, &mini_pos);
else a = collect_seed_hits(b->km, opt, opt->max_occ, mi, qname, &mv, qlen_sum, &n_a, &rep_len, &n_mini_pos, &mini_pos);
a = mm_chain_dp(max_chain_gap_ref, max_chain_gap_qry, opt->bw, opt->max_chain_skip, opt->max_chain_iter, opt->min_cnt, opt->min_chain_score, is_splice, n_segs, n_a, a, &n_regs0, &u, b->km);
a = mm_chain_dp(max_chain_gap_ref, max_chain_gap_qry, opt->bw, opt->max_chain_skip, opt->max_chain_iter, opt->min_cnt, opt->min_chain_score, opt->chain_gap_scale, is_splice, n_segs, n_a, a, &n_regs0, &u, b->km);
}
}
b->frag_gap = max_chain_gap_ref;
b->rep_len = rep_len;
regs0 = mm_gen_regs(b->km, hash, qlen_sum, n_regs0, u, a);
if (mi->n_alt) {
mm_mark_alt(mi, n_regs0, regs0);
mm_hit_sort(b->km, &n_regs0, regs0, opt->alt_drop); // this step can be merged into mm_gen_regs(); will do if this shows up in profile
}
if (mm_dbg_flag & MM_DBG_PRINT_SEED)
for (j = 0; j < n_regs0; ++j)
@@ -360,7 +365,7 @@ void mm_map_frag(const mm_idx_t *mi, int n_segs, const int *qlens, const char **
seg = mm_seg_gen(b->km, hash, n_segs, qlens, n_regs0, regs0, n_regs, regs, a); // split fragment chain to separate segment chains
free(regs0);
for (i = 0; i < n_segs; ++i) {
mm_set_parent(b->km, opt->mask_level, n_regs[i], regs[i], opt->a * 2 + opt->b, opt->flag&MM_F_HARD_MLEVEL); // update mm_reg1_t::parent
mm_set_parent(b->km, opt->mask_level, opt->mask_len, n_regs[i], regs[i], opt->a * 2 + opt->b, opt->flag&MM_F_HARD_MLEVEL, opt->alt_drop); // update mm_reg1_t::parent
regs[i] = align_regs(opt, mi, b->km, qlens[i], seqs[i], &n_regs[i], regs[i], seg[i].a);
mm_set_mapq(b->km, n_regs[i], regs[i], opt->min_chain_score, opt->a, rep_len, is_sr);
}
@@ -398,7 +403,8 @@ mm_reg1_t *mm_map(const mm_idx_t *mi, int qlen, const char *seq, int *n_regs, mm
**************************/
typedef struct {
int mini_batch_size, n_processed, n_threads, n_fp;
int n_processed, n_threads, n_fp;
int64_t mini_batch_size;
const mm_mapopt_t *opt;
mm_bseq_file_t **fp;
const mm_idx_t *mi;
@@ -502,8 +508,8 @@ static void merge_hits(step_t *s)
}
}
}
mm_hit_sort(km, &s->n_reg[k], s->reg[k]);
mm_set_parent(km, opt->mask_level, s->n_reg[k], s->reg[k], opt->a * 2 + opt->b, opt->flag&MM_F_HARD_MLEVEL);
mm_hit_sort(km, &s->n_reg[k], s->reg[k], opt->alt_drop);
mm_set_parent(km, opt->mask_level, opt->mask_len, s->n_reg[k], s->reg[k], opt->a * 2 + opt->b, opt->flag&MM_F_HARD_MLEVEL, opt->alt_drop);
if (!(opt->flag & MM_F_ALL_CHAINS)) {
mm_select_sub(km, opt->pri_ratio, s->p->mi->k*2, opt->best_n, &s->n_reg[k], s->reg[k]);
mm_set_sam_pri(s->n_reg[k], s->reg[k]);
@@ -622,7 +628,7 @@ static mm_bseq_file_t **open_bseqs(int n, const char **fn)
for (i = 0; i < n; ++i) {
if ((fp[i] = mm_bseq_open(fn[i])) == 0) {
if (mm_verbose >= 1)
fprintf(stderr, "ERROR: failed to open file '%s'\n", fn[i]);
fprintf(stderr, "ERROR: failed to open file '%s': %s\n", fn[i], strerror(errno));
for (j = 0; j < i; ++j)
mm_bseq_close(fp[j]);
free(fp);
+10 -3
View File
@@ -58,12 +58,14 @@ typedef struct {
char *name; // name of the db sequence
uint64_t offset; // offset in mm_idx_t::S
uint32_t len; // length
uint32_t is_alt;
} mm_idx_seq_t;
typedef struct {
int32_t b, w, k, flag;
uint32_t n_seq; // number of reference sequences
int32_t index;
int32_t n_alt;
mm_idx_seq_t *seq; // sequence name, length and offset
uint32_t *S; // 4-bit packed sequence
struct mm_idx_bucket_s *B; // index (hidden)
@@ -91,7 +93,7 @@ typedef struct {
int32_t mlen, blen; // seeded exact match length; seeded alignment block length
int32_t n_sub; // number of suboptimal mappings
int32_t score0; // initial chaining score (before chain merging/spliting)
uint32_t mapq:8, split:2, rev:1, inv:1, sam_pri:1, proper_frag:1, pe_thru:1, seg_split:1, seg_id:8, split_inv:1, dummy:7;
uint32_t mapq:8, split:2, rev:1, inv:1, sam_pri:1, proper_frag:1, pe_thru:1, seg_split:1, seg_id:8, split_inv:1, is_alt:1, dummy:6;
uint32_t hash;
float div;
mm_extra_t *p;
@@ -100,7 +102,7 @@ typedef struct {
// indexing and mapping options
typedef struct {
short k, w, flag, bucket_bits;
int mini_batch_size;
int64_t mini_batch_size;
uint64_t batch_size;
} mm_idxopt_t;
@@ -117,8 +119,10 @@ typedef struct {
int max_chain_skip, max_chain_iter;
int min_cnt; // min number of minimizers on each chain
int min_chain_score; // min chaining score
float chain_gap_scale;
float mask_level;
int mask_len;
float pri_ratio;
int best_n; // top best_n chains are subjected to DP alignment
@@ -126,6 +130,8 @@ typedef struct {
int min_join_flank_sc;
float min_join_flank_ratio;
float alt_drop;
int a, b, q, e, q2, e2; // matching score, mismatch, gap-open and gap-ext penalties
int sc_ambi; // score when one or both bases are "N"
int noncan; // cost of non-canonical splicing sites
@@ -143,7 +149,7 @@ typedef struct {
int32_t min_mid_occ;
int32_t mid_occ; // ignore seeds with occurrences above this threshold
int32_t max_occ;
int mini_batch_size; // size of a batch of query bases to process in parallel
int64_t mini_batch_size; // size of a batch of query bases to process in parallel
int64_t max_sw_mat;
const char *split_prefix;
@@ -368,6 +374,7 @@ int mm_idx_index_name(mm_idx_t *mi);
int mm_idx_name2id(const mm_idx_t *mi, const char *name);
int mm_idx_getseq(const mm_idx_t *mi, uint32_t rid, uint32_t st, uint32_t en, uint8_t *seq);
int mm_idx_alt_read(mm_idx_t *mi, const char *fn);
int mm_idx_bed_read(mm_idx_t *mi, const char *fn, int read_junc);
int mm_idx_bed_junc(const mm_idx_t *mi, int32_t ctg, int32_t st, int32_t en, uint8_t *s);
+22 -3
View File
@@ -1,4 +1,4 @@
.TH minimap2 1 "4 May 2019" "minimap2-2.17 (r941)" "Bioinformatics tools"
.TH minimap2 1 "9 April 2021" "minimap2-2.18 (r1015)" "Bioinformatics tools"
.SH NAME
.PP
minimap2 - mapping and alignment between collections of DNA sequences
@@ -121,6 +121,14 @@ provided as the target sequences, options
.BR -w ,
.B -I
will be effectively overridden by the options stored in the index file.
.TP
.BI --alt \ FILE
List of ALT contigs [null]
.TP
.BI --alt-drop \ FLOAT
Drop ALT hits by
.I FLOAT
fraction when ranking and computing mapping quality [0.15]
.SS Mapping options
.TP 10
.BI -f \ FLOAT | INT1 [, INT2 ]
@@ -229,7 +237,14 @@ or more of the shorter chain [0.5]
.B --hard-mask-level
Honor option
.B -M
and disable a heurstic to save unmapped subsequences.
and disable a heurstic to save unmapped subsequences and disables
.BR --mask-len .
.TP
.BI --mask-len \ NUM
Keep an alignment if dropping it leaves an unaligned region on query longer than
.IR INT
[inf]. Effective without
.BR --hard-mask-level .
.TP
.BI --max-chain-skip \ INT
A heuristics that stops chaining early [25]. Minimap2 uses dynamic programming
@@ -245,6 +260,9 @@ Check up to
partial chains during chaining [5000]. This is a heuristic to avoid quadratic
time complexity in the worst case.
.TP
.BI --chain-gap-scale \ FLOAT
Scale of gap cost during chaining [1.0]
.TP
.B --no-long-join
Disable the long gap patching heuristic. When this option is applied, the
maximum alignment gap is mostly controlled by
@@ -373,7 +391,7 @@ BED12 file can be converted from GTF/GFF3 with `paftools.js gff2bed anno.gtf'
.BR --junc-bonus \ INT
Score bonus for a splice donor or acceptor found in annotation (effective with
.BR --junc-bed )
[0].
[9].
.TP
.BI --end-seed-pen \ INT
Drop a terminal anchor if
@@ -638,6 +656,7 @@ s2 i Chaining score of the best secondary chain
NM i Total number of mismatches and gaps in the alignment
MD Z To generate the ref sequence in the alignment
AS i DP alignment score
SA Z List of other supplementary alignments
ms i DP score of the max scoring segment in the alignment
nn i Number of ambiguous bases in the alignment
ts A Transcript strand (splice mode only)
+3 -3
View File
@@ -125,7 +125,7 @@ void mm_err_puts(const char *str)
int ret;
ret = puts(str);
if (ret == EOF) {
fprintf(stderr, "[ERROR] failed to write the results\n");
perror("[ERROR] failed to write the results");
exit(EXIT_FAILURE);
}
}
@@ -135,7 +135,7 @@ void mm_err_fwrite(const void *p, size_t size, size_t nitems, FILE *fp)
int ret;
ret = fwrite(p, size, nitems, fp);
if (ret == EOF) {
fprintf(stderr, "[ERROR] failed to write data\n");
perror("[ERROR] failed to write data");
exit(EXIT_FAILURE);
}
}
@@ -145,7 +145,7 @@ void mm_err_fread(void *p, size_t size, size_t nitems, FILE *fp)
int ret;
ret = fread(p, size, nitems, fp);
if (ret == EOF) {
fprintf(stderr, "[ERROR] failed to read data\n");
perror("[ERROR] failed to read data");
exit(EXIT_FAILURE);
}
}
+385 -28
View File
@@ -1,6 +1,6 @@
#!/usr/bin/env k8
var paftools_version = '2.17-r941';
var paftools_version = '2.18-r1015';
/*****************************
***** Library functions *****
@@ -640,6 +640,23 @@ function paf_asmstat(args)
}
}
function AUN(lens, tot) {
lens.sort(function(a,b) { return b - a; });
if (tot == null) {
tot = 0;
for (var k = 0; k < lens.length; ++k)
tot += lens[k];
}
var x = 0, y = 0;
for (var k = 0; k < lens.length; ++k) {
var l = x + lens[k] <= tot? lens[k] : tot - x;
x += lens[k];
y += l * (l / tot);
if (x >= tot) break;
}
return y.toFixed(0);
}
function count_bp(bp, min_blen, min_gap) {
var n_bp = 0;
for (var k = 0; k < bp.length; ++k)
@@ -660,7 +677,7 @@ function paf_asmstat(args)
return (NM - n_gaps + n_gapo) / (n_M + n_gapo);
}
var labels = ['Length', 'l_cov', 'Rcov', 'Rdup', 'Qcov', 'NG75', 'NG50', 'NGA50', '#breaks', 'bp(' + min_seg_len + ',0)', 'bp(' + min_seg_len + ',10k)'];
var labels = ['Length', 'l_cov', 'Rcov', 'Rdup', 'Qcov', 'NG75', 'NG50', 'NGA50', 'AUNGA', '#breaks', 'bp(' + min_seg_len + ',0)', 'bp(' + min_seg_len + ',10k)'];
var rst = [];
for (var i = 0; i < labels.length; ++i)
rst[i] = [];
@@ -688,11 +705,9 @@ function paf_asmstat(args)
qinfo[t[0]].bp = [];
if (t.length < 9 || t[5] == "*") continue;
if (!/\ttp:A:[PI]/.test(line)) continue;
if ((m = /\tcg:Z:(\S+)/.exec(line)) == null) continue;
var cigar = m[1];
if ((m = /\tNM:i:(\d+)/.exec(line)) == null) continue;
var NM = parseInt(m[1]);
var diff = compute_diff(cigar, NM);
var cigar = (m = /\tcg:Z:(\S+)/.exec(line)) != null? m[1] : null;
var NM = (m = /\tNM:i:(\d+)/.exec(line)) != null? parseInt(m[1]) : null;
var diff = cigar != null && NM != null? compute_diff(cigar, NM) : 0;
t[2] = parseInt(t[2]);
t[3] = parseInt(t[3]);
t[7] = parseInt(t[7]);
@@ -765,10 +780,13 @@ function paf_asmstat(args)
// compute NGA50
rst[7][i] = N50(qblock_len, ref_len, 0.5);
// compute AUNGA
rst[8][i] = AUN(qblock_len, ref_len);
// compute break points
rst[8][i] = n_breaks;
rst[9][i] = count_bp(bp, 500, 0);
rst[10][i] = count_bp(bp, 500, 10000);
rst[9][i] = n_breaks;
rst[10][i] = count_bp(bp, 500, 0);
rst[11][i] = count_bp(bp, 500, 10000);
// nb-plot; NOT USED
/*
@@ -887,23 +905,24 @@ function paf_asmgene(args)
gene_nr[gene_list[last][0]] = 1;
// count and print
var col1 = ["full_sgl", "full_dup", "frag", "part50+", "part10+", "part10-"];
var col1 = ["full_sgl", "full_dup", "frag", "part50+", "part10+", "part10-", "dup_cnt", "dup_sum"];
var rst = [];
for (var k = 0; k < col1.length; ++k) {
rst[k] = [];
for (var i = 0; i < n_fn; ++i)
rst[k][i] = 0;
}
for (var g in gene) {
for (var g in gene) { // count single-copy genes
if (gene[g][0] == null || gene[g][0][0] != 1) continue;
if (gene_nr[g] == null) continue;
if (auto_only && /^(chr)?[XY]$/.test(refpos[g][2])) continue;
for (var i = 0; i < n_fn; ++i) {
if (gene[g][i] == null) {
rst[4][i]++;
rst[5][i]++;
if (print_err) print('M', header[i], refpos[g].join("\t"));
} else if (gene[g][i][0] == 1) rst[0][i]++;
else if (gene[g][i][0] > 1) {
} else if (gene[g][i][0] == 1) {
rst[0][i]++;
} else if (gene[g][i][0] > 1) {
rst[1][i]++;
if (print_err) print('D', header[i], refpos[g].join("\t"));
} else if (gene[g][i][1] >= opt.min_cov) {
@@ -921,6 +940,19 @@ function paf_asmgene(args)
}
}
}
for (var g in gene) { // count multi-copy genes
if (gene[g][0] == null || gene[g][0][0] <= 1) continue;
if (gene_nr[g] == null) continue;
if (auto_only && /^(chr)?[XY]$/.test(refpos[g][2])) continue;
for (var i = 0; i < n_fn; ++i) {
if (gene[g][i] != null) rst[7][i] += gene[g][i][0];
if (gene[g][i] != null && gene[g][i][0] > 1) {
rst[6][i]++;
} else if (print_err) {
print('d', header[i], gene[g][0][0], refpos[g].join("\t"));
}
}
}
print('H', 'Metric', header.join("\t"));
for (var k = 0; k < rst.length; ++k) {
print('X', col1[k], rst[k].join("\t"));
@@ -930,12 +962,13 @@ function paf_asmgene(args)
function paf_stat(args)
{
var c, gap_out_len = null;
while ((c = getopt(args, "l:")) != null)
var c, gap_out_len = null, count_err = false;
while ((c = getopt(args, "cl:")) != null)
if (c == 'l') gap_out_len = parseInt(getopt.arg);
else if (c == 'c') count_err = true;
if (getopt.ind == args.length) {
print("Usage: paftools.js stat [-l gapOutLen] <in.sam>|<in.paf>");
print("Usage: paftools.js stat [-c] [-l gapOutLen] <in.sam>|<in.paf>");
exit(1);
}
@@ -966,7 +999,7 @@ function paf_stat(args)
if (line.charAt(0) != '@') {
var t = line.split("\t", 12);
var m, rs, cigar = null, is_pri = false, is_sam = false, is_rev = false, tname = null;
var atlen = null, aqlen, qs, qe, mapq, ori_qlen;
var atlen = null, aqlen, qs, qe, mapq, ori_qlen, NM = null;
if (t.length < 2) continue;
if (t[4] == '+' || t[4] == '-' || t[4] == '*') { // PAF
if (t[4] == '*') continue; // unmapped
@@ -974,6 +1007,8 @@ function paf_stat(args)
++n_2nd;
continue;
}
if ((m = /\tNM:i:(\d+)/.exec(line)) != null)
NM = parseInt(m[1]);
if ((m = /\tcg:Z:(\S+)/.exec(line)) != null)
cigar = m[1];
if (cigar == null) {
@@ -995,6 +1030,8 @@ function paf_stat(args)
++n_2nd;
continue;
}
if ((m = /\tNM:i:(\d+)/.exec(line)) != null)
NM = parseInt(m[1]);
cigar = t[5];
tname = t[2];
rs = parseInt(t[3]) - 1;
@@ -1013,11 +1050,13 @@ function paf_stat(args)
++n_seq, last = t[0];
}
var M = 0, tl = 0, ql = 0, clip = [0, 0], n_cigar = 0, sclip = 0;
var n_gapo = 0, n_gap_all = 0, l_match = 0;
while ((m = re.exec(cigar)) != null) {
var l = parseInt(m[1]);
++n_cigar;
if (m[2] == 'M' || m[2] == '=' || m[2] == 'X') {
tl += l, ql += l, M += l;
l_match += l;
} else if (m[2] == 'I' || m[2] == 'D') {
var type;
if (l < 50) type = 0;
@@ -1030,6 +1069,7 @@ function paf_stat(args)
else tl += l, ++n_gap[1][type];
if (gap_out_len != null && l >= gap_out_len)
print(t[0], ql, is_rev? '-' : '+', tname, rs + tl, m[2], l);
++n_gapo, n_gap_all += l;
} else if (m[2] == 'N') {
tl += l;
} else if (m[2] == 'S') {
@@ -1047,6 +1087,12 @@ function paf_stat(args)
qs = clip[is_rev? 1 : 0], qe = qs + ql;
ori_qlen = clip[0] + ql + clip[1];
}
if (count_err && NM != null) {
var n_mm = NM - n_gap_all;
if (n_mm < 0) warn("WARNING: NM is smaller than the number of gaps at line " + lineno);
if (n_mm < 0) n_mm = 0;
print(t[0], ori_qlen, t[11], ori_qlen - (qe - qs), NM, l_match + n_gap_all, n_mm + n_gapo, l_match + n_gapo);
}
regs.push([qs, qe]);
last_qlen = ori_qlen;
}
@@ -1059,7 +1105,7 @@ function paf_stat(args)
file.close();
buf.destroy();
if (gap_out_len == null) {
if (gap_out_len == null && !count_err) {
print("Number of mapped sequences: " + n_seq);
print("Number of primary alignments: " + n_pri);
print("Number of secondary alignments: " + n_2nd);
@@ -1509,6 +1555,7 @@ function paf_gff2bed(args)
var colors = {
'protein_coding':'0,128,255',
'mRNA':'0,128,255',
'lincRNA':'0,192,0',
'snRNA':'0,192,0',
'miRNA':'0,192,0',
@@ -1541,8 +1588,8 @@ function paf_gff2bed(args)
print(a[0][0], st, en, name, 1000, a[0][3], cds_st, cds_en, color, a.length, sizes.join(",") + ",", starts.join(",") + ",");
}
var re_gtf = /(transcript_id|transcript_type|transcript_biotype|gene_name|transcript_name) "([^"]+)";/g;
var re_gff3 = /(transcript_id|transcript_type|transcript_biotype|gene_name|transcript_name)=([^;]+)/g;
var re_gtf = /\b(transcript_id|transcript_type|transcript_biotype|gene_name|gene_id|gbkey|transcript_name) "([^"]+)";/g;
var re_gff3 = /\b(transcript_id|transcript_type|transcript_biotype|gene_name|gene_id|gbkey|transcript_name)=([^;]+)/g;
var buf = new Bytes();
var file = args[getopt.ind] == '-'? new File() : new File(args[getopt.ind]);
@@ -1559,19 +1606,19 @@ function paf_gff2bed(args)
if (t[2] != "CDS" && t[2] != "exon") continue;
t[3] = parseInt(t[3]) - 1;
t[4] = parseInt(t[4]);
var id = null, type = "", gname = "N/A", biotype = "", m, tname = "N/A";
var id = null, type = "", name = "N/A", biotype = "", m, tname = "N/A";
while ((m = re_gtf.exec(t[8])) != null) {
if (m[1] == "transcript_id") id = m[2];
else if (m[1] == "transcript_type") type = m[2];
else if (m[1] == "transcript_biotype") biotype = m[2];
else if (m[1] == "gene_name") name = m[2];
else if (m[1] == "transcript_biotype" || m[1] == "gbkey") biotype = m[2];
else if (m[1] == "gene_name" || m[1] == "gene_id") name = m[2];
else if (m[1] == "transcript_name") tname = m[2];
}
while ((m = re_gff3.exec(t[8])) != null) {
if (m[1] == "transcript_id") id = m[2];
else if (m[1] == "transcript_type") type = m[2];
else if (m[1] == "transcript_biotype") biotype = m[2];
else if (m[1] == "gene_name") name = m[2];
else if (m[1] == "transcript_biotype" || m[1] == "gbkey") biotype = m[2];
else if (m[1] == "gene_name" || m[1] == "gene_id") name = m[2];
else if (m[1] == "transcript_name") tname = m[2];
}
if (type == "" && biotype != "") type = biotype;
@@ -2479,6 +2526,310 @@ function paf_ov_eval(args)
print((100 * (1 - n_missing / n_ovlp)).toFixed(2) + "% sensitivity");
}
function paf_vcfstat(args)
{
var c, ts = { "AG":1, "GA":1, "CT":1, "TC":1 };
while ((c = getopt(args, "")) != null) {
}
var buf = new Bytes();
var file = args.length == getopt.ind? new File() : new File(args[getopt.ind]);
var x = { sub:0, ts:0, tv:0, ins:0, del:0, ins1:0, del1:0, ins2:0, del2:0, ins50:0, del50:0, ins1k:0, del1k:0, ins7k:0, del7k:0, insinf:0, delinf:0 };
while (file.readline(buf) >= 0) {
var t = buf.toString().split("\t");
if (t[0][0] == '#') continue;
var alt = t[4].split(",");
var ref = t[3];
for (var i = 0; i < alt.length; ++i) {
var a = alt[i];
if (a[0] == '<' || a[1] == '>') continue;
var l = ref.length < a.length? ref.length : a.length;
for (var j = 0; j < l; ++j) {
if (ref[j] != a[j]) {
++x.sub;
if (ts[ref[j] + a[j]]) ++x.ts;
else ++x.tv;
}
}
var d = a.length - ref.length;
if (d > 0) {
++x.ins;
if (d == 1) ++x.ins1;
else if (d == 2) ++x.ins2;
else if (d < 50) ++x.ins50;
else if (d < 1000) ++x.ins1k;
else if (d < 7000) ++x.ins7k;
else ++x.insinf;
} else if (d < 0) {
d = -d;
++x.del;
if (d == 1) ++x.del1;
else if (d == 2) ++x.del2;
else if (d < 50) ++x.del50;
else if (d < 1000) ++x.del1k;
else if (d < 7000) ++x.del7k;
else ++x.delinf;
}
}
}
file.close();
buf.destroy();
print("# substitutions: " + x.sub);
print("ts/tv: " + (x.ts / x.tv).toFixed(3));
print("# insertions: " + x.ins);
print("# 1bp insertions: " + x.ins1);
print("# 2bp insertions: " + x.ins2);
print("# [3,50) insertions: " + x.ins50);
print("# [50,1000) insertions: " + x.ins1k);
print("# [1000,7000) insertions: " + x.ins7k);
print("# >=7000 insertions: " + x.insinf);
print("# deletions: " + x.del);
print("# 1bp deletions: " + x.del1);
print("# 2bp deletions: " + x.del2);
print("# [3,50) deletions: " + x.del50);
print("# [50,1000) deletions: " + x.del1k);
print("# [1000,7000) deletions: " + x.del7k);
print("# >=7000 deletions: " + x.delinf);
}
function paf_parseNum(s) {
var m, x = null;
if ((m = /^(\d*\.?\d*)([mMgGkK]?)/.exec(s)) != null) {
x = parseFloat(m[1]);
if (m[2] == 'k' || m[2] == 'K') x *= 1000;
else if (m[2] == 'm' || m[2] == 'M') x *= 1000000;
else if (m[2] == 'g' || m[2] == 'G') x *= 1000000000;
}
return Math.floor(x + .499);
}
function paf_misjoin(args)
{
var c, min_seg_len = 1000000, max_gap = 1000000, fn_cen = null, show_long = false, show_err = false, cen_ratio = 0.5;
var n_diff = [0, 0], n_gap = [0, 0], n_inv = [0, 0], n_inv_end = [0, 0];
while ((c = getopt(args, "l:g:c:per:")) != null) {
if (c == 'l') min_seg_len = paf_parseNum(getopt.arg);
else if (c == 'g') max_gap = paf_parseNum(getopt.arg);
else if (c == 'c') fn_cen = getopt.arg;
else if (c == 'r') cen_ratio = parseFloat(getopt.arg);
else if (c == 'p') show_long = true;
else if (c == 'e') show_err = true;
}
if (args.length == getopt.ind) {
print("Usage: paftools.js misjoin [options] <in.paf>");
print("Options:");
print(" -c FILE BED for centromeres []");
print(" -r FLOAT count a centromeric event if overlap ratio > FLOAT [" + cen_ratio + "]");
print(" -l NUM min alignment block length [1m]");
print(" -g NUM max gap size [1m]");
print(" -e output misjoins not involving centromeres");
print(" -p output long alignment blocks for debugging");
return;
}
var cen = {};
var file, buf = new Bytes();
if (fn_cen != null) {
file = new File(fn_cen);
while (file.readline(buf) >= 0) {
var t = buf.toString().split("\t");
if (cen[t[0]] == null) cen[t[0]] = [];
cen[t[0]].push([parseInt(t[1]), parseInt(t[2])]);
}
file.close();
}
function test_cen(cen, chr, st, en) {
var b = cen[chr], len = 0;
if (b == null) return false;
for (var j = 0; j < b.length; ++j)
if (b[j][0] < en && b[j][1] > st) {
var s = b[j][0] > st? b[j][0] : st;
var e = b[j][1] < en? b[j][1] : en;
len += e - s;
}
return len < (en - st) * cen_ratio? false : true;
}
function process(a) {
var k = 0;
for (var i = 0; i < a.length; ++i) {
for (var j = 1; j <= 3; ++j) a[i][j] = parseInt(a[i][j]);
for (var j = 6; j <= 11; ++j) a[i][j] = parseInt(a[i][j]);
if (a[i][10] >= min_seg_len) a[k++] = a[i];
}
a.length = k;
if (a.length == 1) return;
a = a.sort(function(x,y){return x[2]-y[2]});
if (show_long) for (var i = 0; i < a.length; ++i) print(a[i].join("\t"));
for (var i = 1; i < a.length; ++i) {
var ov = [false, false];
ov[0] = test_cen(cen, a[i-1][5], a[i-1][7], a[i-1][8]);
ov[1] = test_cen(cen, a[i][5], a[i][7], a[i][8]);
if (a[i-1][5] != a[i][5]) { // different chr
if (ov[0] || ov[1]) ++n_diff[1];
else if (show_err) {
print("J", a[i-1].slice(0, 12).join("\t"));
print("J", a[i].slice(0, 12).join("\t"));
}
++n_diff[0];
} else if (a[i-1][4] == a[i][4]) { // a gap
var dq = a[i][2] - a[i-1][3];
var dr = a[i][4] == '+'? a[i][7] - a[i-1][8] : a[i-1][7] - a[i][8];
var gap = dr > dq? dr - dq : dq - dr;
if (gap > max_gap) {
if (ov[0] || ov[1]) ++n_gap[1];
else if (show_err) {
print("G", a[i-1].slice(0, 12).join("\t"));
print("G", a[i].slice(0, 12).join("\t"));
}
++n_gap[0];
}
} else if (i + 1 < a.length && a[i+1][4] == a[i-1][4]) { // bracketed inversion
if (ov[0] || ov[1]) ++n_inv[1];
else if (show_err) {
print("M", a[i-1].slice(0, 12).join("\t"));
print("M", a[i].slice(0, 12).join("\t"));
print("M", a[i+1].slice(0, 12).join("\t"));
}
++n_inv[0];
++i;
} else { // hanging inversion
if (ov[0] || ov[1]) ++n_inv_end[1];
++n_inv_end[0];
}
}
}
file = args[getopt.ind] == "-"? new File() : new File(args[getopt.ind]);
var a = [];
while (file.readline(buf) >= 0) {
var t = buf.toString().split("\t");
if (a.length > 0 && a[0][0] != t[0]) {
process(a);
a.length = 0;
}
a.push(t);
}
if (a.length > 0) process(a);
file.close();
buf.destroy();
print("# inter-chromosomal misjoins: " + n_diff.join(","));
print("# intra-chromosomal gaps: " + n_gap.join(","));
print("# candidate inversions in the middle: " + n_inv.join(","));
print("# candidate inversions at contig ends: " + n_inv_end.join(","));
}
function paf_sveval(args)
{
var c, min_flt = 30, min_size = 50, max_size = 10000, win_size = 500, print_err = false, bed_fn = null;
while ((c = getopt(args, "f:i:x:w:er:")) != null) {
if (c == 'f') min_flt = paf_parseNum(getopt.arg);
else if (c == 'i') min_size = paf_parseNum(getopt.arg);
else if (c == 'x') max_size = paf_parseNum(getopt.arg);
else if (c == 'w') win_size = paf_parseNum(getopt.arg);
else if (c == 'r') bed_fn = getopt.arg;
else if (c == 'e') print_err = true;
}
if (args.length - getopt.ind < 2) {
print("Usage: paftools.js sveval [options] <base.vcf> <call.vcf>");
print("Options:");
print(" -r FILE confident region in BED []");
print(" -f INT min length to discard [" + min_flt + "]");
print(" -i INT min SV length [" + min_size + "]");
print(" -x INT max SV length [" + max_size + "]");
print(" -w INT fuzzy windown size [" + win_size + "]");
print(" -e print errors");
return;
}
function read_bed(fn) {
var buf = new Bytes();
var file = new File(fn);
var bed = {};
while (file.readline(buf) >= 0) {
var t = buf.toString().split("\t");
if (bed[t[0]] == null) bed[t[0]] = [];
bed[t[0]].push([parseInt(t[1]), parseInt(t[2])]);
}
file.close();
buf.destroy();
for (var x in bed) {
Interval.sort(bed[x]);
Interval.merge(bed[x]);
Interval.index_end(bed[x]);
}
return bed;
}
var bed = bed_fn != null? read_bed(bed_fn) : null;
function read_vcf(fn, bed) {
var buf = new Bytes();
var file = new File(fn);
var v = {};
while (file.readline(buf) >= 0) {
var m, t = buf.toString().split("\t");
if (t[0][0] == '#') continue;
if (bed != null && bed[t[0]] == null) continue;
if (t[4] == '<INV>' || t[4] == '<INVDUP>') continue; // no inversion
if (/[\[\]]/.test(t[4])) continue; // no break points
var st = parseInt(t[1]) - 1, en = st + t[3].length;
if ((m = /((;END)|(^END))=(\d+)/.exec(t[7])) != null)
en = parseInt(m[4]);
if (bed != null && Interval.find_ovlp(bed[t[0]], st, en).length == 0) continue;
// determine svlen
var s = t[4].split(","), max_del = 0, max_ins = 0;
for (var i = 0; i < s.length; ++i) {
var l = s[i].length - t[3].length;
if (l > 0)
max_ins = max_ins > l? max_ins : l;
else if (l < 0)
max_del = max_del > -l? max_del : -l;
}
if (max_ins < min_flt && max_del < min_flt) continue;
var svlen = max_ins > max_del? max_ins : -max_del;
if ((m = /((;SVLEN)|(^SVLEN))=(\d+)/.exec(t[7])) != null)
svlen = parseInt(m[4]);
var abslen = svlen > 0? svlen : -svlen;
if (abslen < min_flt || abslen > max_size) continue;
// insert
if (v[t[0]] == null) v[t[0]] = [];
v[t[0]].push([st, en, svlen, abslen]);
}
file.close();
buf.destroy();
for (var x in v) {
Interval.sort(v[x]);
Interval.index_end(v[x]);
}
return v;
}
function compare_vcf(v0, v1, label) {
var m = 0, n = 0;
for (var x in v1) {
var a1 = v1[x], a0 = v0[x];
for (var i = 0; i < a1.length; ++i) {
if (a1[i][3] < min_size) continue;
++n;
if (a0 == null) continue;
var st = a1[i][0] > win_size? a1[i][0] - win_size : 0;
b = Interval.find_ovlp(a0, st, a1[i][1] + win_size);
if (b.length > 0) ++m;
else if (print_err) print(label, x, a1[i].slice(0, 3).join("\t"));
}
}
return [n, m];
}
var v_base = read_vcf(args[getopt.ind+0], bed);
var v_call = read_vcf(args[getopt.ind+1], bed);
var fn = compare_vcf(v_call, v_base, 'FN');
var fp = compare_vcf(v_base, v_call, 'FP');
print('SN', fn[0], fn[1], (fn[1] / fn[0]).toFixed(6));
print('PC', fp[0], fp[1], (fp[1] / fp[0]).toFixed(6));
print('F1', ((fn[1] / fn[0] + fp[1] / fp[0]) / 2).toFixed(6));
}
/*************************
***** main function *****
*************************/
@@ -2496,10 +2847,13 @@ function main(args)
print("");
print(" stat collect basic mapping information in PAF/SAM");
print(" asmstat collect basic assembly information");
print(" asmgene evaluate gene completeness (EXPERIMENTAL)");
print(" asmgene evaluate gene completeness");
print(" misjoin evaluate large-scale misjoins");
print(" liftover simplistic liftOver");
print(" call call variants from asm-to-ref alignment with the cs tag");
print(" bedcov compute the number of bases covered");
print(" vcfstat VCF statistics");
print(" sveval compare two SV callsets in VCF");
print(" version print paftools.js version");
print("");
print(" mapeval evaluate mapping accuracy using mason2/PBSIM-simulated FASTQ");
@@ -2519,6 +2873,7 @@ function main(args)
else if (cmd == 'stat') paf_stat(args);
else if (cmd == 'asmstat') paf_asmstat(args);
else if (cmd == 'asmgene') paf_asmgene(args);
else if (cmd == 'misjoin') paf_misjoin(args);
else if (cmd == 'liftover' || cmd == 'liftOver') paf_liftover(args);
else if (cmd == 'vcfpair') paf_vcfpair(args);
else if (cmd == 'call') paf_call(args);
@@ -2528,6 +2883,8 @@ function main(args)
else if (cmd == 'pbsim2fq') paf_pbsim2fq(args);
else if (cmd == 'junceval') paf_junceval(args);
else if (cmd == 'ov-eval') paf_ov_eval(args);
else if (cmd == 'vcfstat') paf_vcfstat(args);
else if (cmd == 'sveval') paf_sveval(args);
else if (cmd == 'version') print(paftools_version);
else throw Error("unrecognized command: " + cmd);
}
+6 -12
View File
@@ -4,6 +4,7 @@
#include <assert.h>
#include "minimap.h"
#include "bseq.h"
#include "kseq.h"
#define MM_PARENT_UNSET (-1)
#define MM_PARENT_TMP_PRI (-2)
@@ -35,14 +36,6 @@
extern "C" {
#endif
#ifndef KSTRING_T
#define KSTRING_T kstring_t
typedef struct __kstring_t {
unsigned l, m;
char *s;
} kstring_t;
#endif
typedef struct {
int n_u, n_a;
uint64_t *u;
@@ -59,7 +52,7 @@ uint32_t ks_ksmall_uint32_t(size_t n, uint32_t arr[], size_t kk);
void mm_sketch(void *km, const char *str, int len, int w, int k, uint32_t rid, int is_hpc, mm128_v *p);
void mm_write_sam_hdr(const mm_idx_t *mi, const char *rg, const char *ver, int argc, char *argv[]);
int mm_write_sam_hdr(const mm_idx_t *mi, const char *rg, const char *ver, int argc, char *argv[]);
void mm_write_paf(kstring_t *s, const mm_idx_t *mi, const mm_bseq1_t *t, const mm_reg1_t *r, void *km, int opt_flag);
void mm_write_paf3(kstring_t *s, const mm_idx_t *mi, const mm_bseq1_t *t, const mm_reg1_t *r, void *km, int opt_flag, int rep_len);
void mm_write_sam(kstring_t *s, const mm_idx_t *mi, const mm_bseq1_t *t, const mm_reg1_t *r, int n_regs, const mm_reg1_t *regs);
@@ -69,20 +62,21 @@ void mm_write_sam3(kstring_t *s, const mm_idx_t *mi, const mm_bseq1_t *t, int se
void mm_idxopt_init(mm_idxopt_t *opt);
const uint64_t *mm_idx_get(const mm_idx_t *mi, uint64_t minier, int *n);
int32_t mm_idx_cal_max_occ(const mm_idx_t *mi, float f);
mm128_t *mm_chain_dp(int max_dist_x, int max_dist_y, int bw, int max_skip, int max_iter, int min_cnt, int min_sc, int is_cdna, int n_segs, int64_t n, mm128_t *a, int *n_u_, uint64_t **_u, void *km);
mm128_t *mm_chain_dp(int max_dist_x, int max_dist_y, int bw, int max_skip, int max_iter, int min_cnt, int min_sc, float gap_scale, int is_cdna, int n_segs, int64_t n, mm128_t *a, int *n_u_, uint64_t **_u, void *km);
mm_reg1_t *mm_align_skeleton(void *km, const mm_mapopt_t *opt, const mm_idx_t *mi, int qlen, const char *qstr, int *n_regs_, mm_reg1_t *regs, mm128_t *a);
mm_reg1_t *mm_gen_regs(void *km, uint32_t hash, int qlen, int n_u, uint64_t *u, mm128_t *a);
void mm_mark_alt(const mm_idx_t *mi, int n, mm_reg1_t *r);
void mm_split_reg(mm_reg1_t *r, mm_reg1_t *r2, int n, int qlen, mm128_t *a);
void mm_sync_regs(void *km, int n_regs, mm_reg1_t *regs);
int mm_squeeze_a(void *km, int n_regs, mm_reg1_t *regs, mm128_t *a);
int mm_set_sam_pri(int n, mm_reg1_t *r);
void mm_set_parent(void *km, float mask_level, int n, mm_reg1_t *r, int sub_diff, int hard_mask_level);
void mm_set_parent(void *km, float mask_level, int mask_len, int n, mm_reg1_t *r, int sub_diff, int hard_mask_level, float alt_diff_frac);
void mm_select_sub(void *km, float pri_ratio, int min_diff, int best_n, int *n_, mm_reg1_t *r);
void mm_select_sub_multi(void *km, float pri_ratio, float pri1, float pri2, int max_gap_ref, int min_diff, int best_n, int n_segs, const int *qlens, int *n_, mm_reg1_t *r);
void mm_filter_regs(const mm_mapopt_t *opt, int qlen, int *n_regs, mm_reg1_t *regs);
void mm_join_long(void *km, const mm_mapopt_t *opt, int qlen, int *n_regs, mm_reg1_t *regs, mm128_t *a);
void mm_hit_sort(void *km, int *n_regs, mm_reg1_t *r);
void mm_hit_sort(void *km, int *n_regs, mm_reg1_t *r, float alt_diff_frac);
void mm_set_mapq(void *km, int n_regs, mm_reg1_t *regs, int min_chain_sc, int match_sc, int rep_len, int is_sr);
void mm_est_err(const mm_idx_t *mi, int qlen, int n_regs, mm_reg1_t *regs, const mm128_t *a, int32_t n, const uint64_t *mini_pos);
+5
View File
@@ -1,4 +1,5 @@
#include <stdio.h>
#include <limits.h>
#include "mmpriv.h"
void mm_idxopt_init(mm_idxopt_t *opt)
@@ -24,8 +25,10 @@ void mm_mapopt_init(mm_mapopt_t *opt)
opt->max_gap_ref = -1;
opt->max_chain_skip = 25;
opt->max_chain_iter = 5000;
opt->chain_gap_scale = 1.0f;
opt->mask_level = 0.5f;
opt->mask_len = INT_MAX;
opt->pri_ratio = 0.8f;
opt->best_n = 5;
@@ -34,6 +37,8 @@ void mm_mapopt_init(mm_mapopt_t *opt)
opt->min_join_flank_sc = 1000;
opt->min_join_flank_ratio = 0.5f;
opt->alt_drop = 0.15f;
opt->a = 2, opt->b = 4, opt->q = 4, opt->e = 2, opt->q2 = 24, opt->e2 = 1;
opt->sc_ambi = 1;
opt->zdrop = 400, opt->zdrop_inv = 200;
+5 -2
View File
@@ -6,7 +6,7 @@ cdef extern from "minimap.h":
#
ctypedef struct mm_idxopt_t:
short k, w, flag, bucket_bits
int mini_batch_size
int64_t mini_batch_size
uint64_t batch_size
ctypedef struct mm_mapopt_t:
@@ -20,12 +20,15 @@ cdef extern from "minimap.h":
int max_chain_skip, max_chain_iter
int min_cnt
int min_chain_score
float chain_gap_scale
float mask_level
int mask_len
float pri_ratio
int best_n
int max_join_long, max_join_short
int min_join_flank_sc
float min_join_flank_ratio
float alt_drop
int a, b, q, e, q2, e2
int sc_ambi
int noncan
@@ -41,7 +44,7 @@ cdef extern from "minimap.h":
int32_t min_mid_occ
int32_t mid_occ
int32_t max_occ
int mini_batch_size
int64_t mini_batch_size
int64_t max_sw_mat
const char *split_prefix
+31 -19
View File
@@ -3,7 +3,7 @@ from libc.stdlib cimport free
cimport cmappy
import sys
__version__ = '2.17'
__version__ = '2.18'
cmappy.mm_reset_timer()
@@ -113,6 +113,7 @@ cdef class Aligner:
cdef cmappy.mm_mapopt_t map_opt
def __cinit__(self, fn_idx_in=None, preset=None, k=None, w=None, min_cnt=None, min_chain_score=None, min_dp_score=None, bw=None, best_n=None, n_threads=3, fn_idx_out=None, max_frag_len=None, extra_flags=None, seq=None, scoring=None):
self._idx = NULL
cmappy.mm_set_opt(NULL, &self.idx_opt, &self.map_opt) # set the default options
if preset is not None:
cmappy.mm_set_opt(str.encode(preset), &self.idx_opt, &self.map_opt) # apply preset
@@ -170,6 +171,7 @@ cdef class Aligner:
cdef void *km
cdef cmappy.mm_mapopt_t map_opt
if self._idx == NULL: return
map_opt = self.map_opt
if max_frag_len is not None: map_opt.max_frag_len = max_frag_len
if extra_flags is not None: map_opt.flag |= extra_flags
@@ -186,27 +188,36 @@ cdef class Aligner:
_seq2 = seq2 if isinstance(seq2, bytes) else seq2.encode()
regs = cmappy.mm_map_aux(self._idx, _seq, _seq2, &n_regs, b._b, &map_opt)
for i in range(n_regs):
cmappy.mm_reg2hitpy(self._idx, &regs[i], &h)
cigar, _cs, _MD = [], '', ''
for k in range(h.n_cigar32): # convert the 32-bit CIGAR encoding to Python array
c = h.cigar32[k]
cigar.append([c>>4, c&0xf])
if cs or MD: # generate the cs and/or the MD tag, if requested
if cs:
l_cs_str = cmappy.mm_gen_cs(km, &cs_str, &m_cs_str, self._idx, &regs[i], _seq, 1)
_cs = cs_str[:l_cs_str] if isinstance(cs_str, str) else cs_str[:l_cs_str].decode()
if MD:
l_cs_str = cmappy.mm_gen_MD(km, &cs_str, &m_cs_str, self._idx, &regs[i], _seq)
_MD = cs_str[:l_cs_str] if isinstance(cs_str, str) else cs_str[:l_cs_str].decode()
yield Alignment(h.ctg, h.ctg_len, h.ctg_start, h.ctg_end, h.strand, h.qry_start, h.qry_end, h.mapq, cigar, h.is_primary, h.mlen, h.blen, h.NM, h.trans_strand, h.seg_id, _cs, _MD)
cmappy.mm_free_reg1(&regs[i])
free(regs)
free(cs_str)
try:
i = 0
while i < n_regs:
cmappy.mm_reg2hitpy(self._idx, &regs[i], &h)
cigar, _cs, _MD = [], '', ''
for k in range(h.n_cigar32): # convert the 32-bit CIGAR encoding to Python array
c = h.cigar32[k]
cigar.append([c>>4, c&0xf])
if cs or MD: # generate the cs and/or the MD tag, if requested
if cs:
l_cs_str = cmappy.mm_gen_cs(km, &cs_str, &m_cs_str, self._idx, &regs[i], _seq, 1)
_cs = cs_str[:l_cs_str] if isinstance(cs_str, str) else cs_str[:l_cs_str].decode()
if MD:
l_cs_str = cmappy.mm_gen_MD(km, &cs_str, &m_cs_str, self._idx, &regs[i], _seq)
_MD = cs_str[:l_cs_str] if isinstance(cs_str, str) else cs_str[:l_cs_str].decode()
yield Alignment(h.ctg, h.ctg_len, h.ctg_start, h.ctg_end, h.strand, h.qry_start, h.qry_end, h.mapq, cigar, h.is_primary, h.mlen, h.blen, h.NM, h.trans_strand, h.seg_id, _cs, _MD)
cmappy.mm_free_reg1(&regs[i])
i += 1
finally:
while i < n_regs:
cmappy.mm_free_reg1(&regs[i])
i += 1
free(regs)
free(cs_str)
def seq(self, str name, int start=0, int end=0x7fffffff):
cdef int l
cdef char *s = cmappy.mappy_fetch_seq(self._idx, name.encode(), start, end, &l)
cdef char *s
if self._idx == NULL: return
s = cmappy.mappy_fetch_seq(self._idx, name.encode(), start, end, &l)
if l == 0: return None
r = s[:l] if isinstance(s, str) else s[:l].decode()
free(s)
@@ -224,6 +235,7 @@ cdef class Aligner:
@property
def seq_names(self):
cdef char *p
if self._idx == NULL: return
sn = []
for i in range(self._idx.n_seq):
p = self._idx.seq[i].name
+4 -14
View File
@@ -4,16 +4,6 @@ except ImportError:
from distutils.core import setup
from distutils.extension import Extension
cmdclass = {}
try:
from Cython.Build import build_ext
except ImportError: # without Cython
module_src = 'python/mappy.c'
else: # with Cython
module_src = 'python/mappy.pyx'
cmdclass['build_ext'] = build_ext
import sys, platform
sys.path.append('python')
@@ -33,7 +23,7 @@ def readme():
setup(
name = 'mappy',
version = '2.17',
version = '2.18',
url = 'https://github.com/lh3/minimap2',
description = 'Minimap2 python binding',
long_description = readme(),
@@ -42,8 +32,8 @@ setup(
license = 'MIT',
keywords = 'sequence-alignment',
scripts = ['python/minimap2.py'],
ext_modules = [Extension('mappy',
sources = [module_src, 'align.c', 'bseq.c', 'chain.c', 'format.c', 'hit.c', 'index.c', 'pe.c', 'options.c',
ext_modules = [Extension('mappy',
sources = ['python/mappy.pyx', 'align.c', 'bseq.c', 'chain.c', 'format.c', 'hit.c', 'index.c', 'pe.c', 'options.c',
'ksw2_extd2_sse.c', 'ksw2_exts2_sse.c', 'ksw2_extz2_sse.c', 'ksw2_ll_sse.c',
'kalloc.c', 'kthread.c', 'map.c', 'misc.c', 'sdust.c', 'sketch.c', 'esterr.c', 'splitidx.c'],
depends = ['minimap.h', 'bseq.h', 'kalloc.h', 'kdq.h', 'khash.h', 'kseq.h', 'ksort.h',
@@ -62,4 +52,4 @@ setup(
'Programming Language :: Python :: 3',
'Intended Audience :: Science/Research',
'Topic :: Scientific/Engineering :: Bio-Informatics'],
cmdclass = cmdclass)
setup_requires=["cython"])
+7 -6
View File
@@ -2,6 +2,7 @@
#include <assert.h>
#include <stdlib.h>
#include <stdio.h>
#include <errno.h>
#include "mmpriv.h"
FILE *mm_split_init(const char *prefix, const mm_idx_t *mi)
@@ -13,15 +14,15 @@ FILE *mm_split_init(const char *prefix, const mm_idx_t *mi)
sprintf(fn, "%s.%.4d.tmp", prefix, mi->index);
if ((fp = fopen(fn, "wb")) == NULL) {
if (mm_verbose >= 1)
fprintf(stderr, "[ERROR]\033[1;31m failed to write to temporary file '%s'\033[0m\n", fn);
fprintf(stderr, "[ERROR]\033[1;31m failed to write to temporary file '%s'\033[0m: %s\n", fn, strerror(errno));
exit(1);
}
mm_err_fwrite(&k, 4, 1, fp);
mm_err_fwrite(&mi->n_seq, 4, 1, fp);
for (i = 0; i < mi->n_seq; ++i) {
uint8_t l;
uint32_t l;
l = strlen(mi->seq[i].name);
mm_err_fwrite(&l, 1, 1, fp);
mm_err_fwrite(&l, 1, 4, fp);
mm_err_fwrite(mi->seq[i].name, 1, l, fp);
mm_err_fwrite(&mi->seq[i].len, 4, 1, fp);
}
@@ -41,7 +42,7 @@ mm_idx_t *mm_split_merge_prep(const char *prefix, int n_splits, FILE **fp, uint3
sprintf(fn, "%s.%.4d.tmp", prefix, i);
if ((fp[i] = fopen(fn, "rb")) == 0) {
if (mm_verbose >= 1)
fprintf(stderr, "ERROR: failed to open temporary file '%s'\n", fn);
fprintf(stderr, "ERROR: failed to open temporary file '%s': %s\n", fn, strerror(errno));
for (j = 0; j < i; ++j)
fclose(fp[j]);
free(fn);
@@ -60,8 +61,8 @@ mm_idx_t *mm_split_merge_prep(const char *prefix, int n_splits, FILE **fp, uint3
for (i = j = 0; i < n_splits; ++i) {
uint32_t k;
for (k = 0; k < n_seq_part[i]; ++k, ++j) {
uint8_t l;
mm_err_fread(&l, 1, 1, fp[i]);
uint32_t l;
mm_err_fread(&l, 1, 4, fp[i]);
mi->seq[j].name = (char*)calloc(l + 1, 1);
mm_err_fread(mi->seq[j].name, 1, l, fp[i]);
mm_err_fread(&mi->seq[j].len, 4, 1, fp[i]);