Annotating Genomes via Unsupervised Deep Learning

Vipsania, an unsupervised deep learning ab-initio gene finder from the Gaius-Augustus project, annotates eukaryotic genomes from a FASTA file alone without any reference annotation, curated gene set, or protein evidence. The tool trains purely on raw genomic sequences with a masked language modelling objective, requires python>=3.12, and outputs annotations in GenePred, GFF3, or GTF; it is installable via pip or pre-built Docker and Singularity images based on NVIDIA's NGC TensorFlow container (TensorFlow 2.17 with CUDA 12.8), which need only an NVIDIA driver ≥ 570 for GPU runs. A short finetuning run on the target genome before annotating gives the best results, and models download at roughly 100 MB on first use.

Vipsania is an unsupervised deep learning ab-initio gene finder. Unlike other deep learning gene finders, Vipsania is never shown a reference annotation. It is trained purely on raw genomic sequences with a masked language modelling objective: nucleotides are hidden and the model learns to predict them from their context. Gene structure emerges as the representation a hidden Markov model inside the network needs in order to explain the sequence, so no curated gene set, no related-species annotation and no protein evidence are required at any point. For you, this means: - Annotate any eukaryotic genome from a FASTA file alone. No extrinsic evidence, no reference annotation, no gene-structure training data of any kind. - Adapt the model to your species on the fly. Because training needs nothing but sequence, the genome you want to annotate is itself valid training data. A short finetuning run on your target genome directly before annotating gives the best results — see Finetuning finetuning-recommended . - Get standard output. Annotations are written as GenePred, GFF3 or GTF. Vipsania requires python =3.12 . It is recommended to install into a clean virtual environment e.g. using conda , pyenv or uv . bash $ python -m pip install vipsania This installs the vipsania command, which is everything you need to annotate a genome. For training, use the config files https://github.com/Gaius-Augustus/Vipsania/blob/main/configs provided and follow the instructions here training . Alternatively, you can clone the repository, install Vipsania from source and use the scripts https://github.com/Gaius-Augustus/Vipsania/blob/main/scripts for annotation and training. bash $ git clone https://github.com/gaius-augustus/vipsania $ python -m pip install -e vipsania/ Installing Vipsania pulls in bricks2marble https://github.com/gaius-augustus/bricks2marble and hidten https://github.com/gaius-augustus/hidten , which in turn install TensorFlow with CUDA support. A GPU is strongly recommended; annotating a large genome on CPU only is possible, but slow. If you prefer not to install Python dependencies locally — or if you are working on an HPC cluster — use the pre-built image from Docker Hub: Docker requires sudo unless your user is in the docker group : bash $ sudo docker pull gaiusaugustus/vipsania:latest $ sudo docker run --rm --gpus all \ -v /path/to/data:/data \ gaiusaugustus/vipsania \ vipsania annotate Fungi genome.fa -o annotation.gff3 --finetune Singularity / Apptainer no root needed, suited for HPC : bash $ singularity pull vipsania.sif docker://gaiusaugustus/vipsania:latest $ singularity exec --nv \ -B /path/to/data:/data \ vipsania.sif \ vipsania annotate Fungi /data/genome.fa -o /data/annotation.gff3 --finetune Drop --gpus all Docker or --nv Singularity for CPU-only runs. The image is based on NVIDIA's NGC TensorFlow container TensorFlow 2.17 with CUDA 12.8 , so no CUDA installation on the host is required — only an NVIDIA driver ≥ 570 for GPU runs. The image is linux/amd64 only and also supports Blackwell GPUs e.g. RTX PRO 6000, B200, RTX 50xx . Full details, including supported systems, model-cache persistence and training, are in docs/container.md https://github.com/Gaius-Augustus/Vipsania/blob/main/docs/container.md . Annotation is done with vipsania annotate . It needs a trained model and a FASTA file of the genome you want to annotate. bash $ vipsania annotate <model id genome.fa -o annotation.gff3 --finetune The same command is available as python -m vipsania annotate ... , and from a cloned repository as python scripts/annotate.py ... . The first argument names the model to use, either by the clade it was trained for or by its model ID , both listed below. The first time a model is used, Vipsania downloads it about 100 MB and keeps it, so every later annotation with it starts immediately. bash $ vipsania annotate Insecta genome.fa -o annotation.gff3 --finetune A clade name gives you the model we recommend for that clade at the moment. Should a clade get a better model later, vipsania download <clade picks it up; annotating on its own keeps using the model it already has. A model ID always refers to that one model. A model you trained yourself is given in the same way: its ID is the name of its run folder, which Vipsania looks for in the directories around the working directory. Set --model dir to use a different directory that holds the model folder directly. Progress is printed to the terminal while the genome is annotated, together with a log file next to the output. Vipsania is trained per clade. Use the model of the clade your species belongs to, choosing the most specific one available. | Clade | Model ID | Training species | Test species | Average locus F1 | Average locus F1 pretrained | |---|---|---|---|---|---| | Alveolata | sd2zcj7u | 30 | 6 | 0.613 | 0.582 | | Amoebozoa | ezhpj2qm | 15 | 7 | 0.437 | 0.416 | | Arthropoda | ihe1jk30 | 200 | 6 | 0.207 | 0.217 | | Chlorophyta | faeijtmk | 26 | 6 | 0.579 | 0.558 | | Cnidaria | b5vtieo0 | 82 | 10 | 0.453 | 0.406 | | Discoba | gcra9d9y | 20 | 4 | 0.696 | 0.696 | | Echinodermata | sx9zjl7p | 52 | 8 | 0.444 | 0.440 | | Fungi | fh1kg88z | 200 | 7 | 0.660 | 0.645 | | Insecta | 58hsuobw | 200 | 9 | 0.537 | 0.474 | | Nematoda | zspca2rb | 73 | 8 | 0.539 | 0.517 | | Porifera | 7bjiexcu | 75 | 6 | 0.480 | 0.416 | | Rhodophyta | 6kmw3wme | 20 | 5 | 0.405 | 0.301 | | Spiralia | v5ej8oyt | 200 | 6 | 0.374 | 0.262 | | Stramenopiles | r6p9z9jw | 32 | 6 | 0.426 | 0.413 | | Streptophyta | j9m0cdmk | 200 | 13 | 0.602 | 0.579 | | Tunicata | hcehc7ff | 34 | 6 | 0.523 | 0.494 | | Vertebrata | etb1go6q | 200 | 13 | 0.406 | 0.327 | | none of the above | cg6grhms | 33 | 7 | 0.436 | 0.416 | Use the last entry for eukaryotes that belong to none of the clades above — it was trained on exactly such species. It has no clade name of its own; pass Other or its model ID. The two F1 columns are the average locus F1 over the test species of that clade, measured against their reference annotations: the first with --finetune , the second with the pretrained model alone. Finetuning helps in almost every clade, which is why we recommend it. Whichever model you pick, finetuning it on your genome adapts it to your species — see Finetuning finetuning-recommended . The species every model was trained on are listed in docs/training species.tsv https://github.com/Gaius-Augustus/Vipsania/blob/main/docs/training species.tsv , one row per species with its NCBI assembly accession. Since Vipsania never learns from annotations, the quality of a species' reference annotation, or whether it has one at all, leaves no trace in the model, and unlike for a supervised gene finder, finding your species or a close relative in the list is not an argument against that model. The test species behind the two F1 columns are in docs/test species.tsv https://github.com/Gaius-Augustus/Vipsania/blob/main/docs/test species.tsv , with precision, sensitivity and F1 at base, exon and locus level for each of them, once with and once without finetuning. The first time a model ID is used it is downloaded into ~/.cache/vipsania/models and read from there ever after. See docs/download.md https://github.com/Gaius-Augustus/Vipsania/blob/main/docs/download.md for changing that location, and for fetching models in advance on machines that have no internet access when they annotate. The output format is chosen by the file suffix of -o/--output : | suffix | format | |---|---| | .gp | GenePred | | .gff3 / .gff | GFF3 | | .gtf | GTF | If -o is omitted, the annotation is written next to the input FASTA as vipsania model id .gff , named after the model that produced it even when you asked for it by clade. The protein and coding sequences of the predicted genes can be written out at the same time with --protein proteins.fa and --coding coding.fa . bash $ vipsania annotate <model id genome.fa -o annotation.gff3 --finetune --finetune trains the model on the genome you are about to annotate before predicting anything. Since Vipsania is unsupervised, this needs nothing except the FASTA file that you already have. The model adapts to the codon usage, repeat landscape and intron statistics of your species, which consistently improves the annotation. The finetuned checkpoint is saved next to the output file, so you can reuse it for further annotations of the same genome or closely related species without finetuning again. Use the --model dir argument in these cases. All options of the annotation are listed in docs/annotate.md https://github.com/Gaius-Augustus/Vipsania/blob/main/docs/annotate.md , or with vipsania annotate --help . Paludamentum https://github.com/Gaius-Augustus/Paludamentum is the Nextflow pipeline around the gene finders of the Gaius-Augustus family. With Vipsania as gene finder it splits the genome, runs vipsania annotate on several GPUs in parallel, and, if you give it proteins, RNA-Seq or Iso-Seq, derives high-confidence genes from that evidence and merges them with the Vipsania prediction. Vipsania is a git submodule of Paludamentum; the vipsania command itself does not run the pipeline. bash $ git clone --recursive https://github.com/Gaius-Augustus/Paludamentum $ cd Paludamentum && pip install . $ paludamentum --genefinder vipsania --nf config slurm generic --genome genome.fa --model Fungi \ --proteins proteins.faa Finetuning is off in the pipeline by default; --finetune switches it on and then runs one Vipsania task on the whole genome. See docs/pipeline.md https://github.com/Gaius-Augustus/Vipsania/blob/main/docs/pipeline.md . Vipsania models are trained with vipsania train on a set of local FASTA files. Ready-to-use configurations for the three model sizes are in configs https://github.com/Gaius-Augustus/Vipsania/blob/main/configs ; rather than editing one, you list the FASTA files you want to train on in configs/train.json https://github.com/Gaius-Augustus/Vipsania/blob/main/configs/train.json and pass it as an override: bash $ vipsania train configs/base 10M.json -oc configs/train.json Training runs can be logged to Weights & Biases https://wandb.ai with --online entity/project . Everything a run produces — configuration, checkpoints and metrics — is written to a folder below ./checkpoints , named after the run. That name is exactly the model ID that vipsania annotate expects. A published model can also be trained further on species of your choice, which extends the --finetune option of vipsania annotate to a set of genomes. See docs/training.md https://github.com/Gaius-Augustus/Vipsania/blob/main/docs/training.md for that, for the configuration format, and for what to change when a run does not fit into GPU memory. Vipsania is built on two libraries developed in our group: - bricks2marble https://github.com/gaius-augustus/bricks2marble — efficient handling of nucleotide sequences and genome annotations, plus the pre- and postprocessing around genome annotation. - hidten https://github.com/gaius-augustus/hidten-docs — hidden Markov models as differentiable, highly parallel layers inside deep learning models.