百科.dev
全部条目在线工具AI 编程趋势榜开源项目技术资讯提交条目
登录
< 返回工具列表
G

genomics_general

> 编程语言
开源

基因组分析的通用工具。

394 stars0 点赞1 次浏览
访问官网GitHub

工具介绍

基因组分析的通用工具。

This is a collection of scripts for a range of genomic data processing and analysis.

Below are notes about some useful tools. Not everything is documented yet, but most scripts have some help information if you type python script.py -h

Contents

  • Installation and dependencies
  • Processing VCF files
  • Filtering genotype files prior to further analysis
  • Diversity and divergence analyses in sliding windows
  • Allele frequencies per site
  • Site frequency spectrum
  • Distance matrix
  • ABBA-BABA statistics in sliding windows
  • Trees for sliding windows
  • Classify coding sites

Installation and dependencies

There is no system installation required. Just download this entire repository using the green "Code" button at the top of this page, or with the bash command git clone https://github.com/simonhmartin/genomics_general.git. If you prefer to move scripts to the directory where you run them, bear in mind that many of these scripts require the script genomics.py to be present in the same directory (or on your PYTHONPATH). The easiest is just to leave them in the main directory and specify the full path to the script when running it from elsewhere.

The only dependency is numpy.

Most of the scripts now run in python 3. Some are still written for puthon 2, but those will be updated soon.


Processing VCF files

Most of my scripts use a processed .vcf format that I call .geno. This looks something like this:

#CHROM      POS      ind1      ind2      ind3
scaffold1   1        A/A       A/G       G|A
scaffold1   1        N/N       T/T       T|C

Missing data is denoted as N, and phased and unphased genotypes are shown conventionally with | and /.

The script parseVCF.py in the VCF_processing directory, will convert vcf to this format. It has various options for filtering based on read depth, genotype quality or any other flag in the FORMAT column of the vcf.

Example command

bash
python VCF_processing/parseVCF.py -i input.vcf.gz --skipIndels --minQual 30 --gtf flag=DP min=5 max=50 -o output.geno.gz

You can read more about this script in the VCF_processing directory.


Filtering genotype files prior to further analysis

If you vcf file was not already filtered, or you would like to filter further. The script filtergenotypes.py has many options for filtering. Some examples include:

  • Number of individuals with non-missing N/N genotypes at a site (--minCalls)
  • Number of alleles observed at a site across all individuals (--minAlleles and --maxAlleles)
  • Minor allele count (--minVarCount)
  • Distance between sites for thinning (--thinDist)

It requires the script genomics.py to be present in the same directory, or in your Python path.

Example command

bash
python filterGenotypes.py --threads 4 -i input.geno.gz -o output.geno.gz --minAlleles 2 --minCalls 10 --thinDist 1000

Notes

python filterGenotypes.py -h will print a full list of command options.

By default, this script assumes that the .geno input file is is encoded as diploid genotypes with a phase operator (/ or |):

scaffold1  1        A/A       G/G       G|A

You can specify a different input formats using the -if, but this is not recommended.

You can also specify various putput formats using -of.

Output format Description Example
phased (default) Alleles separates by a phase operator. This doesn't mean the phase is known, just that it is indicated A/A G/G G|A
diplo For diploids only. Genotypes are single bases denoting the diploid genotype, using ambiguity codes for heterozygotes A G R
alleles as above but without the phase operator AA GG GA
randomAllele Randomly pick one allele per individual A G A
coded Coded numerically as in the VCF 0/0 1/1 1|0
bases Separate the alleles for each individual into different columns and also give different headers for each A A G G G A
counts Count of the minor allele in each individual 0 2 1

Diversity and divergence analyses in sliding windows

The script popgenWindows.py computes some standard population genomic statistics in sliding windows: pi, FST and DXY. It requires the script genomics.py to be present in the same directory, or in your Python path.

Example command

bash
python popgenWindows.py -w 50000 -m 5000 -g input.geno.gz -o output.csv.gz -f phased -T 5 -p popA A1,A2,A3,A4 -p popB B1,B2,B3,B4,B6,B6 -p popC -p popD --popsFile pops.txt 

Notes

python popgenWindows.py -h Will print a full list of command arguments.

  • Input is a .geno file as shown above. This can be gzipped (.geno.gz). Output is a .csv. If you add .gz it will be gzipped.

  • Genotype encoding is indicated by the -f flag. -f phased is normally used, see the table above. Other options are -f haplo for haploid data (although phased will also interpret haploid data correctly), -f diplo and f pairs which is like the alleles output in the table above, but assumes diplid data.

  • There are three options for defining the windows to be analysed, using the --windType argument.

Wndow Type Description
coordinate Windows will cover a fixed range in the genome, which is defined as the window size. If there is missing data, this can lead to variable numbers of sites used for each window.
sites Each window will have the same number of sites. If there is missing data, this can lead to different absolute sizes for windows in terms of genome coordinates.
predefined This will analyse predefined windows provided using the --windCoords flag.
  • You can either include sample names after the population name, separated by commas, or provide only the population name, along with a populations file, with the flag --popsFile , which has two columns: the first gives sample names and teh second gives population name:
C1  popC
C2  popC
C3  popC
C4  popC
D1  popD
D2  popD
D3  popD
D4  popD
  • The most common source of errors here involve the -m (--minSites) flag. If you are useing coordinate windows and have any sites with missing data, then -m must be set to a value smaller than the window size. If you have reduced representation data such as RADseq, you will need a much lower -m value (more like 1% of the window size or even less).

  • If some samples are haploid and others are diploid, you can use one of the diploid formats, but indicate that certain samples are haploid by listing them after the --haploid flag. The script will force them to have haploid genotyps, and any apparently heterozygous genotype will be converted to N.

  • The script can run on multiple cores (-T flag). Try different numbers, as using too many can slow the script down (due to the difficulty in sorting the outputs coming from the different cores).


Allele frequencies per site

The script freq.py computes allele frequencies at each site, either for each base, the minor allele, or the derived allele (if an outgroup is provided). It requires the script genomics.py to be present in the same directory, or in your Python path.

Example command

bash
python freq.py -g input.geno.gz -p pop1 -p pop2 --popsFile populations.txt --target derived --threads 10

Notes

  • Input is a .geno file as shown above. This can be gzipped (.geno.gz). Output is a .csv. If you add .gz it will be gzipped.

  • If you do not include the --target option, the script will export the count (i.e. an integer) of each base (A, C, G, T) for ech population.

  • If you set --target, you have two options

Wndow Type Description
--target minor Export the frequency of the minor (rarer) allele at each site (based on all included individuals). This can be at most 0.5 for the whole dataset, but can exceed 0.5 in certain populations
--target derived Export the frequency of the derived allele. NOTE: this assumes that the last population listed with -p is the outgroup.

Site frequency spectrum

The script sfs.py computes site frequency spectrum (SFS, also called the allele frequency spectrum) from input variants. It requires the script genomics.py to be present in the same directory, or in your Python path.

The input for sfs.py is produced by freq.py (see above). These two scripts can be combined with a pipe.

Example commands

bash
#1D folded (derived allele) sfs
python freq.py -g input.geno.gz -f phased --threads 10 -p pop1 -p pop2 -p outgroup | \
python sfs.py --inputType baseCounts -p pop1 -p pop2 -p outgroup \
--FSpops pop1 pop2 --polarized --subsample 10 10 --pref mydata. --suff .subsample10.sfs

#2D unfolded (minor allele) SFS
python freq.py -g input.geno.gz -f phased --threads 10 -p pop1 -p pop2 | \
python sfs.py --inputType baseCounts -p pop1 -p pop2 \
--FSpops pop1 pop2 --doPairs --subsample 10 10 --pref mydata. --suff .subsample10.sfs

Notes

  • Input is a .geno file as shown above. This can be gzipped (.geno.gz).

  • Output is a table giving the number of sites with each allele count.

  • --polarized will produce an unfolded SFS assuming the final population named is the outgroup

  • --doPairs, --doTrios and doQuartets will also export the 2D, 3D and 4D SFS for all pairs, trios or quartets of populations.


Distance matrix

The script distMat.py computes a distance matrix among all pairs of individuals. This can be computed either for the entire input file or in windows, as in the popgenWindows script above. This works for samples of any ploidy or mix of ploidies. For ploidy > 1, the pairwise diatance will be the average diatance among all haplotypes in the two individuals.

Example Command

bash
python distMat.py -g input.geno.gz -f phased --windType cat -o output.dist

python distMat.py -h will print a list of command options.

Notes

  • This script shares several command arguments with popgenWindows.py. And input formats are the same. Please see the notes for that script above.

  • Output format has three otptions: raw, phylip and nexus.

  • To make a single matrix for the entire input file (i.e. not making windows and ignoring scaffold bounaries), use --windType cat

  • If using --windType cat, the entire input file will be read into memory. This often leads to RAM issues for large files. One option is to filter your data hard before doing this analysis using filterGenotypes.py.

  • To make separate matrices for windows, use one of the window type options described above. There will still be a single output file, but with separate matrices separated by blank lines.


ABBA-BABA statistics in sliding windows

The script ABBABABAwindows.py performs analyses described in Martin et al. 2015, MBE, compurting the D statistic and f estimators in windows across the genome. Like the script above, it requires genomics.py.

Example command

bash
python ABBABABAwindows.py -g /zoo/disk1/shm45/vcf/set62/set62.chr21.DP5GQ30.AN100M

Issues· 58 开放

查看全部 Issues在 GitHub 打开

暂无开放 Issues,或尚未同步最近议题。

> 标签

Python

暂无评论,来聊聊你的看法吧

> 工具信息

发布日期2026年8月1日
最后更新2026年9月17日
分类编程语言
定价开源

> 相关工具

T
TypeScript
JavaScript 的超集,为前端与全栈提供静态类型
P
Python
通用编程语言,广泛用于 Web、数据与 AI
G
Go
Google 推出的简洁高效系统语言