# -*- org-export-use-babel: nil -*-
#+title: Comparative Genomics Project
#+subtitle: Duplicate Genes in /Glycine max/
#+date: 2024-2025
#+author: Samuel Ortion
#+LATEX_CLASS: scrartcl
#+LATEX_HEADER: \titlehead{M2 GENIOMHE}
#+LATEX_HEADER: \usepackage{mus}

* General infos

<2024-10-25 Fri>

** Objectives
*** Part 1
1. Determine the amount of duplicate genes in my genome.
   - What is the proportion of duplicate genes in my genome.
   - Give an interval based on two datasets (low stringency dataset and high stringency dataset).
     #+caption: High stringency dataset and low stringency dataset duplicate genes filtering criteria.
     #+name: dataset-criteria
     |            | Low                           | High                          |
     |------------+-------------------------------+-------------------------------|
     | Coverage   | 30%                           | 40%                           |
     | %identity  | 30%                           | 50%                           |
     | Bitscore   | no filtering                  | no filtering                  |
     | Clustering | mcl (with default parameters) | mcl (with default parameters) |

     #+name: low-stringency
     #+begin_src bash
min_coverage=30
min_identity=30
     #+end_src

     #+name: high-stringency
     #+begin_src bash
min_coverage=40
min_identity=50
     #+end_src
2. Determine the amount of Tandemly Arrayed Genes (TAGs) in the genome (definition 0 and 1).
3. Determine the amount in plant's chloroplasts and mitochondria.
4. Is there a lot of duplicated operons in plants?

*** Part 2
1. Compute \(K_s\) values and use it as a proxy of the age of the duplication.
2. Are TAGs younger than other duplicate genes?


** Our species

/Glycine max/ is the soy plant.

* Download the source data
- https://plants.ensembl.org/Glycine_max/Info/Index
- https://ftp.ensemblgenomes.ebi.ac.uk/pub/plants/release-60/fasta/glycine_max/pep/Glycine_max.Glycine_max_v2.1.pep.all.fa.gz

* Duplicate gene families
** Develop the Nextflow workflow

*** Filtering the proteome to keep the longest isoform

I used three awk families_and_TAGs/scripts to filter the FASTA file to keep only the longest isoforms:
 1. =protein_lengths.awk= associate each FASTA record with the length of its sequence,
 2. =filter_longest.awk= keep the isoform ID that has the maximum length,
 3. =filter_records.awk= based on the list of isoform ID, retrieve the sequence of from the original file.

=protein_lengths.awk=

#+include: workflow/families_and_TAGs/scripts/protein_lengths.awk src awk

=filter_longest.awk=

#+include: workflow/families_and_TAGs/scripts/filter_longest.awk src awk

=filter_records.awk=

#+include: workflow/families_and_TAGs/scripts/filter_records_fasta.awk src awk

How many protein coding gene are there?
#+name: protein-coding-gene-count
#+begin_src bash :export both
zcat ../data/Glycine_max.Glycine_max_v2.1.pep.all.fa.gz \
    | grep -v 'supercontig' \
    | awk ' /^>/ { print $4 }' | sort | uniq | wc -l
#+end_src

#+RESULTS: protein-coding-gene-count
: 55589

#+RESULTS:
: 55897

*** Filtering out Chloroplasts and Mitochondria proteins

From the Ensembl Plants index page we can see that there is no chloroplast nor mitochondria proteins in the sequenced genome of /Glycine max/.

*** Filter out supercontigs

<2024-10-28 Mon>

#+include: workflow/families_and_TAGs/scripts/remove_supercontigs_chloroplasts_mitochondria_fasta.awk src awk

#+begin_src bash
zcat ../data/Glycine_max.Glycine_max_v2.1.pep.all.fa.gz \
    | awk -f ./workflow/families_and_TAGs/scripts/remove_supercontigs_chloroplasts_mitochondria_fasta.awk \
    > ./tmp/Glycine_max.Glycine_max_v2.1.pep.all.nosupercontig.fa
#+end_src

#+RESULTS:

/Sanity check:/ Has the script removed all supercontigs proteins and only them?

/How many records in supercontigs were there?/
#+begin_src bash :export both
zcat ../data/Glycine_max.Glycine_max_v2.1.pep.all.fa.gz  \
    | awk -F' ' ' /^>/ { print $3}'  | grep -c "^supercontig"
#+end_src

#+RESULTS:
: 435

/How many records have been removed?/

#+name: all-isoforms
#+begin_src bash
zgrep -c "^>" ../data/Glycine_max.Glycine_max_v2.1.pep.all.fa.gz
#+end_src

#+RESULTS: all-isoforms
: 88412

#+name: all-isoforms-no-supercontig
#+begin_src bash
grep -c "^>" ./tmp/Glycine_max.Glycine_max_v2.1.pep.all.nosupercontig.fa
#+end_src

#+RESULTS: all-isoforms-no-supercontig
: 87977

#+begin_src bash :var ALL_ISOFORMS=all-isoforms :var NO_SUPERCONTIGS=all-isoforms-no-supercontig :exports both
expr $ALL_ISOFORMS - $NO_SUPERCONTIGS
#+end_src

#+RESULTS:
: 435

*** BLASTp All Against All

#+begin_src bash
makeblastdb -in "$proteome" -dbtype prot -out 'tmp/blastdb' -title "Glycine max"
#+end_src

#+begin_src bash
blastp -query "$proteome" -db 'tmp/blastdb' -outfmt 6 -out "tmp/Glycine_max.blastp.tsv"
#+end_src

*** Clustering

Using =mcl=.

1. Extract the homology in ABC format
   #+begin_src bash
awk 'BEGIN { OFS="\t" } { print \$14, \$16, \$12 }' "${blastp}" > 'graph.abc'
   #+end_src
2. Run =mcl= with default parameters
   #+begin_src bash
mcl 'graph.abc' --abc -o 'custering.mcl'
   #+end_src
3. Convert =mcl= output to TSV with two columns =gene id=, =family id=.
   #+include: workflow/families_and_TAGs/scripts/mcl_to_tsv.awk src awk

   #+begin_src bash
awk -f workflow/families_and_TAGs/scripts/mcl_to_tsv.awk 'clustering.mcl' > 'families.tsv'
   #+end_src


** Running the Nextflow workflow

Update the parameters in =nextflow.config= and run
#+begin_src bash
nextflow run main.nf -profile conda -resume
#+end_src

from the =./workflow/= folder

*** From the given BLASTp format 6 file
<2024-11-02 Sat>
/How many BLASTp HSP have been reported?/
#+begin_src bash
cat ../data/Glycine_max_Blastp_longIsoforme | wc -l
#+end_src

#+RESULTS:
: 4456730

*** Remove HSP on proteins found on supercontigs

Extract the isoform ID whose gene is on supercontigs.
#+begin_src bash
zcat ../data/Glycine_max.Glycine_max_v2.1.pep.all.fa.gz| awk -F' ' '/^>/ && $3 ~ /^supercontig/ { sequence_id = $1; gsub(">", "", sequence_id); print sequence_id }' > ./tmp/protein_on_supercontigs.list
#+end_src

#+RESULTS:

#+begin_src bash
cat ./tmp/protein_on_supercontigs.list | wc -l
#+end_src

#+RESULTS:
: 435

Remove HSP with subject or query being on a supercontig

=filter_blastp_sequence_id.awk=

#+include: workflow/families_and_TAGs/scripts/filter_blastp_sequence_id_remove.awk src awk

#+begin_src bash
awk -f workflow/families_and_TAGs/scripts/filter_blastp_sequence_id.awk ./tmp/protein_on_supercontigs.list ../data/Glycine_max_Blastp_longIsoforme > ./tmp/Glycine_max_Blastp_longIsoforme_nosupercontig
#+end_src

#+RESULTS:

How many HSP have been removed?

#+begin_src bash
length() {
    cat $1 | wc -l
 }
expr $(length ../data/Glycine_max_Blastp_longIsoforme) - $(length ./tmp/Glycine_max_Blastp_longIsoforme_nosupercontig)
#+end_src

#+RESULTS:
: 36598

#+begin_src bash
cat ./tmp/Glycine_max_Blastp_longIsoforme_nosupercontig | wc -l
#+end_src

#+RESULTS:
: 4420132


*** Filter by coverage and identity percentage

*** Add protein length and gene name columns

**** Extract the protein length and gene id from the FASTA file


#+begin_src bash
zcat ../data/Glycine_max.Glycine_max_v2.1.pep.all.fa.gz | awk -f ./workflow/families_and_TAGs/scripts/protein_lengths.awk > ./tmp/Glycine_max_protein_length_gene.tsv
#+end_src

#+RESULTS:


**** Join the BLASTp HSP file with the gene id and gene lengths

#+begin_src bash
protein_length=./tmp/Glycine_max_protein_length_gene.tsv
blastp=./tmp/Glycine_max_Blastp_longIsoforme_nosupercontig
sort -k 1 "${blastp}" > tmp/blastp_s
sort -k 1 "${protein_length}" > tmp/protein_length_s
join -1 1 -2 1 -t $'\t' tmp/blastp_s tmp/protein_length_s > tmp/join1.tsv
sort -k 2 tmp/join1.tsv > tmp/join1.tsv_s
join -1 2 -2 1 -t $'\t' tmp/join1.tsv_s tmp/protein_length_s > tmp/joined.blastp.tsv
#+end_src

#+RESULTS:

**** Filter the BLASTp according to coverage and identity percentage

#+name: filter-blastp
#+begin_src bash
blastp=./tmp/joined.blastp.tsv
awk -f "workflow/families_and_TAGs/scripts/filter_blastp.awk" \
        -v min_coverage="${min_coverage}" \
        -v min_identity="${min_identity}" \
        "${blastp}" > "tmp/Glycine_max_Blastp_filtered_coverage${min_coverage}_identity${min_identity}.tsv"
#+end_src

#+RESULTS: filter-blastp

***** First dataset

#+begin_src bash :noweb strip-export
<<low-stringency>>
<<filter-blastp>>
#+end_src

#+RESULTS:

#+begin_src bash :noweb strip-export
<<high-stringency>>
<<filter-blastp>>
#+end_src

#+RESULTS:

#+begin_src bash
cat ./tmp/Glycine_max_Blastp_filtered_coverage30_identity30.tsv | wc -l
#+end_src

#+RESULTS:
: 1520850

#+begin_src bash
cat ./tmp/Glycine_max_Blastp_filtered_coverage40_identity50.tsv | wc -l
#+end_src

#+RESULTS:
: 342180

*** Clustering
<2024-11-03 Sun>
*** Extract the homology graph

#+name: extract-abc
#+begin_src bash
blastp="tmp/Glycine_max_Blastp_filtered_coverage${min_coverage}_identity${min_identity}.tsv"
abc="tmp/Glycine_max_Blastp_filtered_coverage${min_coverage}_identity${min_identity}.abc"
awk 'BEGIN { FS = OFS = "\t" } { print $14, $16, $12 }' "$blastp" > "$abc"
#+end_src


#+begin_src bash :noweb strip-export
<<low-stringency>>
<<extract-abc>>
#+end_src

#+RESULTS:

#+begin_src bash :noweb strip-export
<<high-stringency>>
<<extract-abc>>
#+end_src

#+RESULTS:

#+begin_src bash
head -n 1 "tmp/Glycine_max_Blastp_filtered_coverage30_identity30.abc"
#+end_src

#+RESULTS:
| GLYMA_10G098500 | GLYMA_U008600 | 154 |

#+begin_src bash
wc -l "./tmp/Glycine_max_Blastp_filtered_coverage30_identity30.abc"
#+end_src

#+RESULTS:
: 1520850 ./tmp/Glycine_max_Blastp_filtered_coverage30_identity30.abc

*** Filter the homology graph to keep only the heaviest weight


#+include: workflow/families_and_TAGs/scripts/keep_heaviest_edge_abc.awk src awk

#+name: filter-abc
#+begin_src bash
original_abc="tmp/Glycine_max_Blastp_filtered_coverage${min_coverage}_identity${min_identity}.abc"
filtered_abc="tmp/Glycine_max_Blastp_filtered_coverage${min_coverage}_identity${min_identity}_f.abc"
awk -f "workflow/families_and_TAGs/scripts/keep_heaviest_edge_abc.awk" "$original_abc" > "$filtered_abc"
#+end_src

#+begin_src bash :noweb strip-export
<<low-stringency>>
<<filter-abc>>
#+end_src

#+RESULTS:

#+begin_src bash :noweb strip-export
<<high-stringency>>
<<filter-abc>>
#+end_src

#+RESULTS:

*** Run Markov Clustering

#+name: mcl
#+begin_src bash
abc="tmp/Glycine_max_Blastp_filtered_coverage${min_coverage}_identity${min_identity}_f.abc"
mcl="tmp/Glycine_max_Blastp_filtered_coverage${min_coverage}_identity${min_identity}.mcl"
mcl "${abc}" --abc --discard-loops=yes -o "$mcl"
#+end_src

#+RESULTS: mcl

#+begin_src bash :noweb strip-export
<<low-stringency>>
<<mcl>>
#+end_src

#+RESULTS:

#+begin_src bash :noweb strip-export
<<high-stringency>>
<<mcl>>
#+end_src

#+RESULTS:

*** Transform MCL output into a gene to family mapping TSV file

#+name: mcl-to-tsv
#+begin_src bash
mcl="tmp/Glycine_max_Blastp_filtered_coverage${min_coverage}_identity${min_identity}.mcl"
tsv="tmp/Glycine_max_Blastp_filtered_coverage${min_coverage}_identity${min_identity}.mcl.tsv"
awk -f "workflow/families_and_TAGs/scripts/mcl_to_tsv.awk" "${mcl}" > "${tsv}"
#+end_src

#+begin_src bash :noweb strip-export
<<low-stringency>>
<<mcl-to-tsv>>
#+end_src

#+RESULTS:

#+begin_src bash :noweb strip-export
<<high-stringency>>
<<mcl-to-tsv>>
#+end_src

#+RESULTS:

#+name: duplicate-genes-count-high-stringency
#+begin_src bash
tsv="tmp/Glycine_max_Blastp_filtered_coverage40_identity50.mcl.tsv"
cat "${tsv}" | wc -l
#+end_src

#+RESULTS: duplicate-genes-count-high-stringency
: 46769

#+name: duplicate-genes-count-low-stringency
#+begin_src bash
tsv="tmp/Glycine_max_Blastp_filtered_coverage30_identity30.mcl.tsv"
cat "${tsv}" | wc -l
#+end_src

#+RESULTS: duplicate-genes-count-low-stringency
: 50254

#+begin_src bash
tsv="tmp/Glycine_max_Blastp_filtered_coverage40_identity50.mcl.tsv"
head "${tsv}"
#+end_src

#+RESULTS:
| GLYMA_12G034500 | 1 |
| GLYMA_12G138300 | 1 |
| GLYMA_12G124600 | 1 |
| GLYMA_12G126700 | 1 |
| GLYMA_12G125200 | 1 |
| GLYMA_12G034900 | 1 |
| GLYMA_U036400   | 1 |
| GLYMA_12G124200 | 1 |
| GLYMA_12G126300 | 1 |
| GLYMA_12G035100 | 1 |

#+begin_src bash
tsv="tmp/Glycine_max_Blastp_filtered_coverage30_identity30.mcl.tsv"
head "${tsv}"
#+end_src

#+RESULTS:
| GLYMA_07G053000 | 1 |
| GLYMA_10G247700 | 1 |
| GLYMA_12G188000 | 1 |
| GLYMA_07G159200 | 1 |
| GLYMA_13G313300 | 1 |
| GLYMA_15G016600 | 1 |
| GLYMA_14G159200 | 1 |
| GLYMA_18G287500 | 1 |
| GLYMA_20G132900 | 1 |
| GLYMA_16G204800 | 1 |

What is the interval of duplicate gene count? There are between 48109 and 49391 duplicate genes in /Glycine max/.

What is the number of families?

#+begin_src bash :exports both
tsv="tmp/Glycine_max_Blastp_filtered_coverage30_identity30.mcl.tsv"
cut -f2 "${tsv}" | sort | uniq | wc -l
#+end_src

#+RESULTS:
: 8426

#+begin_src bash :exports both
tsv="tmp/Glycine_max_Blastp_filtered_coverage40_identity50.mcl.tsv"
cut -f2 "${tsv}" | sort | uniq | wc -l
#+end_src

#+RESULTS:
: 11997

What is the number of singletons?

#+name: singletons-low-stringency
#+begin_src bash :var ALL=protein-coding-gene-count :var DUPLICATED=duplicate-genes-count-low-stringency
expr $ALL - $DUPLICATED
#+end_src

#+RESULTS: singletons-low-stringency
: 5643

#+name: singletons-high-stringency
#+begin_src bash :var ALL=protein-coding-gene-count :var DUPLICATED=duplicate-genes-count-high-stringency
expr $ALL - $DUPLICATED
#+end_src

#+RESULTS: singletons-high-stringency
: 9128
* Amount of Tandemly Arrayed Genes

** Extract gene position from a GFF3 file

#+include: workflow/families_and_TAGs/scripts/gff3_to_gene_positions_table.awk src awk

#+begin_src bash
zcat "../data/Glycine_max.Glycine_max_v2.1.60.chr.gff3.gz" \
    | awk -f "./workflow/families_and_TAGs/scripts/gff3_to_gene_positions_table.awk" \
    > "./tmp/Glycine_max.gene_positions.tsv"
#+end_src

#+RESULTS:

#+begin_src bash
head "./tmp/Glycine_max.gene_positions.tsv"
#+end_src

#+RESULTS:
| GLYMA_01G000100 | 1 |  27355 |  28320 |
| GLYMA_01G000200 | 1 |  58975 |  67527 |
| GLYMA_01G000300 | 1 |  67770 |  69968 |
| GLYMA_01G000400 | 1 |  90152 |  95947 |
| GLYMA_01G000500 | 1 |  90289 |  91197 |
| GLYMA_01G000600 | 1 | 116094 | 127845 |
| GLYMA_01G000700 | 1 | 143467 | 155573 |
| GLYMA_01G000800 | 1 | 157030 | 157772 |
| GLYMA_01G000900 | 1 | 170534 | 193342 |
| GLYMA_01G001000 | 1 | 196256 | 201895 |

#+name: gene-positions-count
#+begin_src bash :exports both
cat "./tmp/Glycine_max.gene_positions.tsv" \
    | wc -l
#+end_src

#+RESULTS:
: 55589

There is 55897 protein coding genes in the sequence file, but only 55589 gene positions.

#+begin_src bash :var POSITIONS=gene-positions-count :var GENES=protein-coding-gene-count :exports both
expr $GENES - $POSITIONS
#+end_src

#+RESULTS:
: 308

** Extract TAGs

#+name: extract-TAGs
#+begin_src bash
positions="./tmp/Glycine_max.gene_positions.tsv"
families="./tmp/Glycine_max_Blastp_filtered_coverage${min_coverage}_identity${min_identity}.mcl.tsv"
./rust/tagfinder/target/release/tagfinder \
    --families "$families" \
    --positions "$positions" \
    --definitions 0,1 \
    > "./tmp/Glycine_max_TAGs_coverage${min_coverage}_identity${min_identity}.tsv"
#+end_src

#+RESULTS: extract-TAGs

#+begin_src bash :noweb strip-export
<<low-stringency>>
<<extract-TAGs>>
#+end_src

#+RESULTS:

#+begin_src bash :noweb strip-export
<<high-stringency>>
<<extract-TAGs>>
#+end_src

#+RESULTS:

# LocalWords:  operons

#+begin_src bash
head "./tmp/Glycine_max_TAGs_coverage30_identity30.tsv"
#+end_src

#+RESULTS:
| gene            |     family | tag0 | tag1 |
| GLYMA_19G258600 | spacer3040 | -    | -    |
| GLYMA_08G277500 |         68 | -    | -    |
| GLYMA_08G342200 |        363 | 2467 | 2749 |
| GLYMA_08G072200 |       3499 | 2317 | 2592 |
| GLYMA_15G143500 | spacer1810 | -    | -    |
| GLYMA_09G191100 |       3783 | -    | -    |
| GLYMA_02G090200 |       7931 | -    | -    |
| GLYMA_17G091400 |        584 | -    | -    |
| GLYMA_13G199100 |         79 | -    | 621  |

How many TAGs for definition 0?

#+begin_src bash :exports both
awk 'BEGIN {
           max=0
    }
    NR > 1 {
        if ($3 > max) {
            max=$3
        }
    }
    END  {
        print max
    }
    ' "./tmp/Glycine_max_TAGs_coverage30_identity30.tsv"
#+end_src

#+RESULTS:
: 2620

For TAG1?

#+begin_src bash :exports both
awk 'BEGIN {
           max=0
    }
    NR > 1 {
        if ($4 > max) {
            max=$4
        }
    }
    END  {
        print max
    }
    ' "./tmp/Glycine_max_TAGs_coverage30_identity30.tsv"
#+end_src

#+RESULTS:
: 2916


#+begin_src bash :exports both
awk 'BEGIN {
           max=0
    }
    NR > 1 {
        if ($3 > max) {
            max=$3
        }
    }
    END  {
        print max
    }
    ' "./tmp/Glycine_max_TAGs_coverage40_identity50.tsv"
#+end_src

#+RESULTS:
: 2157


#+begin_src bash
awk 'BEGIN {
           max=0
    }
    NR > 1 {
        if ($4 > max) {
            max=$4
        }
    }
    END  {
        print max
    }
    ' "./tmp/Glycine_max_TAGs_coverage40_identity50.tsv"
#+end_src

#+RESULTS:
: 2438


** Size of the greatest TAG

*** High stringency

/TAG₀/

#+begin_src bash :exports both
awk '
    {
    tag_index=$3
    if (tag_index != "-")
        gene_count[tag_index]++
    }
    END {
    max = 0
    max_tag_index = ""
    for (tag_index in gene_count) {
        if (gene_count[tag_index] > max) {
            max_tag_index = tag_index
            max = gene_count[tag_index]
        }
    }
    print max_tag_index, max
    }' "./tmp/Glycine_max_TAGs_coverage40_identity50.tsv"
#+end_src

#+RESULTS:
: 2134 32

/TAG₁/

#+begin_src bash :exports both
awk '
    {
    tag_index=$4
    if (tag_index != "-")
        gene_count[tag_index]++
    }
    END {
    max = 0
    max_tag_index = ""
    for (tag_index in gene_count) {
        if (gene_count[tag_index] > max) {
            max_tag_index = tag_index
            max = gene_count[tag_index]
        }
    }
    print max_tag_index, max
    }' "./tmp/Glycine_max_TAGs_coverage40_identity50.tsv"
#+end_src

#+RESULTS:
: 416 43

*** Low stringency


/TAG₀/

#+begin_src bash :exports both
awk '
    {
    tag_index=$3
    if (tag_index != "-")
        gene_count[tag_index]++
    }
    END {
    max = 0
    max_tag_index = ""
    for (tag_index in gene_count) {
        if (gene_count[tag_index] > max) {
            max_tag_index = tag_index
            max = gene_count[tag_index]
        }
    }
    print max_tag_index, max
    }' "./tmp/Glycine_max_TAGs_coverage30_identity30.tsv"
#+end_src

#+RESULTS:
: 2591 32

/TAG₁/

#+begin_src bash :exports both
awk '
    {
    tag_index=$4
    if (tag_index != "-")
        gene_count[tag_index]++
    }
    END {
    max = 0
    max_tag_index = ""
    for (tag_index in gene_count) {
        if (gene_count[tag_index] > max) {
            max_tag_index = tag_index
            max = gene_count[tag_index]
        }
    }
    print max_tag_index, max
    }' "./tmp/Glycine_max_TAGs_coverage30_identity30.tsv"
#+end_src

#+RESULTS:
: 525 43


** Amount of TAGs

How many TAG genes are there for low and high stringency datasets?

#+name: count-tags
#+begin_src bash
awk 'BEGIN { count0=0; count1; } { if ($3 != "-") count0++; if($4 != "-") count1++; } END { print count0, count1 }' $tags
#+end_src

#+RESULTS: count-tags

#+begin_src bash :noweb strip-export
tags=./tmp/Glycine_max_TAGs_coverage30_identity30.tsv
<<count-tags>>
#+end_src

#+RESULTS:
: 6680 7876

#+begin_src bash :noweb strip-export
tags=./tmp/Glycine_max_TAGs_coverage40_identity50.tsv
<<count-tags>>
#+end_src

#+RESULTS:
: 5397 6480

* Complete slides
<2024-11-23 Sat>

* Family sizes

#+begin_src bash
head -1 ./tmp/Glycine_max_Blastp_filtered_coverage30_identity30.mcl \
    | awk '{ print NF }'
#+end_src

#+RESULTS:
: 437

#+begin_src bash
head -1 ./tmp/Glycine_max_Blastp_filtered_coverage40_identity50.mcl \
    | awk '{ print NF }'
#+end_src

#+RESULTS:
: 159


* $K_s$ For all duplicated pairs within a family

Let $A$ be a family of $n$ duplicate members. The number of duplicate pairs is $n(n-1) / 2$.

Install PAML, Pal2Nal and Clustal packages with conda:
#+begin_src bash
conda install paml pal2nal clustalw -c bioconda
#+end_src

** Testing each step
<2024-12-22 Sun>

*** All pair of duplicate genes

Generated by a rust program in =rust/pairs= folder.
The algorithm is simple.
We assume the gene-families mapping file is sorted by family index.
For each family, loop for i from 0 to the size of the family minus one, loop for j from i plus one to the size of the family and output the pair gene i - gene j.

Example:
#+begin_src bash :exports both
cat ./tmp/Glycine_max_Blastp_filtered_coverage30_identity30.mcl.tsv | ./rust/pairs/target/release/pairs > ./tmp/Glycine_max_duplicate_gene_pairs.tsv
head ./tmp/Glycine_max_duplicate_gene_pairs.tsv
#+end_src

#+RESULTS:
| GLYMA_07G053000 | GLYMA_10G247700 |
| GLYMA_07G053000 | GLYMA_12G188000 |
| GLYMA_07G053000 | GLYMA_07G159200 |
| GLYMA_07G053000 | GLYMA_13G313300 |
| GLYMA_07G053000 | GLYMA_15G016600 |
| GLYMA_07G053000 | GLYMA_14G159200 |
| GLYMA_07G053000 | GLYMA_18G287500 |
| GLYMA_07G053000 | GLYMA_20G132900 |
| GLYMA_07G053000 | GLYMA_16G204800 |
| GLYMA_07G053000 | GLYMA_08G302300 |

*** PAML Steps
Example for the first gene pair.

/Extract the CDS sequence and protein sequence of the genes/

#+name: pair-gene-id
#+begin_src bash
gene_id_1="GLYMA_07G053000"
gene_id_2="GLYMA_10G247700"
#+end_src

- CDS:
#+name: pair-gene-cds-extract
#+begin_src bash :noweb strip-export
output_file="./tmp/${gene_id_1}_${gene_id_2}.cds.fst"
echo "" > "${output_file}"
for gene_id in "${gene_id_1}" "${gene_id_2}"; do
        zcat ../data/Glycine_max.Glycine_max_v2.1.cds.all.fa.gz | awk -v gene_id="${gene_id}" '
        BEGIN {
            on_gene=0
        }
        /^[^>]/ && on_gene == 1 {
            print $0
        }
        /^>/ {
            gene = $4
            gsub("gene:", "", gene)
            if (gene == gene_id) {
                on_gene=1
                print $0
            } else {
                on_gene=0
            }
        }
        ' >> "${output_file}"
done
#+end_src

#+RESULTS: pair-gene-cds-extract

#+begin_src bash :noweb strip-export
<<pair-gene-id>>
<<pair-gene-cds-extract>>
#+end_src

#+RESULTS:

- Proteins:

#+name: pair-gene-protein-extract
#+begin_src bash :noweb strip-export
output_file="./tmp/${gene_id_1}_${gene_id_2}.prot.fst"
echo "" > "${output_file}"
for gene_id in "${gene_id_1}" "${gene_id_2}"; do
        cat ./tmp/proteome_filtered.fa | awk -v gene_id="${gene_id}" '
        BEGIN {
            on_gene=0
        }
        /^[^>]/ && on_gene == 1 {
            print $0
        }
        /^>/ {
            gene = $4
            gsub("gene:", "", gene)
            if (gene == gene_id) {
                on_gene=1
                print $0
            } else {
                on_gene=0
            }
        }
        ' >> "${output_file}"
done
#+end_src

#+RESULTS: pair-gene-protein-extract

#+begin_src bash :noweb strip-export
<<pair-gene-id>>
<<pair-gene-protein-extract>>
#+end_src

#+RESULTS:

/Clustalw2 alignment/

#+begin_src bash :noweb strip-export
<<pair-gene-id>>
protein_sequence="./tmp/${gene_id_1}_${gene_id_2}.prot.fst"
clustalw2 -quiet -align -infile="${protein_sequence}" -outfile="./tmp/${gene_id_1}_${gene_id_2}.prot.ali.aln"
#+end_src

#+RESULTS:

/Convert protein alignment to coding sequence alignment/
#+begin_src bash :noweb strip-export
<<pair-gene-id>>
protein_alignment="./tmp/${gene_id_1}_${gene_id_2}.prot.ali.aln"
coding_sequence="./tmp/${gene_id_1}_${gene_id_2}.cds.fst"
pal2nal "${protein_alignment}" "${coding_sequence}" -output paml > "./tmp/${gene_id_1}_${gene_id_2}.cds.ali.phy"
#+end_src

/YN00 Ka-Ks computation/

#+begin_src bash :noweb strip-export
<<pair-gene-id>>
phylip_file="./tmp/${gene_id_1}_${gene_id_2}.cds.ali.phy"
echo "seqfile = ${phylip_file}\noutfile = ${phylip_file}.yn\nverbose = 0\nicode = 0\nweighting = 0\ncommonf3x4 = 0" > yn00.ctl
yn00
#+end_src

/Extraction of Ka-Ks values/

#+begin_src bash :noweb strip-export :exports both
<<pair-gene-id>>
yn_file="./tmp/${gene_id_1}_${gene_id_2}.cds.ali.phy.yn"
awk '
    BEGIN {
          on_good_section=0
          skip=0
    }
    $1 == "(B)" {
         skip=8
         on_good_section=1
    }

    on_good_section == 1 {
        if (skip == 0) {
            Ka=$8
            Ks=$11
            print Ka, Ks
            exit
        } else {
            skip -= 1
        }
    }
    ' "${yn_file}"
#+end_src

#+RESULTS:
: 0.6207 1.9409

*** Run the Nextflow workflow
<2024-12-23 Mon>

The  Nextflow workflow is stuck on the first steps, because the channel containing the queue of jobs for each gene pair is too huge in memory.

*** Back to basics

I wrote a bash script that compute the Ka/Ks not loading the whole pairs file in memory.

<2024-12-26 Thu> The script is still running.

I did not handle the case where several coding sequences were present in the CDS file for the same gene, so the script ignores the proteins with several isoforms.

Let us filter the CDS to match the proteome file:

#+begin_src bash
awk '/^>/ { gsub(">", "", $1); print $1 }' ./tmp/proteome_filtered.fa > ./tmp/proteome_filtered.list
#+end_src

#+RESULTS:

#+begin_src bash
cat ./tmp/proteome_filtered.list | wc -l
#+end_src

#+RESULTS:
: 55589 ./tmp/proteome_filtered.list

#+begin_src bash
awk -f ./workflow/families_and_TAGs/scripts/filter_records_fasta.awk ./tmp/proteome_filtered.list ../data/Glycine_max.Glycine_max_v2.1.cds.all.fa > ./tmp/cds_filtered.fa
#+end_src

#+RESULTS:

#+begin_src bash :exports both
grep -c "^>" ./tmp/cds_filtered.fa
#+end_src

#+RESULTS:
: 55589

* Are TAG pairs different in age from non-TAG pairs ?

Run the bash script [[./workflow/KaKs/ugly.sh]].

<2025-01-07 Tue>
Computation ended after about a week.

<2025-01-08 Wed>

#+begin_src bash
awk -F',' '$4 < 1' ./workflow/KaKs/results/complete_uniq.csv > ./workflow/KaKs/results/complete_uniq_ks_below1.csv
cat ./workflow/KaKs/results/complete_uniq_ks_below1.csv| wc -l
#+end_src

#+RESULTS:
: 294226

#+begin_src R :session *R* :results graphics file :file results/plots/Glycine_max_Ks_distribution.png :width 8 :height 8 :res 200 :units cm
library(ggplot2)
library(scales)

data <- read.table("./workflow/KaKs/complete2.csv", sep=",", header=TRUE)
colnames(data) <- c("gene_a","gene_b","ka","ks")

theme_set(theme_gray(base_size=8))
gg <- ggplot(data, aes(x=ks))
gg <- gg + geom_density()
gg <- gg + scale_x_continuous("Age (Ks)")
gg <- gg + scale_y_continuous("Density")
gg <- gg + xlim(0, 1)
gg <- gg + ggtitle("Proportion of duplicate gene pairs age")
gg <- gg + theme(plot.title=element_text(hjust=0.5))
gg
#+end_src

#+RESULTS:
[[file:results/plots/Glycine_max_Ks_distribution.png]]

#+caption: Density of pair with a given Ks. (Inspired from Blanc and Wolfe, 2004)

#+RESULTS:
[[file:results/plots/Glycine_max_Ks_distribution.png]]

Then, let us plot the same data separating TAG from non-TAG pairs.

#+begin_src R :session *R* :results silent
data <- read.csv("./workflow/KaKs/complete2.csv")
colnames(data) <- c("gene_a", "gene_b", "ka", "ks")
tag_df2 <- read.table("./tmp/Glycine_max_TAGs_coverage30_identity30.tsv", header=TRUE)
tag_def <- "tag1"
data$category <- apply(data, 1, function(row) {
  gene_a <- row["gene_a"]
  gene_b <- row["gene_b"]
  tag_a <- tag_df2[tag_df2["gene"] == gene_a, tag_def]
  tag_b <- tag_df2[tag_df2["gene"] == gene_b, tag_def]
  # Filter out pairs tag character(0)
    if (length(tag_a) == 0 || length(tag_b) == 0) {
        return("notag")
    } else if (tag_a == "-" || tag_b == "-") {
        return("notag")
    } else if (tag_a == tag_b) {
        return("tag")
    } else {
        return("notag")
    }
})
#+end_src

#+RESULTS:

#+begin_src R :session *R* :results graphics file :file results/plots/ks_density_tag_and_not_tag.png :width 8 :height 8 :res 200 :units cm
library(ggplot2)
library(scales)
theme_set(theme_gray(base_size = 6))
gg <- ggplot(data)
gg <- gg + geom_density(mapping=aes(x=ks, color=category))
gg <- gg + scale_x_discrete("Age (Ks)")
gg <- gg + scale_y_continuous("Density")
gg <- gg + xlim(0, 5)
gg <- gg + ggtitle("Proportion of duplicate gene pairs age (TAG and non-TAG)")
gg <- gg + theme(plot.title = element_text(hjust = 0.5))
gg
#+end_src

#+RESULTS:
[[file:results/plots/ks_density_tag_and_not_tag.png]]

We will now test whether TAG gene duplication is more recent than non-TAG gene pairs.
As the distributions of the Ks values is not normally distributed, I will not use the Student \(t\)-test, but the Wilcoxon-Mann-Whitney U test.

The hypotheses are
\[
\begin{cases}
(H_0) & \text{for randomly selected values $X$ and $Y$ from `non-TAG' and `TAG' respectively, the probability of $X$ to be greater than $Y$ is equal to the probability of $Y$ being greater than $X$} \\
(H_1) & \text{for randomly selected values $X$ and $Y$ from `non-TAG' and `TAG' respectively, the probability of $X$ to be greater than $Y$ is greater than the probability of $Y$ being greater than $X$}
\end{cases}
\]

#+begin_src R :session *R* :results output :exports both
tag_ks <- data[data$category == "tag", "ks"]
non_tag_ks <- data[data$category == "notag", "ks"]
wilcox.test(non_tag_ks, tag_ks, alternative="greater")
#+end_src

#+RESULTS:
:
: 	Wilcoxon rank sum test with continuity correction
:
: data:  non_tag_ks and tag_ks
: W = 358006116, p-value < 2.2e-16
: alternative hypothesis: true location shift is greater than 0

The \(p\)-value is lower than $0.05$ so at level \(\alpha = 5 \%\), we reject the null hypotheses. Given two duplicate gene pairs, the probably that a TAG gene pair duplication event occurred before the non-TAG gene pair duplication event is greater than the otherway around, remaining TAG genes tends to be more recently duplicated.

* Are genes inside a TAG orientated in the same way more often than random?

We did not keep the orientation of the genes in the TAGs file. Let extract the gene orientation first, and join this information to the TAGs file.
We will do our analysis on the less stringent dataset. (coverage >30%, identity >30%).

#+begin_src bash
awk 'BEGIN {
    OFS="\t"
    }
    /^>/ {
          locus=$3
          split(locus, arr, ":")
          orientation=arr[6]
          gene=$4
          sub("gene:", "", gene)
          print gene, orientation
    }' ./tmp/proteome_filtered.fa > ./tmp/filtered_gene_orientation.tsv
head ./tmp/filtered_gene_orientation.tsv
#+end_src

#+RESULTS:
| GLYMA_01G141900 | -1 |
| GLYMA_01G234000 | -1 |
| GLYMA_01G157500 | -1 |
| GLYMA_01G031500 | -1 |
| GLYMA_01G132000 | -1 |
| GLYMA_01G038100 | -1 |
| GLYMA_01G184300 |  1 |
| GLYMA_01G231700 |  1 |
| GLYMA_01G031400 |  1 |
| GLYMA_01G008100 |  1 |

#+begin_src bash
head ./tmp/Glycine_max
#+end_src

#+RESULTS:

#+begin_src bash
tail -n +1 ./tmp/filtered_gene_orientation.tsv | sort -d > ./tmp/filtered_gene_orientation_s.tsv
tail -n +1 ./tmp/Glycine_max_TAGs_coverage30_identity30.tsv  | sort -d > ./tmp/Glycine_max_TAGs_coverage30_identity30_s.tsv
join -t$'\t' -1 1 -2 1 ./tmp/filtered_gene_orientation_s.tsv ./tmp/Glycine_max_TAGs_coverage30_identity30_s.tsv > ./tmp/Glycine_max_TAGs_coverage30_identity30_oriented.tsv
#+end_src

#+RESULTS:

#+begin_src bash
head ./tmp/Glycine_max_TAGs_coverage30_identity30_oriented.tsv
#+end_src

#+RESULTS:
| GLYMA_01G000100 | -1 |    5506 | - | - |
| GLYMA_01G000200 | -1 | spacer1 | - | - |
| GLYMA_01G000300 |  1 | spacer2 | - | - |
| GLYMA_01G000400 | -1 |     146 | - | 1 |
| GLYMA_01G000500 |  1 | spacer3 | - | - |
| GLYMA_01G000600 |  1 |     146 | - | 1 |
| GLYMA_01G000700 |  1 | spacer4 | - | - |
| GLYMA_01G000800 |  1 |    3898 | - | - |
| GLYMA_01G000900 |  1 |    7954 | - | - |
| GLYMA_01G001000 |  1 |    7875 | - | - |

#+RESULTS:

#+begin_src R :session *R* :colnames yes
tag_df = read.table("./tmp/Glycine_max_TAGs_coverage30_identity30_oriented.tsv",
                    sep="\t")
colnames(tag_df) <- c("gene", "orientation", "family", "tag0", "tag1")
head(tag_df)
#+end_src

#+RESULTS:
| gene            | orientation | family  | tag0 | tag1 |
|-----------------+-------------+---------+------+------|
| GLYMA_01G000100 |          -1 | 5506    | -    | -    |
| GLYMA_01G000200 |          -1 | spacer1 | -    | -    |
| GLYMA_01G000300 |           1 | spacer2 | -    | -    |
| GLYMA_01G000400 |          -1 | 146     | -    | 1    |
| GLYMA_01G000500 |           1 | spacer3 | -    | -    |
| GLYMA_01G000600 |           1 | 146     | -    | 1    |

We perform a Fischer test on the contingency table:
|         | Coherent | Convergent | Divergent |
|---------+----------+------------+-----------|
| pair TAG0 |          |            |           |
| pair not TAG0 |          |            |           |



#+begin_src R :session *R*
get_strand <- function(gene) {
    return(
        tag_df[tag_df[,"gene"] == gene, "orientation"]
    )
}
get_convergence <- function(gene_a, gene_b) {
    strand_a <- get_strand(gene_a)
    strand_b <- get_strand(gene_b)
    if (strand_a == strand_b) {
        return('coherent')
    } else if (strand_a == -1) {
        return('divergent')
    } else if (strand_a == 1) {
        return('convergent')
    } else {
        message("Error: I do not understand why this does not fall in these case (coherent, divergent, convergent, what else?)")
    }
}
#+end_src

#+RESULTS:

#+begin_src R :session *R* :colnames yes :rownames yes
contingency_table <- matrix(0, nrow=2, ncol=3)
colnames(contingency_table) <- c("coherent", "convergent", "divergent")
rownames(contingency_table) <- c("tag", "nottag")
definition <- 0
for (i in 1:(nrow(tag_df) - (definition + 1))) {
  j <- i + 1
  gene_i <- tag_df[i, "gene"]
  gene_j <- tag_df[j, "gene"]
  convergence <- get_convergence(gene_i, gene_j)
  if (tag_df[i, "tag0"] == "-" || tag_df[j, "tag0"] == "-") {
    category <- "nottag"
  } else if (tag_df[i, "tag0"] == tag_df[j, "tag0"]) {
    category <- "tag"
  } else {
    category <- "nottag"
  }
  contingency_table[category, convergence] <- contingency_table[category, convergence] + 1
}
contingency_table
#+end_src

#+RESULTS:
|        | coherent | convergent | divergent |
|--------+----------+------------+-----------|
| tag    |     3353 |        342 |       361 |
| nottag |    25699 |      12926 |     12907 |


#+RESULTS:
: org_babel_R_eoe

#+begin_src R :session *R* :results output
fisher.test(contingency_table, workspace=2e6)
#+end_src

#+RESULTS:
:
: 	Fisher's Exact Test for Count Data
:
: data:  contingency_table
: p-value < 2.2e-16
: alternative hypothesis: two.sided


\(p\)-value < 2.2e-16, so at level $\alpha = 0.05$, we reject the null hypothesis: the orientation of a pair of gene is significantly different between TAG and non TAG pairs.
More specifically, TAG genes tends to be more coherent than non TAG genes.
* Does /Glycine max/ have big TAGs and which function are the big TAG gene implied?
<2024-12-26 Thu>

Uses Gprofiler2

#+begin_src R :session *R* :results silent
library(gprofiler2)
#+end_src

First we want to extract the list of genes belonging to the largest TAG:

We once again focus on the least stringent dataset, with the TAG₀ definition.
#+begin_src R :session *R*
tag_df_tag <- tag_df[tag_df["tag1"] != "-",]
tag1_count <- table(tag_df_tag["tag1"])
largest_tag1 <- names(tag1_count[which(tag1_count == max(tag1_count))])
#+end_src

#+RESULTS:
: 525

#+begin_src R :session *R*
largest_tag1_genes <- tag_df_tag[tag_df_tag["tag1"] == largest_tag1, "gene"]
gostres <- gost(query=largest_tag1_genes, organism="gmax")
head(gostres$result)
#+end_src

#+RESULTS:

#+begin_src R :session *R*  :results graphics file :file results/plots/gprofiler_tag0579_enrichment_plot.png :width 8 :height 8 :res 200 :units cm
p <- gostplot(gostres, capped=FALSE, interactive=FALSE)
p
#+end_src

#+RESULTS:
[[file:results/plots/gprofiler_tag0579_enrichment_plot.png]]

Our genes are mostly involved in auxin related signaling pathways, and response to hormones.

#+begin_example
                                                     term_name
1                           regulation of cellular respiration
2 regulation of generation of precursor metabolites and energy
3                                         cellular respiration
4          energy derivation by oxidation of organic compounds
5               generation of precursor metabolites and energy
6                     regulation of cellular metabolic process
#+end_example

* TAG size in function of TAG definition
<2024-12-27 Fri>
#+begin_src bash
./rust/tagfinder/target/release/tagfinder --families  ./tmp/Glycine_max_Blastp_filtered_coverage30_identity30.mcl.tsv --positions ./tmp/Glycine_max.gene_positions.tsv --definitions 0,1,2,3,4,5,6,7,8,9,10,11,12,13,15,15,16,17,18,19,20,25,30,35,40,45,50 > "./tmp/Glycine_max_TAGs_coverage30_identity30_v2.tsv"
#+end_src

#+RESULTS:

#+begin_src R :session *R*  :results graphics file  :file results/plots/nb_TAG_against_definition.png :width 8 :height 8 :res 200 :units cm :exports both
library(ggplot2)
library(scales)

data <- read.table("./tmp/Glycine_max_TAGs_coverage30_identity30_v2.tsv", na.strings=c("-"), header=TRUE)

definitions <- c(0,1,2,3,4,5,6,7,8,9,10,11,12,13,15,15,16,17,18,19,20,25,30,35,40,45,50)

nb_TAGs <- sapply(definitions, function(definition) {
  tags <- data[,paste0("tag", definition)]
  tags <- tags[! is.na(tags)]
  return(max(tags))
    })

data <- data.frame(list(definition=definitions, nb=nb_TAGs))

theme_set(theme_gray(base_size = 8))
gg <- ggplot(data, aes(x=definition, y=nb))
gg <- gg + geom_point()
gg <- gg + scale_x_continuous("Definition")
gg <- gg + scale_y_continuous("TAGs")
gg <- gg + ggtitle("Number of TAGs for a TAG definition")
gg <- gg + theme(plot.title = element_text(hjust = 0.5))
                      gg
#+end_src

#+RESULTS:
[[file:results/plots/nb_TAG_against_definition.pdf]]

* Computing some statistics on a selection of plants

<2024-12-29 Sun>

We want to extract the following statistics for the whole set of plant on eCampus:
- number of families
- number of duplicate genes
- number of singletons
- number of TAG0


We have to filter out some genes that are from Mitochondria or Chloroplasts (for /Oryza sativa ssp. Japonica/ for instance).

First we extract the protein ID that corresponds to mitochondrial or chloroplastic genes.
#+begin_src bash
fasta="/media/data/sync/Documents/course/master/M2/S1/comparative_genomics/data/Oryza_sativa.IRGSP-1.0.pep.all.fa.gz"
zcat "${fasta}" | awk '
     /^>/ {
     locus=$3
     split(":", arr, locus)
     chromosome=arr[3]
     if (chromosome == "Mt" || chromosome == "Pt" || chromosome ~ /^supercontig*/) {
        protein=$1
        gsub(">", "", protein)
        print protein
     }
    }' > ./tmp/Oryza_sativa_faulty_proteins.list
#+end_src

#+RESULTS:

#+begin_src bash
head ./tmp/Oryza_sativa_faulty_proteins.list
#+end_src

#+RESULTS:

Then, remove all BLASTp hits that matches a protein ID in either query or subject.

We include this step in a Nextflow workflow. It does not compute the BLASTp all against all, It does rather use the TSV file provided on eCampus.
