diff --git a/.codeboarding/Genomic Data Loaders.md b/.codeboarding/Genomic Data Loaders.md new file mode 100644 index 0000000..fa4a343 --- /dev/null +++ b/.codeboarding/Genomic Data Loaders.md @@ -0,0 +1,140 @@ +```mermaid +graph LR + Genomic_Dataclasses["Genomic Dataclasses"] + Sequence_Transformers["Sequence Transformers"] + Genomic_Data_Loaders["Genomic Data Loaders"] + Multi_Interval_Extractors["Multi-Interval Extractors"] + VCF_Processors["VCF Processors"] + GTF_Feature_Fetchers["GTF Feature Fetchers"] + Variant_Sequence_Generators["Variant Sequence Generators"] + Protein_Sequence_Processors["Protein Sequence Processors"] + Genomic_Data_Loaders -- "uses" --> Genomic_Dataclasses + Genomic_Data_Loaders -- "applies" --> Sequence_Transformers + Genomic_Data_Loaders -- "extracts via" --> Multi_Interval_Extractors + Multi_Interval_Extractors -- "operates on" --> Genomic_Dataclasses + VCF_Processors -- "produces" --> Genomic_Dataclasses + VCF_Processors -- "queries with" --> Genomic_Dataclasses + VCF_Processors -- "integrates" --> Variant_Sequence_Generators + GTF_Feature_Fetchers -- "produces" --> Genomic_Dataclasses + Protein_Sequence_Processors -- "utilizes" --> GTF_Feature_Fetchers + Protein_Sequence_Processors -- "processes with" --> VCF_Processors + Protein_Sequence_Processors -- "translates using" --> Sequence_Transformers + Variant_Sequence_Generators -- "modifies" --> Genomic_Dataclasses + Variant_Sequence_Generators -- "depends on" --> Sequence_Transformers + Multi_Interval_Extractors -- "uses" --> Sequence_Transformers + Protein_Sequence_Processors -- "uses" --> Multi_Interval_Extractors +``` +[![CodeBoarding](https://img.shields.io/badge/Generated%20by-CodeBoarding-9cf?style=flat-square)](https://github.com/CodeBoarding/GeneratedOnBoardings)[![Demo](https://img.shields.io/badge/Try%20our-Demo-blue?style=flat-square)](https://www.codeboarding.org/demo)[![Contact](https://img.shields.io/badge/Contact%20us%20-%20contact@codeboarding.org-lightgrey?style=flat-square)](mailto:contact@codeboarding.org) + +## Component Details + +The `kipoiseq` library provides a comprehensive framework for handling genomic data, focusing on sequence manipulation and variant analysis. It defines core genomic data structures, offers various data loaders for different genomic data types (intervals, BED, GTF, splicing, protein), and includes robust components for sequence transformations, multi-interval extraction, VCF processing, GTF feature fetching, variant sequence generation, and protein sequence processing. The system's main flow involves loading genomic data, transforming sequences, extracting relevant regions, and integrating variant information to generate processed sequences for downstream analysis, particularly in machine learning applications. + +### Genomic Dataclasses +This component defines fundamental data structures for representing genomic entities such as Variants and Intervals. These classes provide a standardized way to store and manipulate genomic coordinates and variant information, serving as basic building blocks for other components in the library. + + +**Related Classes/Methods**: + +- `kipoiseq.kipoiseq.dataclasses.Variant` (17:149) +- `kipoiseq.kipoiseq.dataclasses.Interval` (152:374) + + +### Sequence Transformers +This component encompasses functions and classes responsible for various sequence transformations. This includes operations like one-hot encoding DNA/RNA sequences, resizing genomic intervals, padding, trimming, and reverse complementing sequences, which are crucial for preparing data for machine learning models. + + +**Related Classes/Methods**: + +- `kipoiseq.kipoiseq.transforms.transforms.ReorderedOneHot` (127:225) +- `kipoiseq.kipoiseq.transforms.transforms.OneHot` (91:124) +- `kipoiseq.kipoiseq.transforms.transforms.ResizeInterval` (77:86) +- `kipoiseq.kipoiseq.transforms.functional.one_hot2string` (26:32) +- `kipoiseq.kipoiseq.transforms.functional.tokenize` (72:96) +- `kipoiseq.kipoiseq.transforms.functional.one_hot` (112:115) +- `kipoiseq.kipoiseq.transforms.functional.one_hot_dna` (118:133) +- `kipoiseq.kipoiseq.transforms.functional.fixed_len` (177:230) +- `kipoiseq.kipoiseq.transforms.functional.rc_dna` (35:51) +- `kipoiseq.kipoiseq.transforms.functional.translate` (300:322) + + +### Genomic Data Loaders +This component provides various data loaders for handling genomic sequences based on intervals, BED files, and GTF annotations, supporting both string and one-hot encoded sequence outputs, including specialized loaders for splicing and protein data. + + +**Related Classes/Methods**: + +- `kipoiseq.kipoiseq.dataloaders.sequence.SeqIntervalDl` (268:384) +- `kipoiseq.kipoiseq.dataloaders.sequence.StringSeqIntervalDl` (135:261) +- `kipoiseq.kipoiseq.dataloaders.sequence.BedDataset` (26:131) +- `kipoiseq.kipoiseq.dataloaders.splicing.MMSpliceDl` (182:276) +- `kipoiseq.kipoiseq.dataloaders.splicing.ExonInterval` (32:129) +- `kipoiseq.kipoiseq.dataloaders.protein.SingleVariantProteinDataLoader` (30:164) +- `kipoiseq.kipoiseq.dataloaders.protein.SingleVariantUTRDataLoader` (168:356) + + +### Multi-Interval Extractors +This component offers a generic framework for extracting and concatenating sequences across multiple genomic intervals. It serves as a base for more specialized extractors, providing common functionalities for handling collections of intervals and their corresponding sequences. + + +**Related Classes/Methods**: + +- `kipoiseq.kipoiseq.extractors.multi_interval.BaseMultiIntervalSeqExtractor` (30:83) +- `kipoiseq.kipoiseq.extractors.multi_interval.GenericMultiIntervalSeqExtractor` (129:195) +- `kipoiseq.kipoiseq.extractors.multi_interval.BaseMultiIntervalVCFSeqExtractor` (202:307) + + +### VCF Processors +This component is dedicated to processing Variant Call Format (VCF) files. It provides functionalities for reading, querying, and filtering genomic variants, enabling the retrieval of specific variants or batches of variants based on genomic regions or sample IDs. + + +**Related Classes/Methods**: + +- `kipoiseq.kipoiseq.extractors.vcf_query.VariantIntervalQueryable` (113:281) +- `kipoiseq.kipoiseq.extractors.vcf.MultiSampleVCF` (21:211) +- `kipoiseq.kipoiseq.extractors.vcf_matching.PyrangesVariantFetcher` (86:110) +- `kipoiseq.kipoiseq.extractors.vcf_matching.BaseVariantMatcher` (113:196) +- `kipoiseq.kipoiseq.extractors.vcf_matching.SingleVariantMatcher` (199:258) + + +### GTF Feature Fetchers +This component specializes in extracting specific genomic features, such as Coding Sequences (CDS) and Untranslated Regions (UTRs), from GTF annotation files. It provides methods to filter and retrieve intervals corresponding to these features, often used in gene and protein analysis. + + +**Related Classes/Methods**: + +- `kipoiseq.kipoiseq.extractors.gtf.GTFMultiIntervalFetcher` (88:151) +- `kipoiseq.kipoiseq.extractors.gtf.CDSFetcher` (154:265) +- `kipoiseq.kipoiseq.extractors.gtf.UTRFetcher` (268:377) + + +### Variant Sequence Generators +This component focuses on generating DNA sequences that incorporate genomic variants. It handles the complex logic of applying variants (SNVs, indels) to a reference sequence, managing overlaps, and ensuring correct sequence length and strand orientation. + + +**Related Classes/Methods**: + +- `kipoiseq.kipoiseq.extractors.vcf_seq.IntervalSeqBuilder` (23:57) +- `kipoiseq.kipoiseq.extractors.vcf_seq.VariantSeqExtractor` (60:303) +- `kipoiseq.kipoiseq.extractors.vcf_seq._BaseVCFSeqExtractor` (306:325) +- `kipoiseq.kipoiseq.extractors.vcf_seq.SingleVariantVCFSeqExtractor` (328:341) +- `kipoiseq.kipoiseq.extractors.vcf_seq.SingleSeqVCFSeqExtractor` (344:355) + + +### Protein Sequence Processors +This component is designed for extracting and translating protein sequences from genomic data, with the capability to incorporate genetic variants. It leverages GTF annotations to identify coding regions and VCF data to introduce variants, ultimately producing amino acid sequences. + + +**Related Classes/Methods**: + +- `kipoiseq.kipoiseq.extractors.protein.UTRSeqExtractor` (35:69) +- `kipoiseq.kipoiseq.extractors.protein.TranscriptSeqExtractor` (112:166) +- `kipoiseq.kipoiseq.extractors.protein.ProteinSeqExtractor` (169:180) +- `kipoiseq.kipoiseq.extractors.protein.TranscriptVCFSeqExtractor` (183:207) +- `kipoiseq.kipoiseq.extractors.protein.ProteinVCFSeqExtractor` (210:258) +- `kipoiseq.kipoiseq.extractors.protein.cut_transcript_seq` (72:108) + + + + +### [FAQ](https://github.com/CodeBoarding/GeneratedOnBoardings/tree/main?tab=readme-ov-file#faq) \ No newline at end of file diff --git a/.codeboarding/Genomic Data Models.md b/.codeboarding/Genomic Data Models.md new file mode 100644 index 0000000..4082fd0 --- /dev/null +++ b/.codeboarding/Genomic Data Models.md @@ -0,0 +1,93 @@ +```mermaid +graph LR + GenomicDataStructures["GenomicDataStructures"] + VCFDataExtractor["VCFDataExtractor"] + VariantQueryingSystem["VariantQueryingSystem"] + SyntheticVariantGenerator["SyntheticVariantGenerator"] + VariantSequenceExtractor["VariantSequenceExtractor"] + GenomicRegionDataLoaders["GenomicRegionDataLoaders"] + VCFDataExtractor -- "produces" --> GenomicDataStructures + VariantQueryingSystem -- "processes" --> GenomicDataStructures + SyntheticVariantGenerator -- "generates" --> GenomicDataStructures + VariantSequenceExtractor -- "consumes" --> GenomicDataStructures + GenomicRegionDataLoaders -- "creates" --> GenomicDataStructures + VCFDataExtractor -- "provides query interface to" --> VariantQueryingSystem + VariantSequenceExtractor -- "depends on" --> VCFDataExtractor + GenomicRegionDataLoaders -- "utilizes" --> VariantSequenceExtractor +``` +[![CodeBoarding](https://img.shields.io/badge/Generated%20by-CodeBoarding-9cf?style=flat-square)](https://github.com/CodeBoarding/GeneratedOnBoardings)[![Demo](https://img.shields.io/badge/Try%20our-Demo-blue?style=flat-square)](https://www.codeboarding.org/demo)[![Contact](https://img.shields.io/badge/Contact%20us%20-%20contact@codeboarding.org-lightgrey?style=flat-square)](mailto:contact@codeboarding.org) + +## Component Details + +This graph illustrates the core components of the kipoiseq library, focusing on how genomic data structures are defined, extracted, queried, generated, and loaded. The central `GenomicDataStructures` component provides fundamental data types like `Variant` and `Interval`, which are consumed and produced by other components. `VCFDataExtractor` handles reading VCF files, feeding data to `VariantQueryingSystem` for filtering and `VariantSequenceExtractor` for sequence generation. `SyntheticVariantGenerator` creates new variants, while `GenomicRegionDataLoaders` manage loading various genomic file formats and preparing data for analysis, often utilizing the `VariantSequenceExtractor` for sequence processing. + +### GenomicDataStructures +This component defines fundamental data structures for representing genomic information, specifically `Variant` for genetic variations and `Interval` for genomic regions. These classes provide methods for creation, manipulation (e.g., shifting, resizing intervals), and conversion from/to other formats. They serve as the foundational data types used across various other components in the kipoiseq library. + + +**Related Classes/Methods**: + +- `kipoiseq.dataclasses.Variant` (17:149) +- `kipoiseq.dataclasses.Interval` (152:374) + + +### VCFDataExtractor +This component is responsible for reading, parsing, and fetching variant information from VCF (Variant Call Format) files. It provides methods to iterate through variants, fetch specific variants by interval or ID, and check for the presence of variants in samples. It relies on the `GenomicDataStructures` component for representing variants and intervals. + + +**Related Classes/Methods**: + +- `kipoiseq.extractors.vcf.MultiSampleVCF` (21:211) + + +### VariantQueryingSystem +This component provides a flexible framework for querying and filtering genomic variants, often in conjunction with genomic intervals. It allows for defining custom query logic and applying it to collections of variants. It heavily utilizes the `GenomicDataStructures` for defining query parameters and results. + + +**Related Classes/Methods**: + +- `kipoiseq.extractors.vcf_query.VariantIntervalQueryable` (113:281) +- `kipoiseq.extractors.vcf_query.VariantQuery` (33:45) +- `kipoiseq.extractors.vcf_query.FilterVariantQuery` (48:54) +- `kipoiseq.extractors.vcf_query.VariantIntervalQuery` (83:89) +- `kipoiseq.extractors.vcf_query.NumberVariantQuery` (92:107) + + +### SyntheticVariantGenerator +This component is designed to generate synthetic single nucleotide variants (SNVs), insertions, and deletions within specified genomic intervals. It uses a FASTA file for reference sequences and can generate combinations of variants. The generated variants are represented using the `GenomicDataStructures` component. + + +**Related Classes/Methods**: + +- `kipoiseq.extractors.variant_combinations.VariantCombinator` (9:98) + + +### VariantSequenceExtractor +This component extracts DNA sequences from a reference genome, applying genetic variants (SNVs, indels) to generate alternative sequences. It handles complex scenarios like overlapping variants and ensures fixed-length output. It consumes `GenomicDataStructures` (Variants and Intervals) and often depends on `VCFDataExtractor` to get variant information. + + +**Related Classes/Methods**: + +- `kipoiseq.extractors.vcf_seq.VariantSeqExtractor` (60:303) +- `kipoiseq.extractors.vcf_seq.IntervalSeqBuilder` (23:57) +- `kipoiseq.extractors.vcf_seq.SingleVariantVCFSeqExtractor` (328:341) +- `kipoiseq.extractors.vcf_seq.SingleSeqVCFSeqExtractor` (344:355) + + +### GenomicRegionDataLoaders +This component provides various data loaders for reading genomic data from different file formats, such as BED and GTF, and preparing it for downstream analysis. It includes functionalities for resizing intervals and converting sequences to one-hot encoding. It creates and manipulates `GenomicDataStructures` (Intervals) and can utilize `VariantSequenceExtractor` for sequence generation. + + +**Related Classes/Methods**: + +- `kipoiseq.dataloaders.sequence.AnchoredGTFDl` (387:539) +- `kipoiseq.dataloaders.sequence.StringSeqIntervalDl` (135:261) +- `kipoiseq.dataloaders.sequence.SeqIntervalDl` (268:384) +- `kipoiseq.dataloaders.sequence.BedDataset` (26:131) +- `kipoiseq.extractors.gtf.gtf_row2interval` (70:85) +- `kipoiseq.extractors.vcf_matching.pyranges_to_intervals` (42:60) + + + + +### [FAQ](https://github.com/CodeBoarding/GeneratedOnBoardings/tree/main?tab=readme-ov-file#faq) \ No newline at end of file diff --git a/.codeboarding/Genomic Feature Extractors.md b/.codeboarding/Genomic Feature Extractors.md new file mode 100644 index 0000000..5c75eb5 --- /dev/null +++ b/.codeboarding/Genomic Feature Extractors.md @@ -0,0 +1,97 @@ +```mermaid +graph LR + Genomic_Feature_Extractors["Genomic Feature Extractors"] + Sequence_Extractor["Sequence Extractor"] + Multi_Interval_Sequence_Extractor["Multi-Interval Sequence Extractor"] + GTF_Feature_Fetcher["GTF Feature Fetcher"] + Protein_Sequence_Extractor["Protein Sequence Extractor"] + Sequence_Transformation["Sequence Transformation"] + Genomic_Feature_Extractors -- "uses" --> Sequence_Extractor + Genomic_Feature_Extractors -- "uses" --> Multi_Interval_Sequence_Extractor + Genomic_Feature_Extractors -- "uses" --> GTF_Feature_Fetcher + Genomic_Feature_Extractors -- "uses" --> Protein_Sequence_Extractor + Multi_Interval_Sequence_Extractor -- "extends" --> Sequence_Extractor + Multi_Interval_Sequence_Extractor -- "uses" --> Sequence_Transformation + GTF_Feature_Fetcher -- "extends" --> Multi_Interval_Sequence_Extractor + GTF_Feature_Fetcher -- "uses" --> Sequence_Transformation + Protein_Sequence_Extractor -- "extends" --> Multi_Interval_Sequence_Extractor + Protein_Sequence_Extractor -- "uses" --> Sequence_Transformation +``` +[![CodeBoarding](https://img.shields.io/badge/Generated%20by-CodeBoarding-9cf?style=flat-square)](https://github.com/CodeBoarding/GeneratedOnBoardings)[![Demo](https://img.shields.io/badge/Try%20our-Demo-blue?style=flat-square)](https://www.codeboarding.org/demo)[![Contact](https://img.shields.io/badge/Contact%20us%20-%20contact@codeboarding.org-lightgrey?style=flat-square)](mailto:contact@codeboarding.org) + +## Component Details + +The Genomic Feature Extractors component provides a comprehensive set of functionalities for extracting various types of genomic data. It integrates specialized extractors to retrieve DNA sequences from FASTA files, specific features like CDS and UTRs from GTF files, and protein sequences. This component is designed to handle extractions across multiple genomic intervals and supports sequence transformations such as reverse complementation and translation, serving as a core utility for downstream genomic analysis. + +### Genomic Feature Extractors +Extracts genomic sequences from FASTA files and features from GTF files (e.g., CDS, UTRs). It also provides functionalities for extracting sequences across multiple genomic intervals and protein sequences. This component serves as a unified interface for various specialized sequence and feature extraction functionalities within the kipoiseq library. + + +**Related Classes/Methods**: + +- `kipoiseq.extractors.fasta.FastaStringExtractor` (7:63) +- `kipoiseq.extractors.gtf.CDSFetcher` (154:265) +- `kipoiseq.extractors.gtf.UTRFetcher` (268:377) +- `kipoiseq.extractors.gtf.GTFMultiIntervalFetcher` (88:151) +- `kipoiseq.extractors.protein.ProteinSeqExtractor` (169:180) +- `kipoiseq.extractors.protein.TranscriptSeqExtractor` (112:166) +- `kipoiseq.extractors.protein.UTRSeqExtractor` (35:69) +- `kipoiseq.extractors.multi_interval.GenericMultiIntervalSeqExtractor` (129:195) + + +### Sequence Extractor +The Sequence Extractor component is responsible for extracting DNA sequences from FASTA files based on specified genomic intervals. It can handle strand information and force uppercase output. This component is a fundamental building block for other extractors and data loaders. + + +**Related Classes/Methods**: + +- `kipoiseq.extractors.fasta.FastaStringExtractor` (7:63) + + +### Multi-Interval Sequence Extractor +The Multi-Interval Sequence Extractor component is designed to extract and concatenate sequences across multiple genomic intervals. It serves as a base class for more specialized extractors that handle variants or specific genomic features like CDS or UTRs. It can also apply reverse complementation if strand information is provided. + + +**Related Classes/Methods**: + +- `kipoiseq.extractors.multi_interval.BaseMultiIntervalSeqExtractor` (30:83) +- `kipoiseq.extractors.multi_interval.GenericMultiIntervalSeqExtractor` (129:195) + + +### GTF Feature Fetcher +The GTF Feature Fetcher component is responsible for parsing GTF (Gene Transfer Format) files and extracting specific genomic features like Coding Sequences (CDS) and Untranslated Regions (UTRs). It can filter features based on various criteria such as biotype and transcript support level. + + +**Related Classes/Methods**: + +- `kipoiseq.extractors.gtf.GTFMultiIntervalFetcher` (88:151) +- `kipoiseq.extractors.gtf.CDSFetcher` (154:265) +- `kipoiseq.extractors.gtf.UTRFetcher` (268:377) + + +### Protein Sequence Extractor +The Protein Sequence Extractor component is responsible for extracting protein and transcript sequences from genomic data, often utilizing GTF and FASTA files. It can handle translation from DNA to amino acid sequences. + + +**Related Classes/Methods**: + +- `kipoiseq.extractors.protein.ProteinSeqExtractor` (169:180) +- `kipoiseq.extractors.protein.TranscriptSeqExtractor` (112:166) +- `kipoiseq.extractors.protein.UTRSeqExtractor` (35:69) + + +### Sequence Transformation +The Sequence Transformation component provides functional utilities for manipulating DNA sequences. This includes operations like reverse complementing DNA, one-hot encoding sequences, padding, trimming, and translating DNA to amino acid sequences. + + +**Related Classes/Methods**: + +- `kipoiseq.transforms.functional.rc_dna` (35:51) +- `kipoiseq.transforms.functional.one_hot_dna` (118:133) +- `kipoiseq.transforms.functional.resize_interval` (233:256) +- `kipoiseq.transforms.functional.translate` (300:322) + + + + +### [FAQ](https://github.com/CodeBoarding/GeneratedOnBoardings/tree/main?tab=readme-ov-file#faq) \ No newline at end of file diff --git a/.codeboarding/Sequence Processing.md b/.codeboarding/Sequence Processing.md new file mode 100644 index 0000000..d11a360 --- /dev/null +++ b/.codeboarding/Sequence Processing.md @@ -0,0 +1,135 @@ +```mermaid +graph LR + Variant_Data_Fetchers["Variant Data Fetchers"] + Interval_Data_Loaders["Interval Data Loaders"] + Sequence_Extractors["Sequence Extractors"] + Sequence_and_Interval_Transforms["Sequence and Interval Transforms"] + Functional_Transformations["Functional Transformations"] + Utility_Functions["Utility Functions"] + Sequence_Processing["Sequence Processing"] + Variant_Data_Fetchers -- "uses" --> Utility_Functions + Interval_Data_Loaders -- "loads data for" --> Sequence_Extractors + Interval_Data_Loaders -- "applies" --> Functional_Transformations + Interval_Data_Loaders -- "uses" --> Utility_Functions + Sequence_and_Interval_Transforms -- "implements" --> Functional_Transformations + Sequence_and_Interval_Transforms -- "uses" --> Utility_Functions + Sequence_Processing -- "uses" --> Functional_Transformations + Sequence_Processing -- "uses" --> Utility_Functions + Sequence_Processing -- "transforms" --> Sequence_and_Interval_Transforms +``` +[![CodeBoarding](https://img.shields.io/badge/Generated%20by-CodeBoarding-9cf?style=flat-square)](https://github.com/CodeBoarding/GeneratedOnBoardings)[![Demo](https://img.shields.io/badge/Try%20our-Demo-blue?style=flat-square)](https://www.codeboarding.org/demo)[![Contact](https://img.shields.io/badge/Contact%20us%20-%20contact@codeboarding.org-lightgrey?style=flat-square)](mailto:contact@codeboarding.org) + +## Component Details + +This graph illustrates the key components of the kipoiseq library, focusing on how biological sequence data is fetched, loaded, transformed, and processed for machine learning applications. The main flow involves fetching variant data, loading genomic intervals, extracting sequences based on these intervals, and then applying various transformations to prepare the data. Utility functions are leveraged across multiple components for common tasks. + +### Variant Data Fetchers +This component provides an abstract base class for fetching genetic variants and a concrete implementation for multi-sample VCF files. It allows iterating through variants in batches and querying specific variants or regions. + + +**Related Classes/Methods**: + +- `kipoiseq.kipoiseq.variant_source.VariantFetcher` (8:39) +- `kipoiseq.kipoiseq.variant_source.VariantFetcher:batch_iter` (23:30) +- `kipoiseq.kipoiseq.variant_source.VariantFetcher:fetch_variants` (14:21) +- `kipoiseq.kipoiseq.variant_source.VariantFetcher:__iter__` (33:39) +- `kipoiseq.kipoiseq.extractors.vcf.MultiSampleVCF` (21:211) +- `kipoiseq.kipoiseq.extractors.vcf.MultiSampleVCF:batch_iter` (69:77) +- `kipoiseq.kipoiseq.extractors.vcf.MultiSampleVCF:fetch_variants` (28:32) +- `kipoiseq.kipoiseq.extractors.vcf.MultiSampleVCF:__init__` (23:26) +- `kipoiseq.kipoiseq.extractors.vcf.MultiSampleVCF:_variants_from_cyvcf2` (34:44) +- `kipoiseq.kipoiseq.extractors.vcf.MultiSampleVCF:has_variant` (50:52) +- `kipoiseq.kipoiseq.extractors.vcf.MultiSampleVCF:query_variants` (79:105) +- `kipoiseq.kipoiseq.extractors.vcf.MultiSampleVCF:get_variant` (123:145) +- `kipoiseq.kipoiseq.extractors.vcf.MultiSampleVCF:get_samples` (201:211) + + +### Interval Data Loaders +This component is responsible for loading genomic interval data from various file formats like BED and GTF. It provides functionalities to read intervals, associated labels, and anchor points for sequence extraction. + + +**Related Classes/Methods**: + +- `kipoiseq.kipoiseq.dataloaders.sequence.BedDataset` (26:131) +- `kipoiseq.kipoiseq.dataloaders.sequence.BedDataset:__init__` (60:108) +- `kipoiseq.kipoiseq.dataloaders.sequence.BedDataset:__getitem__` (110:125) +- `kipoiseq.kipoiseq.dataloaders.sequence.StringSeqIntervalDl` (135:261) +- `kipoiseq.kipoiseq.dataloaders.sequence.StringSeqIntervalDl:__init__` (187:217) +- `kipoiseq.kipoiseq.dataloaders.sequence.StringSeqIntervalDl:__getitem__` (222:252) +- `kipoiseq.kipoiseq.dataloaders.sequence.AnchoredGTFDl` (387:539) +- `kipoiseq.kipoiseq.dataloaders.sequence.AnchoredGTFDl:__init__` (456:488) +- `kipoiseq.kipoiseq.dataloaders.sequence.AnchoredGTFDl:__getitem__` (507:519) +- `kipoiseq.kipoiseq.dataloaders.sequence.AnchoredGTFDl._create_anchored_interval` (490:502) + + +### Sequence Extractors +This component provides tools for extracting DNA sequences from FASTA files based on genomic intervals. It supports strand-aware extraction and forcing uppercase output. + + +**Related Classes/Methods**: + +- `kipoiseq.extractors.fasta.FastaStringExtractor` (7:63) +- `kipoiseq.extractors.fasta.FastaStringExtractor:__init__` (21:27) +- `kipoiseq.extractors.fasta.FastaStringExtractor:extract` (29:60) + + +### Sequence and Interval Transforms +This component offers a collection of classes for transforming genomic intervals and DNA sequences. It includes operations like resizing intervals, one-hot encoding sequences, and reordering axes for different data formats. + + +**Related Classes/Methods**: + +- `kipoiseq.kipoiseq.transforms.transforms.ResizeInterval` (77:86) +- `kipoiseq.kipoiseq.transforms.transforms.ResizeInterval:__call__` (85:86) +- `kipoiseq.kipoiseq.transforms.transforms.OneHot` (91:124) +- `kipoiseq.kipoiseq.transforms.transforms.OneHot:__call__` (112:124) +- `kipoiseq.kipoiseq.transforms.transforms.ReorderedOneHot` (127:225) +- `kipoiseq.kipoiseq.transforms.transforms.ReorderedOneHot:__init__` (146:190) +- `kipoiseq.kipoiseq.transforms.transforms.Compose` (10:38) +- `kipoiseq.kipoiseq.transforms.transforms.DummyAxis` (43:54) +- `kipoiseq.kipoiseq.transforms.transforms.SwapAxes` (57:71) + + +### Functional Transformations +This component contains core, stateless functions for performing common transformations on sequences and intervals, such as one-hot encoding DNA, resizing intervals, and reverse complementing sequences. + + +**Related Classes/Methods**: + +- `kipoiseq.kipoiseq.transforms.functional` (full file reference) +- `kipoiseq.kipoiseq.transforms.functional:one_hot_dna` (118:133) +- `kipoiseq.kipoiseq.transforms.functional:one_hot` (112:115) +- `kipoiseq.kipoiseq.transforms.functional:resize_interval` (233:256) + + +### Utility Functions +This component provides various utility functions used across the kipoiseq library, including batch iteration, type parsing, and scalar conversion. + + +**Related Classes/Methods**: + +- `kipoiseq.utils` (full file reference) +- `kipoiseq.utils.batch_iter` (48:57) +- `kipoiseq.utils.to_scalar` (18:24) +- `kipoiseq.utils.parse_alphabet` (27:31) +- `kipoiseq.utils.parse_dtype` (34:42) + + +### Sequence Processing +Offers a suite of functions and classes for transforming biological sequences, including one-hot encoding, resizing intervals, and handling sequence axes, crucial for preparing data for machine learning models. It also includes general-purpose helper functions. + + +**Related Classes/Methods**: + +- `kipoiseq.kipoiseq.transforms.transforms.OneHot` (91:124) +- `kipoiseq.kipoiseq.transforms.functional.one_hot_dna` (118:133) +- `kipoiseq.kipoiseq.transforms.transforms.ResizeInterval` (77:86) +- `kipoiseq.utils.batch_iter` (48:57) +- `kipoiseq.utils.parse_alphabet` (27:31) +- `kipoiseq.utils.parse_dtype` (34:42) +- `kipoiseq.utils.to_scalar` (18:24) + + + + +### [FAQ](https://github.com/CodeBoarding/GeneratedOnBoardings/tree/main?tab=readme-ov-file#faq) \ No newline at end of file diff --git a/.codeboarding/Variant Data Management.md b/.codeboarding/Variant Data Management.md new file mode 100644 index 0000000..dc8e7c7 --- /dev/null +++ b/.codeboarding/Variant Data Management.md @@ -0,0 +1,125 @@ +```mermaid +graph LR + VariantSource["VariantSource"] + MultiSampleVCFHandler["MultiSampleVCFHandler"] + VCFQueryAPI["VCFQueryAPI"] + VariantMatchingEngine["VariantMatchingEngine"] + VariantCombinationGenerator["VariantCombinationGenerator"] + VCFSequenceExtractor["VCFSequenceExtractor"] + ProteinDataLoader["ProteinDataLoader"] + ProteinVCFSequenceExtractor["ProteinVCFSequenceExtractor"] + MultiIntervalVCFSequenceExtractor["MultiIntervalVCFSequenceExtractor"] + MultiSampleVCFHandler -- "inherits from" --> VariantSource + MultiSampleVCFHandler -- "provides query interface to" --> VCFQueryAPI + VCFQueryAPI -- "queries" --> MultiSampleVCFHandler + VariantMatchingEngine -- "reads variants from" --> MultiSampleVCFHandler + VariantMatchingEngine -- "utilizes" --> VariantSource + VCFSequenceExtractor -- "accesses" --> MultiSampleVCFHandler + ProteinDataLoader -- "initializes with" --> MultiSampleVCFHandler + ProteinDataLoader -- "initializes with" --> VariantMatchingEngine + ProteinVCFSequenceExtractor -- "initializes with" --> MultiSampleVCFHandler + ProteinVCFSequenceExtractor -- "initializes with" --> VariantMatchingEngine +``` +[![CodeBoarding](https://img.shields.io/badge/Generated%20by-CodeBoarding-9cf?style=flat-square)](https://github.com/CodeBoarding/GeneratedOnBoardings)[![Demo](https://img.shields.io/badge/Try%20our-Demo-blue?style=flat-square)](https://www.codeboarding.org/demo)[![Contact](https://img.shields.io/badge/Contact%20us%20-%20contact@codeboarding.org-lightgrey?style=flat-square)](mailto:contact@codeboarding.org) + +## Component Details + +The Variant Data Management subsystem in `kipoiseq` is designed to handle comprehensive operations on Variant Call Format (VCF) files. It provides foundational capabilities for fetching and iterating genetic variants through the `VariantSource` and `MultiSampleVCFHandler` components. Complex querying and filtering of VCF data are managed by the `VCFQueryAPI`, while the `VariantMatchingEngine` facilitates matching variants against genomic intervals, often leveraging `pyranges`. Additionally, the system supports the generation of synthetic variant combinations via `VariantCombinationGenerator` and the extraction of genomic sequences around variants using `VCFSequenceExtractor`. Specialized components like `ProteinDataLoader` and `ProteinVCFSequenceExtractor` extend these core functionalities for protein-centric analyses, demonstrating the subsystem's adaptability in handling diverse genetic data processing needs. + +### VariantSource +Manages the initial fetching and iteration of genetic variants from various sources, serving as a foundational component for variant access. + + +**Related Classes/Methods**: + +- `kipoiseq.kipoiseq.variant_source.VariantFetcher` (8:39) + + +### MultiSampleVCFHandler +Provides core functionalities for reading, querying, and iterating through variants in multi-sample VCF files, including fetching specific variants or checking for variant presence. It extends VariantSource capabilities. + + +**Related Classes/Methods**: + +- `kipoiseq.kipoiseq.extractors.vcf.MultiSampleVCF` (21:211) + + +### VCFQueryAPI +Offers a programmatic interface for constructing and applying complex queries on VCF data, supporting logical operations (AND, OR) and interval-based filtering of variants. + + +**Related Classes/Methods**: + +- `kipoiseq.kipoiseq.extractors.vcf_query.VariantQuery` (33:45) +- `kipoiseq.kipoiseq.extractors.vcf_query.VariantIntervalQueryable` (113:281) +- `kipoiseq.kipoiseq.extractors.vcf_query.BaseVariantQuery` (19:30) +- `kipoiseq.kipoiseq.extractors.vcf_query.BaseVariantIntervalQuery` (57:80) + + +### VariantMatchingEngine +Responsible for matching variants against defined intervals or other variant sets, often utilizing pyranges for efficient genomic interval operations. It includes base matching functionalities and specialized single/multi-variant matchers. + + +**Related Classes/Methods**: + +- `kipoiseq.kipoiseq.extractors.vcf_matching.SingleVariantMatcher` (199:258) +- `kipoiseq.kipoiseq.extractors.vcf_matching.variants_to_pyranges` (22:39) +- `kipoiseq.kipoiseq.extractors.vcf_matching.intervals_to_pyranges` (63:83) +- `kipoiseq.kipoiseq.extractors.vcf_matching.PyrangesVariantFetcher` (86:110) +- `kipoiseq.kipoiseq.extractors.vcf_matching.BaseVariantMatcher` (113:196) +- `kipoiseq.kipoiseq.extractors.vcf_matching.MultiVariantsMatcher` (261:273) + + +### VariantCombinationGenerator +Generates various combinations of genetic variants, including single nucleotide variants (SNVs), insertions, and deletions, for downstream analysis. It can output these combinations to a VCF file. + + +**Related Classes/Methods**: + +- `kipoiseq.kipoiseq.extractors.variant_combinations.VariantCombinator` (9:98) + + +### VCFSequenceExtractor +Extracts DNA sequences around genetic variants, allowing for the definition of upstream and downstream regions relative to the variant position. It leverages MultiSampleVCF to fetch variants and a FastaStringExtractor for reference sequences. + + +**Related Classes/Methods**: + +- `kipoiseq.kipoiseq.extractors.vcf_seq.VariantSeqExtractor` (60:303) +- `kipoiseq.kipoiseq.extractors.vcf_seq.SingleVariantVCFSeqExtractor` (328:341) +- `kipoiseq.kipoiseq.extractors.vcf_seq.SingleSeqVCFSeqExtractor` (344:355) +- `kipoiseq.kipoiseq.extractors.vcf_seq._BaseVCFSeqExtractor` (306:325) + + +### ProteinDataLoader +A specialized data loader for protein-related sequences, integrating variant information from VCF files with UTR (Untranslated Region) data. + + +**Related Classes/Methods**: + +- `kipoiseq.kipoiseq.dataloaders.protein.SingleVariantUTRDataLoader` (168:356) + + +### ProteinVCFSequenceExtractor +A specialized sequence extractor designed to extract protein-coding sequences based on VCF variants, often used in conjunction with variant matching and building upon multi-interval extraction. + + +**Related Classes/Methods**: + +- `kipoiseq.kipoiseq.extractors.protein.ProteinVCFSeqExtractor` (210:258) +- `kipoiseq.kipoiseq.extractors.protein.SingleVariantProteinVCFSeqExtractor` (265:266) +- `kipoiseq.kipoiseq.extractors.protein.SingleSeqProteinVCFSeqExtractor` (261:262) + + +### MultiIntervalVCFSequenceExtractor +Extracts sequences for multiple genomic intervals based on VCF variants, serving as a foundational component for other VCF sequence extractors. + + +**Related Classes/Methods**: + +- `kipoiseq.kipoiseq.extractors.multi_interval.BaseMultiIntervalVCFSeqExtractor` (202:307) + + + + +### [FAQ](https://github.com/CodeBoarding/GeneratedOnBoardings/tree/main?tab=readme-ov-file#faq) \ No newline at end of file diff --git a/.codeboarding/on_boarding.md b/.codeboarding/on_boarding.md new file mode 100644 index 0000000..bb477a8 --- /dev/null +++ b/.codeboarding/on_boarding.md @@ -0,0 +1,109 @@ +```mermaid +graph LR + Genomic_Data_Models["Genomic Data Models"] + Sequence_Processing["Sequence Processing"] + Genomic_Data_Loaders["Genomic Data Loaders"] + Genomic_Feature_Extractors["Genomic Feature Extractors"] + Variant_Data_Management["Variant Data Management"] + Genomic_Data_Models -- "provides data models to" --> Sequence_Processing + Genomic_Data_Models -- "provides data models to" --> Genomic_Data_Loaders + Genomic_Data_Models -- "provides data models to" --> Genomic_Feature_Extractors + Genomic_Data_Models -- "provides data models to" --> Variant_Data_Management + Sequence_Processing -- "applies transformations to data for" --> Genomic_Data_Loaders + Sequence_Processing -- "applies transformations to extracted data" --> Genomic_Feature_Extractors + Genomic_Data_Loaders -- "loads data using" --> Genomic_Feature_Extractors + Genomic_Data_Loaders -- "integrates variant information from" --> Variant_Data_Management + Genomic_Feature_Extractors -- "provides sequences and features to" --> Genomic_Data_Loaders + Genomic_Feature_Extractors -- "extracts sequences for variant analysis" --> Variant_Data_Management + Variant_Data_Management -- "provides variant data to" --> Genomic_Data_Loaders + Variant_Data_Management -- "queries and filters variants for extraction" --> Genomic_Feature_Extractors + click Genomic_Data_Models href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/kipoiseq/Genomic Data Models.md" "Details" + click Sequence_Processing href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/kipoiseq/Sequence Processing.md" "Details" + click Genomic_Data_Loaders href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/kipoiseq/Genomic Data Loaders.md" "Details" + click Genomic_Feature_Extractors href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/kipoiseq/Genomic Feature Extractors.md" "Details" + click Variant_Data_Management href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/kipoiseq/Variant Data Management.md" "Details" +``` +[![CodeBoarding](https://img.shields.io/badge/Generated%20by-CodeBoarding-9cf?style=flat-square)](https://github.com/CodeBoarding/GeneratedOnBoardings)[![Demo](https://img.shields.io/badge/Try%20our-Demo-blue?style=flat-square)](https://www.codeboarding.org/demo)[![Contact](https://img.shields.io/badge/Contact%20us%20-%20contact@codeboarding.org-lightgrey?style=flat-square)](mailto:contact@codeboarding.org) + +## Component Details + +The `kipoiseq` library provides a comprehensive framework for handling and processing genomic sequence and variant data. Its main flow involves loading biological data from various sources (FASTA, GTF, VCF), transforming these sequences for downstream analysis (e.g., one-hot encoding), extracting specific genomic features or sequences around variants, and managing genetic variant information. The library's purpose is to facilitate the preparation and manipulation of genomic data for machine learning and other bioinformatics applications. + +### Genomic Data Models +Provides fundamental data structures like Variant for genetic variations and Interval for genomic regions, which are used throughout the kipoiseq library to represent and manipulate biological data. + + +**Related Classes/Methods**: + +- `kipoiseq.kipoiseq.dataclasses.Variant` (17:149) +- `kipoiseq.kipoiseq.dataclasses.Interval` (152:374) + + +### Sequence Processing +Offers a suite of functions and classes for transforming biological sequences, including one-hot encoding, resizing intervals, and handling sequence axes, crucial for preparing data for machine learning models. It also includes general-purpose helper functions. + + +**Related Classes/Methods**: + +- `kipoiseq.kipoiseq.transforms.transforms.OneHot` (91:124) +- `kipoiseq.kipoiseq.transforms.functional.one_hot_dna` (118:133) +- `kipoiseq.kipoiseq.transforms.transforms.ResizeInterval` (77:86) +- `kipoiseq.utils.batch_iter` (48:57) +- `kipoiseq.utils.parse_alphabet` (27:31) +- `kipoiseq.utils.parse_dtype` (34:42) +- `kipoiseq.utils.to_scalar` (18:24) + + +### Genomic Data Loaders +Provides various data loaders for handling genomic sequences based on intervals, BED files, and GTF annotations, supporting both string and one-hot encoded sequence outputs, including specialized loaders for splicing and protein data. + + +**Related Classes/Methods**: + +- `kipoiseq.kipoiseq.dataloaders.sequence.SeqIntervalDl` (268:384) +- `kipoiseq.kipoiseq.dataloaders.sequence.StringSeqIntervalDl` (135:261) +- `kipoiseq.kipoiseq.dataloaders.sequence.BedDataset` (26:131) +- `kipoiseq.kipoiseq.dataloaders.splicing.MMSpliceDl` (182:276) +- `kipoiseq.kipoiseq.dataloaders.splicing.ExonInterval` (32:129) +- `kipoiseq.kipoiseq.dataloaders.protein.SingleVariantProteinDataLoader` (30:164) +- `kipoiseq.kipoiseq.dataloaders.protein.SingleVariantUTRDataLoader` (168:356) + + +### Genomic Feature Extractors +Extracts genomic sequences from FASTA files and features from GTF files (e.g., CDS, UTRs). It also provides functionalities for extracting sequences across multiple genomic intervals and protein sequences. + + +**Related Classes/Methods**: + +- `kipoiseq.extractors.fasta.FastaStringExtractor` (7:63) +- `kipoiseq.kipoiseq.extractors.gtf.CDSFetcher` (154:265) +- `kipoiseq.kipoiseq.extractors.gtf.UTRFetcher` (268:377) +- `kipoiseq.kipoiseq.extractors.gtf.GTFMultiIntervalFetcher` (88:151) +- `kipoiseq.kipoiseq.extractors.protein.ProteinSeqExtractor` (169:180) +- `kipoiseq.kipoiseq.extractors.protein.TranscriptSeqExtractor` (112:166) +- `kipoiseq.kipoiseq.extractors.protein.UTRSeqExtractor` (35:69) +- `kipoiseq.kipoiseq.extractors.multi_interval.GenericMultiIntervalSeqExtractor` (129:195) + + +### Variant Data Management +Manages core operations related to Variant Call Format (VCF) files, including fetching, querying, filtering, and matching genetic variants. It also handles the extraction of genomic sequences around variants and the generation of variant combinations. + + +**Related Classes/Methods**: + +- `kipoiseq.kipoiseq.extractors.vcf.MultiSampleVCF` (21:211) +- `kipoiseq.kipoiseq.extractors.vcf_query.VariantQuery` (33:45) +- `kipoiseq.kipoiseq.extractors.vcf_query.VariantIntervalQueryable` (113:281) +- `kipoiseq.kipoiseq.extractors.vcf_matching.SingleVariantMatcher` (199:258) +- `kipoiseq.kipoiseq.extractors.vcf_matching.variants_to_pyranges` (22:39) +- `kipoiseq.kipoiseq.extractors.vcf_matching.intervals_to_pyranges` (63:83) +- `kipoiseq.kipoiseq.extractors.vcf_seq.VariantSeqExtractor` (60:303) +- `kipoiseq.kipoiseq.extractors.vcf_seq.SingleVariantVCFSeqExtractor` (328:341) +- `kipoiseq.kipoiseq.extractors.vcf_seq.SingleSeqVCFSeqExtractor` (344:355) +- `kipoiseq.kipoiseq.variant_source.VariantFetcher` (8:39) +- `kipoiseq.kipoiseq.extractors.variant_combinations.VariantCombinator` (9:98) + + + + +### [FAQ](https://github.com/CodeBoarding/GeneratedOnBoardings/tree/main?tab=readme-ov-file#faq) \ No newline at end of file