✨ NEW: Add Nucleus Detection Engine #538

shaneahmed · 2023-02-16T13:10:58Z

Add Nucleus Detection Engine
Fix documentation in patch_predictor.py

- Add Nucleus Detection Engine - Fix documentation in patch_predictor.py

- Improve structure of init

codecov · 2023-02-16T16:35:01Z

Codecov Report

Merging #538 (6368325) into develop (25f5f3d) will decrease coverage by 0.15%.
The diff coverage is 100.00%.

❗ Current head 6368325 differs from pull request most recent head 7989aa7. Consider uploading reports for the commit 7989aa7 to get more accurate results

@@             Coverage Diff             @@
##           develop     #538      +/-   ##
===========================================
- Coverage    99.77%   99.63%   -0.15%     
===========================================
  Files           64       63       -1     
  Lines         6817     6642     -175     
  Branches      1101     1078      -23     
===========================================
- Hits          6802     6618     -184     
- Misses           7       15       +8     
- Partials         8        9       +1

Impacted Files	Coverage Δ
tiatoolbox/models/engine/__init__.py	`100.00% <ø> (ø)`
tiatoolbox/models/engine/semantic_segmentor.py	`100.00% <ø> (ø)`
tiatoolbox/models/engine/nucleus_detector.py	`100.00% <100.00%> (ø)`
tiatoolbox/models/engine/patch_predictor.py	`100.00% <100.00%> (ø)`

... and 24 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

…-engine

- Add documentation for mapde and sccnn - Add a test for nucleus detection - Fix input in patch predictor test

- Fix mpp for cell detection

…-engine # Conflicts: # docs/pretrained.rst # tiatoolbox/data/pretrained_model.yaml

- Fix bug with loading NucleusDetector

- Fix bug with loading sccnn

- Fix test bug in test_patch_predictor.py

- Add test for NucleusDetector

- Set GPU Signed-off-by: Shan E Ahmed Raza <13048456+shaneahmed@users.noreply.github.com>

- Add test to improve coverage

…-engine

mostafajahanifar

Thanks, Shan for doing this important PR.
I made some comments. As we mentioned before, I believe this engine requires some special consideration in the design, especially for post-processing. However, if we want to have it in its current shape for now, it needs some changes as well as some more rigorous testing. For example, how does the algorithm work on a slightly large WSI on a mediocre system (16-32GB RAM)?
Also, we need to consider the detection class and probability in the outputs as well because there might be some models that predict these parameters too.

tiatoolbox/models/engine/nucleus_detector.py

mostafajahanifar · 2023-03-09T13:19:30Z

tiatoolbox/models/engine/nucleus_detector.py

+)
+
+
+class IONucleusDetectorConfig(IOSegmentorConfig):


In the current shape, defining this new class seems unnecessary because there is no change to IOSegmentorConfig. I see no reason why we could not use the main class.

Agree I can remove this for now.

tiatoolbox/models/engine/nucleus_detector.py

mostafajahanifar · 2023-03-09T14:15:38Z

tiatoolbox/models/engine/nucleus_detector.py

+            # Coordinates in output resolution for the current canvas.
+            cum_canvas = np.expand_dims(cum_canvas, axis=0)
+            coordinates_canvas = pd.DataFrame(
+                self.model.postproc_func(cum_canvas), columns=["x", "y"]


Now that engine is written like this, the post_proc function should be working with all detection models. There are two major things to consider:
1- this part should be compatible with all the detection models
2- this part should also support prediction class and prediction probability of detections (if available)

Therefore, it seems that we either need to update the postproc function of the detection networks to return the results in a predefined structure ([x, y, cls, prob]) and then convert the results properly here.

Also, I wonder if the post-processing works with large WSIs? as it seems that results are saved in high resolution and then the whole canvas is passed to the post-processing.

- Add example models

# Conflicts: # tiatoolbox/models/engine/patch_predictor.py # tiatoolbox/models/engine/semantic_segmentor.py

- Pin `torch` version

- Try cuda 11.8 to run the tests as it passes locally. Signed-off-by: Shan E Ahmed Raza <13048456+shaneahmed@users.noreply.github.com>

# 🚀 Summary This PR introduces a new **`NucleusDetector` engine** to the TIAToolbox framework, enabling detection of nuclei from whole slide images (WSIs) or image patches using models such as **`MapDe`** and **`SCCNN`**. It supersedes PR #538 by leveraging **`dask`** for efficient, parallelized post-processing and result merging. The update also includes: --- ## ✨ Key Features ### **New Engine: `NucleusDetector`** - Detects nuclei centroids and probabilities from WSIs or patches. - Produces a **detection map** aligned with segmentation dimensions. - Serializes detections into **detection arrays** (`[[x], [y], [type], [probs]]`). - Supports multiple output backends: - **SQLiteStore** (chunked storage for WSI/patch). - **Dictionary** (flat or patch-indexed). - **Zarr** (arrays for coordinates, classes, probabilities). - Compatible with nucleus detection models: - **MapDe** (implemented). - **SCCNN** (integration in progress/debugging). - Supports both **patch-mode** and **WSI-mode** workflows. ### Technical Implementation The detection pipeline operates as follows: 1. **Segmentation**: A WSI-level segmentation map (dask array) is generated using `SemanticSegmentor.infer_wsi()`. 2. **Parallel Post-processing**: For WSI inference. Use `dask.array.map_overlap` to apply the model's post-processing function across the entire segmentation map. This allows the function to execute in parallel on chunks, after which the results are automatically merged back into a unified **"detection_map"**, which is saved as zarr in a cache directory for further processing. 3. **Detection Map**: * Maintains the same dimensions as the segmentation map. * Nuclei centroids contain the detection probability values (defaults to `1` if the model does not produce probabilities). 4. **Serialization**: The "detection_map" is converted into **"detection_arrays"** (format: `[[x], [y], [type], [probs]]`) representing the detected nuclei. These records are then saved into `SQLiteStore` (chunk-by-chunk), `zarr`, or a `dict` (patch mode only). ### Output Formats #### SQLiteStore * **WSI Mode**: Returns a single `SQLiteStore`. * **Patch Mode**: Returns one `SQLiteStore` per patch. * **Format**: ```python Annotation(Point(x,y), properties={'type': 'nuclei', 'probs': 0.9}) ``` #### Dictionary * **WSI Mode**: ```python { 'x': [...], 'y': [...], 'classes': [...], 'probs': [...] } ``` * **Patch Mode** (One sub-dictionary per patch index): ```python { 0: { 'x': [...], 'y': [...], 'classes': [...], 'probs': [...] }, 1: { ... } } ``` #### Zarr * **WSI Mode**: ```python { 'x': [...], 'y': [...], 'classes': [...], 'probs': [...] } ``` * **Patch Mode**: Each key maps to a list of `da.array` objects, where each array corresponds to a patch. ```python { 'x': [[...], ...], 'y': [[...], ...], 'classes': [[...], ...], 'probs': [[...], ...] } ``` ### **Codebase Integration** - Registers `NucleusDetector` in `tiatoolbox.models` and engine registry. - Refactors detection logic from PR #538 into modular components. - Updates `MapDe` implementation to use the new engine. - Begins integration of `SCCNN` with `NucleusDetector`. - Adds utilities for serialization into SQLite, dict, and zarr formats. - Introduces unit tests for detection workflows. - Removes unused parameters `prediction_shape` and `prediction_dtype` from `post_process_patches()` and `post_process_wsi()` functions in all engines. - `post_process_patches()` and `post_process_wsi()` now takes `raw_predictions` instead of `raw_predictions["probabilities"]`. ### Tasks - [x] Port code from PR #538 to supersede and close it. - [x] Add `NucleusDetector` engine. - [x] Update existing detection models (`MapDe` implementation complete; `SCCNN` implementation in progress/debugging). - [x] Add unit tests. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Shan E Ahmed Raza <13048456+shaneahmed@users.noreply.github.com> Co-authored-by: Jiaqi Lv <jiaqilv@Jiaqis-MacBook-Pro.local>

✨ NEW: Add Nucleus Detection Engine

a6960f8

- Add Nucleus Detection Engine - Fix documentation in patch_predictor.py

shaneahmed marked this pull request as draft February 16, 2023 13:18

🎨 DEV: Improve structure of init

6c47506

- Improve structure of init

shaneahmed self-assigned this Feb 16, 2023

Merge branch 'develop' into new-cell-detection-engine

c902372

shaneahmed added 5 commits February 20, 2023 15:55

Merge remote-tracking branch 'origin/develop' into new-cell-detection…

73cfe90

…-engine

📝 DOC: Add documentation for mapde and sccnn

584e43a

- Add documentation for mapde and sccnn - Add a test for nucleus detection - Fix input in patch predictor test

💥 DEV: Fix mpp for cell detection

46ffdae

- Fix mpp for cell detection

Merge remote-tracking branch 'origin/develop' into new-cell-detection…

951ea7d

…-engine # Conflicts: # docs/pretrained.rst # tiatoolbox/data/pretrained_model.yaml

🐛 BUG: Fix bug with loading NucleusDetector

4e2ca8b

- Fix bug with loading NucleusDetector

shaneahmed requested a review from mostafajahanifar February 21, 2023 14:19

shaneahmed added 3 commits February 21, 2023 14:33

🐛 BUG: Fix bug with loading sccnn

1af8140

- Fix bug with loading sccnn

🐛 BUG: Fix test bug in test_patch_predictor.py

527444a

- Fix test bug in test_patch_predictor.py

✅ TST: Add test for NucleusDetector

37b82f9

- Add test for NucleusDetector

shaneahmed marked this pull request as ready for review February 21, 2023 17:42

shaneahmed added 2 commits February 21, 2023 18:26

🐛 TST: Set GPU

1405fa0

- Set GPU Signed-off-by: Shan E Ahmed Raza <13048456+shaneahmed@users.noreply.github.com>

✅ TST: Add test to improve coverage

71cb5c6

- Add test to improve coverage

shaneahmed added the enhancement New feature or request label Feb 22, 2023

shaneahmed added 2 commits February 24, 2023 15:23

Merge remote-tracking branch 'origin/develop' into new-cell-detection…

e4625c0

…-engine

Merge branch 'develop' into new-cell-detection-engine

61d811a

mostafajahanifar requested changes Mar 9, 2023

View reviewed changes

shaneahmed added 3 commits March 10, 2023 16:32

Merge branch 'develop' into new-cell-detection-engine

4a0a940

Merge branch 'develop' into new-cell-detection-engine

a6052fc

Merge branch 'develop' into new-cell-detection-engine

6368325

shaneahmed added this to the Release v2.0.0 milestone Apr 10, 2023

shaneahmed added 2 commits May 2, 2023 16:47

Merge branch 'develop' into new-cell-detection-engine

8b67d78

📝 Add example models

b11e6f0

- Add example models

shaneahmed modified the milestones: Release v2.0.0, Release v1.5.0 May 4, 2023

shaneahmed added 7 commits May 5, 2023 17:39

Merge branch 'develop' into new-cell-detection-engine

f538289

Merge branch 'develop' into new-cell-detection-engine

57bde23

Merge branch 'develop' into new-cell-detection-engine

861532d

# Conflicts: # tiatoolbox/models/engine/patch_predictor.py # tiatoolbox/models/engine/semantic_segmentor.py

📌 Pin torch version

2c3884a

- Pin `torch` version

📌 Pin torch version

e670455

- Pin `torch` version

Merge branch 'develop' into new-cell-detection-engine

8d92c5e

🔨 Try cuda 11.8

7989aa7

- Try cuda 11.8 to run the tests as it passes locally. Signed-off-by: Shan E Ahmed Raza <13048456+shaneahmed@users.noreply.github.com>

shaneahmed added the stale Old PRs/Issues which are inactive label Jun 29, 2023

shaneahmed modified the milestones: Release v1.5.0, Release v2.0.0 Jul 19, 2023

shaneahmed mentioned this pull request Aug 16, 2024

Nucleus Detection gives error #844

Closed

shaneahmed marked this pull request as draft October 4, 2024 11:36

shaneahmed mentioned this pull request Jan 24, 2025

Tuple index out of range error during post-processing step in Nucleus Instance Segmentation task #899

Closed

Jiaqi-Lv mentioned this pull request Nov 6, 2025

🆕 Define NucleusDetector #967

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

✨ NEW: Add Nucleus Detection Engine #538

✨ NEW: Add Nucleus Detection Engine #538

Uh oh!

shaneahmed commented Feb 16, 2023

Uh oh!

codecov bot commented Feb 16, 2023 •

edited

Loading

Uh oh!

mostafajahanifar left a comment

Uh oh!

Uh oh!

mostafajahanifar Mar 9, 2023

Uh oh!

shaneahmed Mar 10, 2023

Uh oh!

Uh oh!

Uh oh!

mostafajahanifar Mar 9, 2023 •

edited

Loading

Uh oh!

mostafajahanifar Mar 9, 2023 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

✨ NEW: Add Nucleus Detection Engine #538

Are you sure you want to change the base?

✨ NEW: Add Nucleus Detection Engine #538

Uh oh!

Conversation

shaneahmed commented Feb 16, 2023

Uh oh!

codecov bot commented Feb 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

mostafajahanifar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mostafajahanifar Mar 9, 2023

Choose a reason for hiding this comment

Uh oh!

shaneahmed Mar 10, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mostafajahanifar Mar 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mostafajahanifar Mar 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov bot commented Feb 16, 2023 •

edited

Loading

mostafajahanifar Mar 9, 2023 •

edited

Loading

mostafajahanifar Mar 9, 2023 •

edited

Loading