Skip to content

Add tracking for Celery canvas primitives (map/starmap/chunks)#38

Merged
snopoke merged 1 commit intomainfrom
sk/signals
Feb 5, 2026
Merged

Add tracking for Celery canvas primitives (map/starmap/chunks)#38
snopoke merged 1 commit intomainfrom
sk/signals

Conversation

@snopoke
Copy link
Collaborator

@snopoke snopoke commented Feb 4, 2026

Summary

  • Add tracking for tasks spawned via Celery canvas primitives (map, starmap, chunks)
  • Canvas primitives now create TaskBadger tasks using the inner task's name with metadata about canvas_type and item_count
  • Add new tests for test_task_map, test_task_starmap, and test_task_chunks

Background

Previously, tasks using canvas primitives like task.chunks(items, n).group() or task.map(items) were not tracked because these execute as built-in celery.map/celery.starmap tasks, which were filtered out by the before_task_publish signal handler.

Implementation

Added _maybe_create_task function called during task_prerun that:

  • Creates TaskBadger tasks for canvas primitives using the inner task's name
  • Includes metadata: {"canvas_type": "celery.starmap", "item_count": 3}
  • Carefully orders checks to avoid thread-local Badger state pollution
  • Respects existing tracking intent via taskbadger_track header

Test plan

  • All existing celery tests pass
  • New tests for map, starmap, and chunks canvas primitives
  • Manual verification with eager mode

🤖 Generated with Claude Code

@snopoke snopoke force-pushed the sk/signals branch 2 times, most recently from 0a65d2b to 802d3f3 Compare February 4, 2026 18:51
Previously, tasks spawned via Celery canvas primitives like map, starmap,
and chunks were not tracked because these execute as built-in celery.map
or celery.starmap tasks, which were filtered out by the before_task_publish
signal handler.

This change adds a _maybe_create_task function called during task_prerun
that:
- Creates TaskBadger tasks for canvas primitives with descriptive names
  like "(map 5) myapp.tasks.process_item"
- Includes metadata about canvas_type and item_count in task data
- Optionally includes celery_task_items when record_task_args is enabled
- Carefully orders checks to avoid thread-local Badger state pollution
- Respects existing tracking intent via taskbadger_track header

For example, when using task.chunks(items, 2).group(), each chunk now
creates a TaskBadger task named "(starmap 2) myapp.tasks.process_item"
with data showing the canvas type and item count.
@snopoke snopoke merged commit 235d7fc into main Feb 5, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant