[Modular]add a real quick start guide #13029

yiyixuxu · 2026-01-26T00:07:51Z

some of the workflow stuff is not yet support in main, this is the PR #13028

docs/source/en/modular_diffusers/quickstart.md

HuggingFaceDocBuilderDev · 2026-01-26T00:19:19Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

stevhliu

very nice and easier to start with!

docs/source/en/modular_diffusers/overview.md

docs/source/en/modular_diffusers/quickstart.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

sayakpaul

Very very cool rewrite!

sayakpaul · 2026-01-27T04:36:50Z

docs/source/en/modular_diffusers/quickstart.md

 # Quickstart

-Modular Diffusers is a framework for quickly building flexible and customizable pipelines. At the core of Modular Diffusers are [`ModularPipelineBlocks`] that can be combined with other blocks to adapt to new workflows. The blocks are converted into a [`ModularPipeline`], a friendly user-facing interface developers can use.
+Modular Diffusers is a framework for quickly building flexible and customizable pipelines. At the core of Modular Diffusers are [`ModularPipelineBlocks`] that can be combined with other blocks to adapt to new workflows. The blocks are converted into a [`ModularPipeline`], a friendly user-facing interface for running generation tasks.


Suggested change

Modular Diffusers is a framework for quickly building flexible and customizable pipelines. At the core of Modular Diffusers are [`ModularPipelineBlocks`] that can be combined with other blocks to adapt to new workflows. The blocks are converted into a [`ModularPipeline`], a friendly user-facing interface for running generation tasks.

Modular Diffusers is a framework for quickly building flexible and customizable pipelines. These pipelines can go beyond what standard `DiffusionPipeline`s can do. At the core of Modular Diffusers are [`ModularPipelineBlocks`] that can be combined with other blocks to adapt to new workflows. The blocks are converted into a [`ModularPipeline`], a friendly user-facing interface for running generation tasks.

sayakpaul · 2026-01-27T04:38:54Z

docs/source/en/modular_diffusers/quickstart.md

 specific language governing permissions and limitations under the License.
 -->

 # Quickstart


@stevhliu @yiyixuxu do you think the introduction to Modular Diffusers (that we have below) should be moved overview.md?

I will understand if that's out of scope for this PR. Maybe in a followup, we can clarify the vision we have for modular?

i think its probably ok to keep an intro in both. you can have a more detailed one in overview.md and keep it brief in the quickstart

sayakpaul · 2026-01-27T04:42:32Z

docs/source/en/modular_diffusers/quickstart.md

-## Customizing blocks
+pipe = ModularPipeline.from_pretrained("Qwen/Qwen-Image")
+pipe.load_components(torch_dtype=torch.bfloat16)
+pipe.to("cuda")


Should we instead do auto offloading here? I am suggesting it since I envision this guide to reach many people and if they see we're doing a direct CUDA placement for a fairly large model like QwenImage, they might think differently about it.

Maybe we could add a note clarifying that we discuss this later in the doc?

sayakpaul · 2026-01-27T04:48:31Z

docs/source/en/modular_diffusers/quickstart.md

-Call [`SequentialPipelineBlocks.from_blocks_dict`] on the blocks to create a `SequentialPipelineBlocks`.
+### Workflows

+`QwenImageAutoBlocks` is a [`ConditionalPipelineBlocks`], so this pipeline supports multiple workflows and adapts its behavior based on the inputs you provide. For example, if you pass `image` to the pipeline, it runs an image-to-image workflow instead of text-to-image.


Suggested change

`QwenImageAutoBlocks` is a [`ConditionalPipelineBlocks`], so this pipeline supports multiple workflows and adapts its behavior based on the inputs you provide. For example, if you pass `image` to the pipeline, it runs an image-to-image workflow instead of text-to-image.

`QwenImageAutoBlocks` is a [`ConditionalPipelineBlocks`], so this pipeline supports multiple workflows and adapts its behavior based on the inputs you provide. For example, if you pass `image` to the pipeline, it runs an image-to-image workflow instead of text-to-image. Let's see this in action with an example.

sayakpaul · 2026-01-27T04:50:14Z

docs/source/en/modular_diffusers/quickstart.md

+).images[0]
+```

+Use `get_workflow()` to extract the blocks for a specific workflow.


A bit unclear as to how get_workflow() should be called. For example, if I wanted to get the I2I workflow, how should I call it?

sayakpaul · 2026-01-27T04:51:00Z

docs/source/en/modular_diffusers/quickstart.md

+### Sub-blocks

-### IP-Adapter
+`QwenImageAutoBlocks` is itself composed of smaller blocks: `text_encoder`, `vae_encoder`, `controlnet_vae_encoder`, `denoise`, and `decode`. Access them through the `sub_blocks` property.


Should we briefly clarify the difference between blocks and sub_blocks and theirt scope of usage?

sayakpaul · 2026-01-27T04:51:35Z

docs/source/en/modular_diffusers/quickstart.md


-Use the [`sub_blocks.insert`] method to insert it into the [`ModularPipeline`]. The example below inserts the `ip_adapter_block` at position `0`. Print the pipeline to see that the `ip_adapter_block` is added and it requires an `ip_adapter_image`. This also added two components to the pipeline, the `image_encoder` and `feature_extractor`.
-
+This block can be converted to a pipeline and run on its own with [`~ModularPipelineBlocks.init_pipeline`].


Suggested change

This block can be converted to a pipeline and run on its own with [`~ModularPipelineBlocks.init_pipeline`].

This block can be converted to a pipeline so that it can run on its own with [`~ModularPipelineBlocks.init_pipeline`].

sayakpaul · 2026-01-27T04:56:34Z

docs/source/en/modular_diffusers/quickstart.md

-dd_auto_blocks = SequentialPipelineBlocks.from_blocks_dict(DIFFDIFF_AUTO_BLOCKS)
-dd_pipeline = dd_auto_blocks.init_pipeline("YiYiXu/modular-demo-auto", collection="diffdiff")
-dd_pipeline.load_components(torch_dtype=torch.float16)
-```


Not strongly opinionated but we could also add the example of prompt upsampling that uses Gemini. This might inspire users to explore a combination of open and closed models -- something that is made quite seamless with Modular.

add a real quick start guide

318f2bf

yiyixuxu commented Jan 26, 2026

View reviewed changes

docs/source/en/modular_diffusers/quickstart.md Outdated Show resolved Hide resolved

Update docs/source/en/modular_diffusers/quickstart.md

809fc36

yiyixuxu added 2 commits January 26, 2026 02:01

update a bit more

56dd6cc

fix

fe4e4d7

yiyixuxu requested review from sayakpaul and stevhliu January 26, 2026 01:05

stevhliu reviewed Jan 26, 2026

View reviewed changes

yiyixuxu and others added 4 commits January 26, 2026 08:26

Apply suggestions from code review

8483c06

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

Update docs/source/en/modular_diffusers/quickstart.md

b6d05bb

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

Update docs/source/en/modular_diffusers/quickstart.md

7dc454f

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

update more

077b697

sayakpaul reviewed Jan 27, 2026

View reviewed changes

	`QwenImageAutoBlocks` is a [`ConditionalPipelineBlocks`], so this pipeline supports multiple workflows and adapts its behavior based on the inputs you provide. For example, if you pass `image` to the pipeline, it runs an image-to-image workflow instead of text-to-image.
	`QwenImageAutoBlocks` is a [`ConditionalPipelineBlocks`], so this pipeline supports multiple workflows and adapts its behavior based on the inputs you provide. For example, if you pass `image` to the pipeline, it runs an image-to-image workflow instead of text-to-image. Let's see this in action with an example.


		Use the [`sub_blocks.insert`] method to insert it into the [`ModularPipeline`]. The example below inserts the `ip_adapter_block` at position `0`. Print the pipeline to see that the `ip_adapter_block` is added and it requires an `ip_adapter_image`. This also added two components to the pipeline, the `image_encoder` and `feature_extractor`.

		This block can be converted to a pipeline and run on its own with [`~ModularPipelineBlocks.init_pipeline`].

[Modular]add a real quick start guide #13029

Are you sure you want to change the base?

[Modular]add a real quick start guide #13029

Conversation

yiyixuxu commented Jan 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

HuggingFaceDocBuilderDev commented Jan 26, 2026

Uh oh!

stevhliu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sayakpaul left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

yiyixuxu commented Jan 26, 2026 •

edited

Loading