dualpipe

We have hosted the application dualpipe in order to run this application in our online workstations with Wine or directly.

Quick description about dualpipe:

DualPipe is a bidirectional pipeline parallelism algorithm open-sourced by DeepSeek, introduced in their DeepSeek-V3 technical framework. The main goal of DualPipe is to maximize overlap between computation and communication phases during distributed training, thus reducing idle GPU time (i.e. �pipeline bubbles�) and improving cluster efficiency. Traditional pipeline parallelism methods (e.g. 1F1B or staggered pipelining) leave gaps because forward and backward phases can�t fully overlap with communication. DualPipe addresses that by scheduling micro-batches from both ends of the pipeline in a bidirectional fashion�i.e. some micro-batches flow forward while others flow backward�so that computation on one partition can coincide with communication for another.

Features:

• Bidirectional scheduling of micro-batches to overlap forward and backward passes
• Reduction of pipeline bubbles and GPU idle times
• Support for composition with other parallelism strategies (MoE, EP, tensor parallelism)
• Python implementation of scheduling logic (dualpipe.py)
• Profile / diagnostics to measure computation-communication overlap and efficiency
• Open source and publicly released as part of DeepSeek�s open infrastructure initiative

Programming Language: Python.
Categories: