Why FSDP not DPP？
Author: noforitCreated Apr 1, 2024Updated Aug 13, 2024
Could I kindly inquire as to why, given the relatively small size of the tinyllama model, the Strategy was made to utilize FSDP (Fully Sharded Data Parallel) instead of DDP (Distributed Data Parallel)?
Source: jzhang38/TinyLlama