Page History

...

Writing to Mirrored Files: When a client writes to a mirrored file, only one primary (preferred) mirror is updated directly during the write operation. The other mirrors are simply marked as "stale" to indicate they're out of sync with the primary mirror.

Mirroring States: The file system has the following states for mirrored files:

Read-only (ro): File cannot be written without a layout state change; when a write starts, trigger transition to:
Write pending (wp): Open for writing, new writes do not need to change the layout.  A resync starting transitions to:
Sync pending (sp): Marked for synchronization - data copy for synchronization can start.  Writes trigger layout to go back to write pending (and sync process fails)

Today, we have manual synchronization: After writing to a mirrored fileafter a write, the lfs mirror resync command must be run to synchronize the stale mirrors with the primary mirror. This command copies data from the synced mirror to the stale mirrors and marks removes the stale flag from successfully copied mirrors as synchronized.

Layout state and staleness is managed through a careful series of layout state changes which are described File-level replication state machine in CoreDesignConcept in this document.

This delayed write approach was implemented in the first phase of FLR to avoid the complexity of maintaining consistency across multiple mirrors during concurrent writes. By updating only one mirror during writes and marking others as stale, the system maintains a consistent view of the file data, at the cost of requiring explicit synchronization after writes complete.

...