Page History

...

• Set or clear the TRASH flag on a given file or directory
• list files in the Trash Can on a given MDT
• Permantently Permanently delete a file or directory within the Trash Can on a given MDT
• Empty the Trash Can on a given MDT
• Restore a file within the Trash Can on a give MDT

...

Also, there needs to be some accounting of files in the trash, so that "df" does not show the filesystem as 100% or 90% full all the time, but rather show only the non-trash space usage (= real usage - trash usage).

Per-

...

User Trash Can

It can define a A per-user TRASHTrash/MDTxxxx/UID/ directory that is owned by that UID and mode 0600should always be created in the top-level directory to avoid world readable access to deleted files, and to de-conflict files/directories of the same name created by users (e.g. tmp/ or data/ or Documents/ or similar. That avoids exposing files to other users that may be private, and also allows tracking space usage more clearly for each UID, so that a user's data can be found and purged more quickly if they are exceeding their quotas.

Per-Tenant Trash Can

Files and directories deleted from within a subdirectory mount of a Nodemap should be stored in a Trash/MDTxxxx/NODEMAP/UID/ directory to isolate the files/directories from different tenants.  The NODEMAP/ directoryname is the configured name of the nodemap for that tenant, and can be found from the client export used to perform the final unlink operation. The UID/ directory name should be the unmapped ID of the user, so that the visible directory name matches the user expectation.  The UID directory ownership should be the mapped ID of the user, so that proper file access controls can be maintained.  By having the multi-level NODEMAP/UID/ naming, it isolates the UID directory names from other tenants that may have the same mapped UID directory name.

The Nodemap for a tenant should allow configuring the UID/GID/PROJID to which files in the Trash are assigned.  These IDs should be within the ID offset range of the Tenant (e.g. 99999) so that they can be accessed and mapped correctly, but are unlikely to cause conflicts with other IDs used by the Tenant.  This will also allow the Tenant project quota group to account for all space used by the tenancy, while still separating Trash Can usage for the regular UID/GID/PROJID of the Tenant users.

In a multi-tenant environment, it would be desirable to have a more sophisticated policy engine to manage Garbage Collection of files within the trash, in order to provide maximum utility to each Tenant.  For example, if Tenant 1 has deleted files an hour ago, but Tenant 2 has written and deleted TB of data since that time, the Tenant 1 files may have expired out of the Trash Can.  Developing a complex policy engine to manage GC in an MTFL environment is out of scope for the initial TCU implementation.  We likely want to leverage and enhance the lpurge utility from Hot Pools to actively monitor the space usage of tenants on different OSTs to decide which objects (files) should be removed.

Trash support for a striped directory

It would better to implement a virtual ".Trash" subdirectory accessible in each directory in the filesystem that can be used to browse files/directories in the trash can and access them for recovery.

The FID of the ".Trash" directory is derived from the FID of the parent directory (pFID), by looking up the corresponding "stub" directory with the FID-named directory: ".lustre/trash/MDTXXXX/pFID". Essentially ".Trash" under each normal directory is just a virtual shortcut to the stub directory (if the parent is not a striped dir) that is accessible in each directory if specified by name ".Trash". The files/directories under ".Trash" directory can be access via normal POSIX file system API such as via readdir()/stat()/getxattr()the stub directory (if the parent is not a striped dir) that is accessible in each directory if specified by name ".Trash". The files/directories under ".Trash" directory can be access via normal POSIX file system API such as via readdir()/stat()/getxattr() so that it can be used by normal tools such as "ls -l .Trash/" or "find .Trash" to locate files for restoration or permanent removal.  If there are no deleted files under a specific directory, then the virtual .Trash directory will not be accessible, and will return -ENOENT for any lookup.

For a striped directory, its ".Trash" directory is also a vitual striped directory with each stripe on the same location (MDTs) where the shard FID is the FID of the corresponding stub directory on that MDT. If the stub directory on a certain MDT does not exist (or not create yet), the client lookup() or readdir() under ".Trash" directory should skip the stripe. The master FID of the virtual ".Trash" directory could be same with the FID of the parent directory but with f_ver setting with 1 (FID_VERSION_TRASH = 1) to distinguish them.

To avoid the inconsistent problem, each access on the virtual striped ".Trash" should check and revalidate the virtual stripe LMV EA. For example, It should add the new shards into the stripe EA after a new stub directory on a certain MDT was created.

It should handle the case that a directory was restriped and the LMV layout was changed. In this case, the files under the directory will be migrated to another MDT. To simplify the impelmentationimplementation, we do not migrate the files according to the new LMV layout in the Trash Can. This may result in the lookup() operation will be issued to a wrong MDT and return -ENOENT  wrongly (after files in the trash can are restored). However, the readdir() operations will still return all the dir entries in the striped trash even if the parent LMV layout was re-striped and changed, since the parent directory FID (pFID) will remain the same as before restriping. Maybe it needs to migrate the files restored from the trash can to the appropriate shard according to their name hash once the LMV layout has been changed.

Orphans in Trash Can

For an orphan file, it means the file is still opened (but not closed) by a certain user. Upon its last unlink, it can directly move into the trash can and mark with LUSTRE_ORPHAN_FL | LUSTRE_TRASH_FL. And the orphans file can not be permanently deleted from the trash can until its last close(). 

...