...

In a multi-tenant environment, it would be desirable to have a more sophisticated policy engine to manage Garbage Collection of files within the trash, in order to provide maximum utility to each Tenant.  For example, if Tenant 1 has deleted files an hour ago, but Tenant 2 has written and deleted TB of data since that time, the Tenant 1 files may have expired out of the Trash Can.  Developing a complex policy engine to manage GC in an MTFL environment is out of scope for the initial TCU implementation.  We likely want to leverage and enhance the lpurge utility from Hot Pools to actively monitor the space usage of tenants on different OSTs to decide which objects (files) should be removed.

Repeated deletion of same filename

If the same filename is repeatedly created and deleted within the same parent directory, then the deleted files will have conflicts when moved into the pFID directory in the Trash Can.  To disambiguate the files in Trash, the conflicting filenames should be disambiguated by appending a timestamp to the filename, like filename.2025-04-03-00:11:24, possibly adding .microseconds if there is still a conflict.  It isn't totally clear whether it would be better to use the timestamp from when the file was deleted, or when the file was created.  Both have some value to help users distinguish between the different versions.

JobID of process deleting a file

In LU-13031 the JobID of the process that first creates a file is stored in the user.job xattr on the MDT inode, for diagnostic purposes and to allow determining provenance of each file later on.  For the Trash Can, it would be useful to also store the JobID of the process that is deleting the file, for diagnostics such as determining rogue processes that are deleting files in the filesystem.  Something like user.del would be a reasonable default xattr name.  The actual xattr name can be configured with the mdt.*.job_xattr_del parameter.

Trash support for a striped directory

It would better useful to implement a virtual ".Trash" subdirectory accessible in each directory in the filesystem that can be used to browse files/directories in the trash can Trash Can and access them for recovery.

The FID of the ".Trash" directory is derived from the FID of the parent directory (pFID), by looking up the corresponding "stub" directory with the FID-named directory: ".lustre/trash/MDTXXXX/UID/pFID". Essentially ".Trash" under each normal directory is just a virtual shortcut to the stub directory (if the parent is not a striped dir) that is accessible in each directory if specified by name ".Trash". The files/directories under ".Trash" directory can be access via normal POSIX file system API such as via readdir()/stat()/getxattr() so that it can be used by normal tools such as "ls -l .Trash/" or "find .Trash" to locate files for restoration or permanent removal.  If there are no deleted files under a specific directory, then the virtual .Trash directory will not be accessible, and will return -ENOENT for any lookup.

For a striped directory, its ".Trash" directory is also a vitual striped directory with each stripe on the same location (MDTs) where the shard FID is the FID of the corresponding stub directory on that MDT. If the stub directory on a certain MDT does not exist (or not create yet), the client lookup() or readdir() under ".Trash" directory should skip the stripe. The master FID of the virtual ".Trash" directory could be same with the FID of the parent directory but with f_ver setting with 1 (FID_VERSION_TRASH = 1) to distinguish them.

...

It should handle the case that a directory was restriped and the LMV layout was changed. In this case, the files under the directory will be migrated to another MDT. To simplify the implementation, we do not migrate the files according to the new LMV layout in the Trash Can. This may result in the lookup() operation will be issued to a wrong MDT and return -ENOENT  wrongly (after files in the trash can are restored). However, the readdir() operations will still return all the dir entries in the striped trash even if the parent LMV layout was re-striped and changed, since the parent directory FID (pFID) will remain the same as before restriping. Maybe it needs to migrate the files restored from the trash can to the appropriate shard according to their name hash once the LMV layout has been changed.

Orphans in Trash Can

For an orphan file, it means the file is still opened (but not closed) by a certain user. Upon its last unlink, it can directly move into the trash can and mark with LUSTRE_ORPHAN_FL | LUSTRE_TRASH_FL. And the orphans file can not be permanently deleted from the trash can until its last close(). 

...