Versions Compared

Old Version 12

changes.mady.by.user Michael MacDonald

Saved on

compared with

New Version 13

changes.mady.by.user Robert Read

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

There is a 5TB size limit in AWS S3 (TODO: check limits for other S3 implementations) , so larger files will need to be sharded over multiple S3 objects. The metadata of the shard UUIDs and their respective ranges would be stored in a container metadata object, and the name of this object would be is returned to the agent as a key. To distinguish container "shard metadata" objects from normal objects, the type in the URL could be changed to s3c, e.g.:

...

name includes al ".shm" extension after the UUID. 

s3://<bucketName>/<prefix>/o/UUID.shm