Venti - Details

Details

Venti is a user space daemon. Clients connect to Venti over TCP and communicate using a simple RPC-protocol. The most important messages of the protocol are listed below. Note that there is no message to delete an address or modify data at a given address.

• read(score, type), returns the data identified by score and type
• write(data, type), stores data at the address calculated by SHA-1 hashing data, combined with type.

The data block stored by Venti must be greater than 512 bytes in length and smaller than 56 kilobytes. So, if a Venti user/client wants to store larger data blocks, it has to make a datastructure (which can be stored in Venti). For example, Fossil uses hash trees to store large files. Venti itself is not concerned with the contents of a data block; it does however store the type of a data block.

The design of Venti has some interesting consequences:

• Since writes are permanent, the file system is append-only (which allows for a simple implementation with lower chance of data-destroying bugs); no file system fragmentation occurs.
• Clients can verify the correctness of the server: the score of the returned data should be the same as the address requested. Since SHA-1 is a cryptographically secure hash, it is computationally infeasible to fabricate data.
• Data cannot be overwritten. If an address is already present, the data is already present.
• There is little need for user authentication: Data cannot be deleted, and can be read only if the score is known. The only potential problem is a user filling up the disks.
• Data can be compressed without making the disk structure complicated.

The data blocks are stored on hard drives. The disks making up the available storage, typically a RAID 5 configuration, is called the data log. This data log is split up in smaller pieces called arenas, which are sized so they can be written to other media such as CD/DVD or magnetic tape. Another set of hard drives is used for the index, which maps scores to addresses in the data log. The data structure used for the index is a hash table with fixed-sized buckets. Venti relies on the scores to be randomly distributed so buckets do not fill up. Since each lookup costs one disk seek time, an index usually consists of multiple small SCSI hard drives with low access time.