From Research to Practice: Experiences Engineering a Production Metadata Database for a Scale Out File System
Venue
Proceedings of the 12th USENIX Conference on File and Storage Technologies (FAST 2014), USENIX
Publication Year
2014
Authors
Charles Johnson, Kimberly Keeton, Charles B. Morrey III, Craig A. N. Soules, Alistair Veitch, Stephen Bacon, Oskar Batuner, Marcelo Condotta, Hamilton Coutinho, Patrick J. Doyle, Rafael Eichelberger, Hugo Kiehl, Guilherme Magalhaes, James McEvoy, Padmanabhan Nagarajan, Patrick Osborne, Joaquim Souza, Andy Sparkes, Mike Spitzer, Sebastien Tandel, Lincoln Thomas, Sebastian Zangaro
BibTeX
Abstract
HP’s StoreAll with Express Query is a scalable commercial file archiving product
that offers sophisticated file metadata management and search capabilities. A new
REST API enables fast, efficient searching to find all files that meet a given set
of metadata criteria and the ability to tag files with custom metadata fields. The
product brings together two significant systems: a scale out file system and a
metadata database based on LazyBase. In designing and building the combined
product, we identified several real-world issues in using a pipelined database
system in a distributed environment, and overcame several interesting design
challenges that were not contemplated by the original research prototype. This
paper highlights our experiences.
