Storage Resource Manager - Use of SRMs in Real Applications

Use of SRMs in Real Applications

The SRM interfaces have been cooperatively defined and multiple implementations developed in the US and Europe. LBNL has introduced the concepts and subsequently led a coordinated effort of defining a community-based common interface. Several implementations have been deployed in various applications including HEP, HENP, ESG as well as new application domains, such as Fusion simulation, biology, and others. Some specifics of SRM usage to date are:

• LBNL’s SRMs have been used in production over the last few years to support intensive robust data movement between BNL to NERSC at a rate of about 10,000 files (about 1 TB) per week in an automated fashion. This arrangement resulted in a 50X reduction in the error rates, from 1% to 0.02% in the STAR project.
• In one application, called GridCollector, SRMs were used in combination with an efficient indexing method to greatly speed up the analysis of STAR. In several cases the analysis task was performed in a day as compared to previous efforts where scientists waited for months to sift out the relevant data. This work received recognition with a Best Paper Award in ISC’05.
• The SRM collaboration has grown as a grass root activity between LBNL, FNAL, and BNL, and later CERN and RAL. Consequently, a common interface was developed, and this activity continues at this time. This standard has been adapted by the WLCG collaboration.
• SRMs have been used in production by several facilities including BNL, NERSC, FNAL, CERN, TJNAF, ORNL and NCAR, and other facilities in Europe and Asia.
• Another example of a successful deployment is the SRM-dCache developed at FNAL. It is widely deployed for use in the CMS project, and it interoperates with the SRM-Castor at CERN. This effort demonstrated the usefulness of SRMs by achieving sustained SRM-to-SRM managed transfers from Castor to FNAL dCache and onto tape at a rate between 40 and 60 MB/s.
• SRMs are used by TJNAF to provide the CLAS and Lattice QCD collaborations with remote access to the JASMine mass storage system. Such access has allowed researchers to utilize computing resources at universities and other collaborating institutions to process and analyze data weeks or months sooner than if done using only TJNAF computing resources.
• LBNL’s SRMs have been used in production in the Earth Systems Grid (ESG) Project to provide transparent access from multiple remote storage systems at NERSC, NCAR, ORNL, LLNL, and LANL, including HPSS and NCAR-MSS. A disk version of an SRM has been used by the ESG portal to manage the disk space when it is shared as file storage for multiple clients.
• The use of SRMs for the CPES fusion project for large-scale robust data movement will be incorporated into workflow engines as part of the SDM center activities.

List of Storage Resource Manager software:

• APTARE StorageConsole
• BMC MAINVIEW Storage Resource Manager
• Computer Associates BrightStor Storage Resource Manager
• CreekPath Acuity (Storage Resource Manager is a component of Acuity)
• DataCore Software Corporation SANmaestro Storage Resources and Performance Management and Analysis
• EMC ControlCenter
• BULL Calypso
• Engenio Storage Performance Analyzer (OEM AppIQ StorageAuthority Suite)
• HDS HiCommand Storage Services Manager (OEM AppIQ StorageAuthority Suite)
• HP SANworks
• HP Storage Essentials (formerly AppIQ StorageAuthority Suite)
• IBM TotalStorage Productivity Center for Data (formerly IBM Tivoli Storage Resource Manager)
• SGI InfiniteStorage Resource Manager (OEM AppIQ StorageAuthority Suite)
• Sun StorEdge Enterprise Storage Manager (OEM AppIQ StorageAuthority Suite)
• Fermilab SRM Fermilab implementation of the Storage Resource Manager interface
• Berkeley Storage Manager Berkeley Lab implementation of the Storage Resource Manager interface
• Veritas Command Central Storage from Symantec