DiPerF / ServMark (Distributed Performance testing Framework)

   DiPerF is a distributed performance‑testing framework, aimed at simplifying and automating service performance evaluation. DiPerF coordinates a pool of machines that test a target service, collects and aggregates performance metrics, and generates performance statistics. The aggregate data collected provide information on service throughput, on service ‘fairness’ when serving multiple clients concurrently, and on the impact of network latency on service performance. Furthermore, using this data, it is possible to build predictive models that estimate a service performance given the service load. We have tested DiPerF on 200+ machines and with 3000+ clients on four testbeds, TeraGrid, Grid3, CS@Uchicago cluster, and PlanetLab, and explored the performance of instance creation, gridftp server, job submission services (pre‑WS GRAM and WS GRAM), MDS (Index Service), DI-GRUBER and other services built or included with Globus Toolkit® 3.2, GT3.9.5 and GT4.0. In a schematic way the framework is presented in the next figure.

, the second phase of DiPerF and GrenchMark, blends the two projects for providing not only better service perfromance evaluation, but also automated Grid environment testing. ServMark is a novel testing architecture and the integration of DiPerF and GrenchMark brings together into one single package two orthogonal issues in performance evaluation, namely the ability to test environments and services in a distributed and scalable way, and the capacity to generate and run dynamic test workloads with complex structure. The ServMark project, jointly developed by people from U. Chicago, TUDelft, UBC and UPB, has been accepted as part of the Globus Incubator project.

DiPerF Download:
  • April 2005: The first release of the framework is available (sources).
  • April 2006: The second release (enhanced analyzer and submitters + bug fixes) of the framework is available (sources, RPM-Suse.9.x, and DEB-Ubuntu.6.x).
  • Additional "tester" examples are posted here.

    DiPerF Project Participants: Acknowledgments:
    • to the Globus Team for using the framework for various test scenarios.
    • to Jennifer Schopf for her suggestions to include new testing schemas.
    • to Peter Gardfjall and Johan Tordsson for their feedback and code patches.
    • to Lingyn Yang for requiring additional scenarios

    ServMark Download:
  • July 2006: A first beta release of the ServMark project is available (sources).
  • July 20, 2006: Project moved to Globus Incubator/ServMark

    ServMark Project Participants:

    DiPerF / ServMark Documentation: DiPerF Papers:
    • DiPerF: An automated DIstributed PERformance testing Framework, C. Dumitrescu, I. Raicu, M. Ripeanu, I. Foster - GW2004, Pittsburgh, PA ([ps|pdf])
    • The Design, Performance, and Use of DiPerF: An automated DIstributed PERformance testing Framework, I. Raicu, C. Dum itrescu, M. Ripeanu, I. Foster - Special Issue of Journal on Grid Computing ([ps |pdf])
    • A Performance Study of the Globus Toolkit and Grid Se rvices via DiPerF, An Automated Distributed Performance Testing Framework, I . Raicu, MSc Thesis, The University of Chicago, May 2005 ([ps|pdf])

    • A Scalability and Performance Measurements of a Distributed Grid Resource Broker, C. Dumitrescu, I. Foster, I. Raicu - iVDGL/GriPhyN TechReport
    • Zebra: The Globus Stripped GridFTP Framework and Server, Allcock, B., Bresnahan, J., Kettimuthu, K., Link, M., Dumitrescu, C., Raicu, I., Foster, I. - SC05, Seattle, WA ([ps|pdf])
    • DI-GRUBER: a Distributed Approach to Grid Resource Brokering, C. Dumitrescu, I. Raicu, I. Foster - SC05, Seattle, WA ([ps|pdf])

    • A nice MDS testing picture (MDS community testing call-meeting 2005):

    • Extending a Distributed Usage SLA Resource Broker to Support Dynamic Grid Environments, C. Dumitrescu (UChicago), I. Raicu (UChicago), I. Foster (UChica go/Argonne) - submitted to EuroSys06 ([ps|pdf ])
    • A Layered Framework for Connecting Client Objectives and Resource Capabil ities - A. Dan (IBM), C. Dumitrescu (TUDelft/CoreGrid), K. Ranganathan (IBM), M. Ripeanu (UBC), - Journal of Digital Libraries and the Cooperative Information Systems 06
    • Problems for Resource Brokering in Large and Dynamic Grid Environments, C. Dumitrescu (TUDelft/CoreGrid) - Euro-Par 2k6, Dresden, 2006 ([ps|pdf])

    • Distributed Monitoring and Information Services for the Grid, J. M. Schopf, NeSC, September 2005 ([ps|pdf|ppt])
    • Monitoring and Discovery in a Web Services Framework: Functionality and P erformance of Globus Toolkit MDS4, Jennifer M. Schopf (Argonne), Ioan Raicu (UChicago), La ura Pearlman (ISI), Neill Miller (Argonne), Carl Kesselman (ISI), Ian Foster (Argonne/UChicago), Mike D'Arcy (Argonne) submitted to HPDC 2006, available as # MCS Preprint #ANL/MCS-P1315-0106, January 2006.

    • Scalable Grid-wide capacity allocation with the SweGrid Accounting System (SGAS), P. Gardfjall, E. Elmroth, L. Johnsson, O. Mulmo, T. Sandholm, Submitted to CONCURRENCY AND COMPUTATION: PRACTICE AND EXPERIENCE, 2006 ([ps|pdf])
    • Capacity Allocation Mechanisms for Grid Environments, P. Gardfjall, licentiate thesis, Umea University, october 2006 ([ps|pdf])
    • A standards-based Grid resource brokering service supporting advance reservations, coallocation and cross-Grid interoperability, E. Elmroth, J. Tordsson, to be submitted to CONCURRENCY AND COMPUTATION: PRACTICE AND EXPERIENCE, November 2006 ([ps|pdf])

    ServMark Papers:
    • ServMark (GrenchMark + DiPerF): A Proposal for Globus Project Incubation, C. Dumitrescu (UMUE), A. Iosup (TUD), D. Epema (TUD), I. Raicu (UChicago), I. Foster (ANL/UChicago), M. Ripeanu (UBC) ([ps|pdf])
    • Towards ServMark, an Architecture for Testing Grid Services, M. Andreica (UPB), N. Tapus (UPB), C. Dumitrescu (UMUE), A. Iosup (TUD), D. Epema (TUD), I. Raicu (UChicago), I. Foster (ANL/UChicago), M. Ripeanu (UBC) ([ps|pdf])
    • ServMark: A Distributed Grid and Services Testing Framework - GGW slides (ps|pdf) and with GlobusAlliance at SC06 (link)
    • Towards ServMark, an Architecture for Testing Grids - DSL Seminar Presentation ([ps|pdf])

    Email: ServMark-User
    Last Updated: 06/01/2007