Incidence matrix approach for calculating readiness levels

Abstract

Contemporary system maturity assessment approaches have failed to provide robust quantitative system evaluations resulting in increased program costs and developmental risks. Standard assessment metrics, such as Technology Readiness Levels (TRL), do not sufficiently evaluate increasingly complex systems. The System Readiness Level (SRL) is a newly developed system development metric that is a mathematical function of TRL and Integration Readiness Level (IRL) values for the components and connections of a particular system. SRL acceptance has been hindered because of concerns over SRL mathematical operations that may lead to inaccurate system readiness assessments. These inaccurate system readiness assessments are called readiness reversals. A new SRL calculation method using incidence matrices is proposed to alleviate these mathematical concerns. The presence of SRL readiness reversal is modeled for four SRL calculation methods across several system configurations. Logistic regression analysis demonstrates that the proposed Incidence Matrix SRL (IMSRL) method has a decreased presence of readiness reversal than other approaches suggested in the literature. Viable SRL methods will foster greater SRL adoption by systems engineering professionals and will support system development risk reduction goals.

Keywords

Technology Readiness Level (TRL) / integration readiness level (IRL) / system readiness level (SRL) / system readiness / graph theory / systems engineering

References

[1] Azizian N, Sarkani S, Mazzuchi T. A comprehensive review and analysis of maturity assessment approaches for improved decision support to achieve efficient defense acquisition. Lecture Notes in Engineering And Computer Science, 2009, 2179(1): 1150-57.

[2] Azizian N, Mazzuchi T, Sarkani S, Rico D F. A framework for evaluating technology readiness, system quality, and program performance of U. DoD Acquisition Systems Engineering, 2011, 14(4): 410-426.

[3] Balbuena C. Incidence matrices of projective planes and of some regular bipartite graphs of girth 6 with few vertices. Society for Industrial and Applied Mathematics Journal on Discrete Mathematics, 2008, 22(4): 1351-1363.

[4] Baron N T, Holland O T, Marchette D J, Wallace S E. Integrating through the fire control loop, 2011

[5] Bilbro J W. A suite of tools for technology assessment. AFRL Maturity Conference, Virginia Beach, VA, September 11–13, 2007

[6] Box G E P, Draper N R. Empirical model building and response surfaces, 1987, New York, NY: John Wiley & Sons, Inc..

[7] Bowles J B. An assessment of RPN prioritization in a failure modes effects and criticality analysis. Journal of IEST., 2004, 47: 51-56.

[8] Dacus C L. Improving acquisition outcomes through simples system technology readiness metrics. Defense Acquisition Review Journal, 2012, 19(4): 444-461.

[9] DeNezza E J, Casey A G. Future space system acquisitions: is the key “what” or “when”?. Defense Acquisition Technology and Logistics, 2014, 9: 9-14.

[10] Department of Defense Technology Readiness Assessment (TRA) Deskbook, Washington, DC., 2009

[11] Diestel R. Graph theory, 2010, 4th ed. Heidelberg: Springer.

[12] Engle M, Sarkani S, Mazzuchi T. Technical maturity evaluations for sensor fusion technologies. IEEE Applied Imagery Pattern Recognition Workshop (AIPRW), 2009 14-16.

[13] Forbes E, Volkert R, Gentile P, Michaud K, Sondi T. Implementation of a methodology supporting a comprehensive system-of-systems maturity analysis for use by the littoral compact ship mission module program. INCOSE Chesapeake Chapter Dinner Meeting, 19 August 2009, 2009

[14] Fulkerson D R, Gross O. Incidence matrices and interval graphs. Pacific Journal of Mathematics, 1965, 3: 835-855.

[15] Government Accountability Office Best practices: better management of technology development can improve weapon system outcomes, 1999

[16] Government Accountability Office Best practices: stronger practices needed to improve DOD technology transition processes, 2006

[17] Government Accountability Office Defense acquisition: assessment of selected weapon systems, 2013

[18] Garrett R K, Anderson S, Baron N T, Moreland J D. Managing the interstitials, a system of systems framework suited for the ballistic missile defense system. Systems Engineering, 2011, 14(1): 87-109.

[19] Gross J L, Yellen J. Graph theory and its applications, 2006, Boca Raton: Chapman & Hall/CRC.

[20] Hosmer D W, Lemeshow S. Applied logistic regression, 2000, 2nd Ed.

[21] Kober B J, Sauser B. A case study in implementing a system maturity metric. 29th American Society of Engineering Management Conference, October, 2008, West Point, NY., 2008

[22] Kujawski E. The trouble with the system readiness level (SRL) index for managing the acquisition of defense systems. 13th Annual National Defense and Industrial Association Systems Engineering Conference, 24. San Diego, CA, 2010

[23] Kujawski E. Analysis and critique of the system readiness level. Institute of Electrical and Electronics Engineers Transactions on Systems, Man, and Cybernetics-Part A, 2013, 99: 1-9.

[24] Long J M. Integration readiness levels. IEEE Aerospace Conference, 5–12 March 2011, 2011

[25] Mahafza S, Componation P, Tippett D. A Performance-based technology assessment methodology to support DOD acquisition. Defense Acquisition Review Journal, 2004, 11(3): 268-283.

[26] Malone P, Wolfarth L. System-of-systems: an architectural framework to support development cost strategies. IEEE Aerospace Conference, 2012

[27] Mandelbaum J. Enabling technology readiness assessments (TRAs) with systems engineering. National Defense Industrial Association 8th Annual Systems Engineering Conference, 2005

[28] Mankins J C. Approaches to strategic research and technology (R&T) analysis and road mapping. Acta Astronautica, 2002, 51(1): 3-21.

[29] Marchette D. An analysis of system readiness functions. Joint Mathematics Meetings, San Diego, CA, 30 January 2013, 2013

[30] Matlab (Version 7.12.0.635, R2011a), Computer Software, Mathworks.

[31] McCabe T J. A complexity measure. IEEE Transactions On Software Engineering, 1976, 4: 308-320.

[32] McConkie E B. A systems engineering approach to mathematical properties of system readiness levels. (Doctoral Dissertation). The George Washington University, ProQuest UMI No. 3543951, 2013

[33] McConkie E B, Mazzuchi T A, Sarkani S, Marchette D. Mathematical properties of system readiness levels. Systems Engineering, 2013, 16(4): 391-400.

[34] Meier S R. Best project management and systems engineering practices in the preacquisition phase for federal intelligence and defense agencies. Project Management Journal, 2008, 39(1): 59-71.

[35] Minitab (Version 17), (Computer Software), Minitab, Inc.

[36] Pampel, F. C. (2000). Logistic regression: a primer. Quantitative Applications in the Social Sciences. Ed. Michael S. Lewis-Beck, Sage Publications, Inc.

[37] Peduzzi P, Concato J, Kemper E, Holford T R, Feinstein A R. Simulation study of the number of events per variable in logistic regression analysis. Journal of Clinical Epidemiology, 1996, 49(12): 1373-1379.

[38] Peng C-Y J, Lee K K, Ingersoll G M. An introduction to logistic regression analysis and reporting. Journal of Educational Research, 2002, 96(1): 3-14.

[39] Raftery A E. Bayesian model selection in social research. Sociological Methodology, 1995, 25: 111-163.

[40] Ramirez-Marquez J E, Sauser B J. System development planning via system maturity optimization. Institute of Electrical and Electronics Engineers Transactions on Engineering Management, 2009, 56(3): 533-548.

[41] Sauser B J, Forbes E, Long M, McGrory S E. Defining an integration readiness level for defense acquisition. International Symposium of International Council on Systems Engineering. Singapore, 2009, 2009

[42] Sauser B, Ramirez-Marquez J E. Multi-objective optimization of system capability satisficing in defense acquisition. Defense Technology Information Center, Accession No., 2012, 56334330: April 2012

[43] Sauser B J, Ramirez-Marquez J E, Henry D, DiMarzio D. A system maturity index for the systems engineerig life cycle. International Journal of Industrial and Systems Engineering, 2008, 3(6): 673-691.

[44] Sauser B J, Ramirez-Marquez J E, Magnaye R B, Tan W. A systems approach to expanding the technology readiness level within defense acquisition. International Journal of Defense Acquisition Management, 2008, 1: 39-58.

[45] Sauser B J, Ramirez-Marquez J E, Nowicki D, Deshmukh A, Sarfaraz M. Development of systems engineering maturity models and management tools. DTIC Technical Report, 2011-TR-014, 2011

[46] Sauser, B. J., Ramirez-Marquez, J. E., Verma, D., & Gove, R. (2006). Determining system interoperability using an integration readiness level. Stevens Institute of Technology, 21.

[47] Sauser, B. J., Verma, D., Ramirez-Marquez, J. E., & Gove, R. (2006). From TRL to SRL: the concept of system readiness levels. Conference on Systems Engineering Research, Stevens Institute of Technology, 1-10.

[48] Smith, J. D. (2005). An alternative to technology readiness levels for non-developmental item (NDI) software. Proceedings of the 38th Annual Hawaii International Conference on System Sciences, HICSS’05.

[49] Stevens S S. On the theory of scales of measurement. Science, 1946, 103(2684): 677-680.

[50] Tan W, Ramirez-Marquez J E, Sauser B. A probabilistic approach to system maturity assessment. Systems Engineering, 2011, 14(3): 279-293.

[51] Tan W, Sauser B J, Ramirez-Marquez J E, Magnaye R B. Multiobjective optimization in multifunction multicapability system development planning. Systems, Man, and Cybernetics: Systems, IEEE Transactions on, 2013, 43(4): 785-800.

[52] Tetlay A, John P. Determining the lines of system maturity, system readiness and capability readiness in the system development lifecycle. 7th Annual Conference on Systems Engineering Research, 2009, 2009

[53] Townsend J T, Ashby F G. Measurement scales and statistics: the misconception misconceived. Psychological Bulletin, 1984, 96(2): 394-401.

[54] Valerdi R, Kohl R. An approach to technology risk management. Engineering Systems Division Symposium, March, 2004 29-31.

[55] Velleman P F, Wilkinson L. Nominal, ordinal, interval, and ratio typologies are misleading. The American Statistician, 1993, 47(1): 65-72.

[56] Volkert R, Stracener J, Yu J. Incorporating a measure of uncertainty into systems of systems development performance measures. Systems Engineering, 2013 1-17.