Managing obsolescence of embedded hardware and software in secure and trusted systems

Abstract

Obsolescence of integrated systems which contain hardware and software is a problem that affects multiple industries and can occur for many reasons, including technological, economic, organizational, and social factors. It is especially acute in products and systems that have long life cycles, where a high rate of technological innovation of the subcomponents result in a mismatch in life cycles between the components and the systems. While several approaches for obsolescence forecasting exist, they often require data that may not be available. This paper describes an approach using non-probabilistic scenarios coupled with decision analysis to investigate how particular scenarios influence priority setting for products and systems. Scenarios are generated from a list of emergent and future conditions related to obsolescence. The key result is an identification of the most and least disruptive scenarios to the decision maker’s priorities. An example is presented related to the selection of technologies for energy islanding, which demonstrates the methodology using six obsolescence scenarios. The paper should be of broad interest to scholars and practitioners engaged with enterprise risk management and similar challenges of large-scale systems.

Keywords

enterprise risk management / diminishing manufacturing sources and material shortages / scenario-based preferences / systems engineering / deep uncertainty / product life cycle

Introduction

Technology obsolescence is an important problem for managers in the private and public sectors. The U.S. Department of Defense (DoD) defines obsolescence as “a lack of availability of an item or raw material resulting from statutory and process changes, as well as new designs. Obsolescence deals with the process or condition by which a piece of equipment becomes no longer useful, or a form and function no longer current or available for production or repair” (^{DoD, 2015a}).

Industries with high rates of innovation and where consumer preferences rapidly shift are prone to obsolescence, and events such as market entry by a competitor or safety concerns can precipitate the obsolescence of a product (^{Song and Zipkin, 1996}). Obsolescence can occur for many reasons: newer technology becomes preferred to the current technology, the item no longer functions as intended, regulations ban the use of the product, the product is no longer supported or maintained, or when there is simply no longer consumer demand for the item (^{DoD, 2016}).

In the literature, the most common type of obsolescence is related to components or parts within larger systems, especially microelectronics (^{Solomon et al., 2000}; ^{Singh and Sandborn, 2006}; ^{Romero Rojo et al., 2010}; ^{Sandborn et al., 2011}; ^{Romero Rojo et al., 2012}; ^{Sandborn, 2013}). In this context, obsolescence is characterized as the “loss or impending loss of original manufacturers of items or suppliers of items or raw materials” and is referred to as diminishing manufacturing sources and material shortages (DMSMS) (^{Sandborn, 2013}). In this case, there is still a demand for a part or material, but there is a lack of supply. This differs from a different type of obsolescence where a stock of excess inventory exists for which there is no longer demand. DMSMS-type obsolescence is common in low-volume, sustainment-dominated sectors such as defense, aviation, and infrastructure, where the costs of sustaining the system over an extended life cycle exceed the costs of original acquisition (^{Singh and Sandborn, 2006}). The high rate of technological innovation in the electronics industry, coupled with the slow and costly process of qualification and certification for new components within the system, creates a mismatch in life cycles between the components and the systems (^{Singh and Sandborn, 2006}). Long system development times can cause electronic components to become obsolete even before the system is fielded (^{Sandborn, 2013}).

^{Romero Rojo et al., 2012} presented a holistic view of obsolescence, beyond the commonly discussed aspect of microelectronic components. They described how software, mechanical parts, materials, test equipment, processes and procedures, and skills can become obsolete or contribute to system obsolescence. ^{Sandborn (2007)} described the issue of software obsolescence, noting that software and hardware must be concurrently sustained. ^{Wnuk et al. (2013)} explored how system requirements themselves can become obsolete. Human capital obsolescence is reviewed by ^{de Grip (2006)}, characterizing how skills can be lost or fall in demand in the job market. ^{Reilly (2013)} described the causes and quantification of obsolescence of real estate.

Management responses to obsolescence issues vary from lifetime buy and last-time buy strategies to complete system redesigns (^{Romero Rojo et al., 2012}; ^{Sandborn, 2013}), costing anywhere from a few thousand dollars to tens of millions of dollars (^{DoD, 2015b}). Obsolescence also opens up the possibility of inadvertently acquiring counterfeit components through secondary markets when trusted sources are no longer available (^{Romero Rojo et al., 2012}; ^{Collier et al., 2014}; ^{DiMase et al., 2016}).

There currently exists a research gap focused on the need for methods and approaches to consider the impacts of obsolescence under conditions without much information. Necessitated by uncertainties in the timing and degree of obsolescence, current methods for obsolescence forecasting build upon techniques from operations research, economics, and reliability theory (^{Song and Zipkin, 1996}; ^{Porter, 1998}; ^{Solomon et al., 2000}; ^{Singh and Sandborn, 2006}; ^{Sandborn et al., 2011}). These methods are typically data-intensive, relying upon information such as units shipped over the component’s life cycle, part life times, and numerous costs which may be difficult to parameterize. However, in complex systems and under deep uncertainty, these inputs may be unknown, or costly to research. Therefore, a tool for ranking and screening scenarios of obsolescence can add value by focusing on a subset of highly impactful scenarios for further detailed analyses.

To address this gap, this paper describes a method for assessing the impacts of obsolescence on management priorities through the use of non-probabilistic scenarios. We develop and demonstrate a model of how decision maker preferences change across a number of obsolescence scenarios using a scenario-based preferences model. By assessing the degree to which decision makers reprioritize system initiatives or product alternatives in the face of different scenarios, we can identify the most and least disruptive scenarios as well as the alternatives that are robust or vulnerable to scenarios. We demonstrate this problem formulation through an example featuring the selection among energy technologies. The paper concludes with a summary and recommendations for future research.

Methods

This section describes a model to identify which scenarios of obsolescence are most and least disruptive to priority setting in large-scale systems.

A scenario-based preferences model is described in ^{Karvetski et al. (2009; 2011a; 2011b)}; ^{Karvetski and Lambert (2012}); ^{Lambert et al. (2012}; ²⁰¹³); ^{Connelly et al. (2015)}; ^{Hamilton et al. (2015)}; ^{Hamilton et al. (2016)}; ^{Collier et al. (2018)}; and ^{Collier and Lambert (2018}; ²⁰¹⁹). This model builds upon multicriteria decision analysis (MCDA) by recognizing that decision maker preferences, represented as criteria weights, are dynamic as new information about the world comes to light. As we learn new information, or imagine ourselves in different futures, our preferences will likely change relative to our preferences today. The scenario-based preferences model allows for the exploration of how various initiatives or alternatives perform across different scenarios, and facilitates inter-scenario comparisons to identify the most and least disruptive scenarios.

The model is formulated as follows. Assume that the decision maker has a set of n alternatives representing various systems or products, {a₁, a₂, …, a_n}. The decision maker also has a set of m criteria, representing various system objectives, {c₁, c₁,…, c_m}. The level of performance along a criterion c_j for a generic alternative a can be defined by a partial value function, v_j(a), which assigns a real number to each alternative such that the performance of alternative a is preferred to the performance of some other alternative b if and only if v_j(a)>v_j(b), or a decision maker is indifferent between the two if and only if v_j(a)=v_j(b). A linear-additive value function assigning a real value to alternative a_i, accounting for all m criteria can be defined as follows:

(1)V({{\displaystyle a}}_i)={{\displaystyle \Sigma }}_{j=\mathrm{1}}^m{{\displaystyle w}}_j\cdot {{\displaystyle v}}_j({{\displaystyle a}}_i),

where w_j is a scaling constant (“weight”) assigned to the jth criterion. Weights are typically defined such that they sum to 1. Details on the linear-additive value function and weights can be found in ^{Belton and Stewart (2002)}.

To extend the MCDA model to accommodate scenarios, we assume that we have a set of q scenarios, {s₁, s₂, …, s_q}, where each scenario consists of one or more emergent and future conditions. These emergent and future conditions represent uncertain factors that may influence future system operations or decision maker preferences. Emergent and future conditions can be individual trends or triggering events and can be combined to create unique scenarios (^{Karvetski and Lambert, 2012}). They can be generated through literature reviews or by brainstorming, including thinking about traditional “PESTLE” factors (political, economic, social, technological, legal, environmental).

Each initiative a_i has a level of performance v_j(a_i) on criterion c_j. Under scenario s_k, there is a scenario-specific weight w_jk for criterion c_j. Given these elements, a linear-additive value function can be defined that assigns a score to initiative a_i under scenario s_k:

(2)V{{\displaystyle ({{\displaystyle a}}_i)}}_k={{\displaystyle \Sigma }}_{j=\mathrm{1}}^m{{\displaystyle w}}_{jk}\cdot {{\displaystyle v}}_j({{\displaystyle a}}_i).

The value function can be used to create prioritized lists of initiatives across scenarios. Within each scenario, initiatives can be rank ordered, resulting in a prioritization within each scenario considered. Given any two ordinally ranked lists, the degree of correlation between the two sets can be measured using the Kendall’s Tau statistic, t (^{Kendall, 1938}). The values of t exist between ‒1 and 1, where 1 is a perfect positive correlation and ‒1 is a perfect negative correlation. Mathematically, Kendall’s Tau compares the number of concordant and discordant pairs of measurements within two ordered sets (^{Abdi, 2007}). Given two pairs of rankings (r_i⁰, r_i^k) and (r_j⁰, r_j^k), the rankings are concordant if both r_i⁰>r_i^k and r_j⁰>r_j^k or both r_i⁰<r_i^k and r_j⁰<r_j^k. The rankings are discordant if both both r_i⁰>r_i^k and r_j⁰<r_j^k or both r_i⁰<r_i^k and r_j⁰>r_j^k (^{de Siqueira Santos et al., 2014}). Kendall’s Tau is calculated as

(3)\tau ={\displaystyle \frac{{{\displaystyle n}}_{con}-{{\displaystyle n}}_{dis}}{\mathrm{0.5}n(n-\mathrm{1})}},

where n_con is the number of concordant pairs, n_dis is the number of discordant pairs, and n is the total number of ranked observations.

Example

This example builds upon an effort that is described in ^{Karvetski and Lambert (2012)} and ^{Hamilton et al. (2016)} related to the selection of energy islanding technologies considering future scenarios. The example involves a scientific research facility composed of multiple buildings connected to the public grid which was experiencing power outages. Due to the nature of the research conducted at this facility, power outages resulted in the sensitive electric equipment being damaged. Weeks of scientific research and data collection could potentially be lost with each outage. Consistent, quality power was a requirement to keep humidity, temperature, and air quality of the laboratories within strict limits. Therefore, the goal of the facility managers was to investigate technologies for obtaining consistent, reliable, and quality electric power (^{Hamilton et al., 2016}).

First, a set of seven decision criteria was developed. Table 1 shows the criteria list and the weights assigned to each criterion. Each criterion is briefly described below.

• C01. Reduce costs. This criterion describes the ability of each alternative to realize cost savings. Specifically, the lifecycle costs accrued relative to taking no action were estimated, over a twenty-year horizon.

• C02. Quality, prime power. This was the main consideration. It is related to the number of outages avoided relative to the number which would have been expected based on the past.

• C03. Energy security. Beyond uninterrupted power, other energy security goals were of importance, including reduced energy consumption, increased energy efficiency, increased use of renewable and/or alternative energy sources, and assured access to sufficient energy.

• C04. Proof of principle. This criterion describes how the facility can serve as an example to other industry and government entities by its demonstration as a case study and dissemination of lessons learned.

• C05. Procure funding. The ability to procure the necessary funding for the upgrade, as measured in years of waiting time.

• C06. Reduce vulnerability. A secondary benefit to the facility was that certain alternatives may function to reduce the overall vulnerability of the system to threats.

• C07. Environmental impact. Some alternatives may provide positive environmental benefits, measured in terms of tons of CO₂ emissions reduced per year.

Criteria weights reported in Table 1 were derived from facilitated discussions among stakeholders with diverse perspectives. This was conducted at a working group meeting and included participants such as facility energy managers, officials from utility companies and the government, building tenants, and private sector representatives. The resultant weights were generated from a consensus among stakeholders (^{Karvetski and Lambert, 2012}; ^{Hamilton et al., 2016}).

As with any situation involving the elicitation of information from experts and stakeholders, care must be taken to not introduce unintentional bias into the results. Following established best practices in elicitation techniques ensures that the collected data are of high quality (^{Morgan, 2014}).

Next, the energy technology alternatives were identified. They ranged from installation of natural gas powered microturbines for electricity, heating, and cooling (i.e., trigeneration), to simple alternatives such as burying the power lines so that they are not disrupted by falling trees and other natural and manmade hazards. The microgrid with backup generators would provide the facility with the ability to disconnect from the public grid in the case of an outage, and the solar photovoltaic (PV) alternative could allow for renewable energy generation and storage in case of outages. A no action alternative was also included to represent the status quo. Table 2 shows the alternatives along the top row, with the criteria along the first column. Within each cell, the degree to which a particular criterion is addressed by the particular alternative was entered. To facilitate the process, the respondent could select several qualitative responses, including “strongly agree,” “agree,” “somewhat agree,” and “disagree or N/A.” Each response was then converted into a numerical value score, v_j(a_i)={1, 0.67, 0.33, 0} for “strongly agree,” “agree,” “somewhat agree,” and “disagree or N/A,” respectively.

This information about criteria weights and value scores assigned to each alternative is enough to produce a ranking of alternatives, and represents the base (no scenario) case. To incorporate scenarios, we searched the literature on obsolescence to identify emergent and future conditions which can form various obsolescence scenarios. Obsolescence is a serious concern for energy technologies since the infrastructure is designed for a long service life, yet increasingly the technological components within them may become obsolete early in the system life cycle. For instance, microgrids are controlled by various hardware, software, and application layers that provide valuable functions like voltage and frequency regulation, generation scheduling and dispatch, and energy trading (^{Hirsch et al., 2018}). A number of general emergent and future conditions, related to technological, economic, and other factors, were identified from the literature and are displayed in Table 3.

From this general list, scenarios were generated by selecting one or more emergent and future conditions. Table 4 describes six scenarios, consisting of the emergent and future conditions identified above. The scenarios represent a combination of hardware, software, and operational factors which could contribute to the obsolescence of advanced energy technology systems as identified in the literature as important obsolescence and supply chain considerations. For example, ^{Davis and Sullivan (2017)} describe drivers of supply chain risk, with the top three being product quality issues, supplier viability, and demand volatility. These considerations are seen in the developed scenarios. For example, S1 (parts unavailable) and S4 (support terminated) relate to supplier viability, while S2 (software incompatible) and S5 (flaw in key part) are related to quality issues. The scenario S3 (difficult to upgrade) is related to the high costs of system qualification and certification, and high costs and times for technology insertion and refresh, as described as key issues by ^{Solomon et al. (2000)} and ^{Singh and Sandborn (2006)}. Scenario S6 (workforce leaving) is based on the idea that beyond hardware and software, employee skills and knowledge are important factors contributing to obsolescence (^{Romero Rojo et al., 2010}), and industry and government supply chains face an aging workforce and pressures from outsourcing (^{Davis and Sullivan, 2017}).

Next, the influence of scenarios on preferences was explored by reweighting the criteria across scenarios. This can be done through a qualitative elicitation procedure. The relative change in importance of each criterion under each scenario is answered by the question, “under scenario s_k, is criterion c_j more or less important in comparison to the other criteria than in the baseline scenario, and if so, what is the extent of the change?”. The respondent can reply “increases,” “increases somewhat,” “no change,” “decreases somewhat,” or “decreases.” As an example, consider scenario S1. parts unavailable. In this scenario, important components of the energy system can no longer be procured. In this obsolescence scenario, it might become necessary to obtain the part from a secondary market, which may include untrusted sources of supply. This will increase the importance of ensuring that vulnerabilities are not introduced into the system. However, without the necessary stock of parts, the system may not operate with the intended functionality, so the importance of energy security increases as well. Facing a shortage of supply, it may be necessary to spend more money to design and test a solution for the system, resulting in a decrease in the importance of reducing costs.

A scaling constant, a, was defined that adjusts the baseline criterion weight up or down corresponding to the qualitative reply. A new scenario-specific weight, w_jk, is defined as w_jk=a∙w_j, and here we let values of a equal {8, 6, 1, 1/6, 1/8} for “increases,” “increases somewhat,” “no change,” “decreases somewhat,” and “decreases,” respectively. The adjusted weights were then normalized to sum to 1, and this procedure was repeated for each scenario to produce a unique set of criteria weights across each scenario. The scaling constant a is intended to be consistent with the process of swing weighting, as described by ^{Karvetski et al. (2009)}. The value a is meant to represent a “worth multiplier,” which represent the ratios of worst-to-best value comparisons within each criterion. The interpretation of the value alpha is that under a particular scenario, exchanging a high level of performance to a low performance on criterion i for a low level of performance to a high level of performance on criterion j is worth alpha times the value of making the exchange under the baseline scenario. Table 5 reports the qualitative assessments regarding the effects of scenarios on decision maker preferences, and Table 6 shows the resultant criteria weights across scenarios.

Following an MCDA approach (^{Belton and Stewart, 2002}), the values from Table 2 representing the performance of each alternative and the weights from Table 6 were integrated using the linear-additive value function Eq. (2) to produce value scores. The higher scores represent better performing alternatives. These scores are reported in Table 7. Using these scores, a unique ranking of energy alternatives can be produced within each scenario. A graph depicting these rankings is shown in Fig. 1. The triangle represents the rank in the baseline scenario, and the bars extending in each direction indicate the highest and lowest ranking achieved for that alternative across all of the scenarios. For instance, A05. microgrid of backup generators was ranked second in the baseline scenario, and across the other six scenarios, was ranked between first and third.

Finally, the disruptiveness of each scenario relative to the baseline was determined by calculating Kendall’s Tau Eq. (3). As described earlier, t can take values from ‒1 to 1. A value of 1 would indicate that there is perfect correlation between the prioritization of initiatives under the baseline scenario and under the scenario being considered. In other words, each ordinally ranked list of initiatives would be identical. Similarly, a value of ‒1 would indicate that the prioritization under a scenario was perfectly reversed relative to the baseline, which is theoretically the largest possible disruption in priorities. Therefore, lower values of t within the range [‒1, 1] indicate a larger disruption relative to the baseline, and higher values represent smaller disruptions. These values are reported in Table 8.

Discussion

From the preceding case study, we can identify several important results (Table 9). First, it appears that the alternatives can be placed into two groups, with the microgrid and both microturbine alternatives ranking highly across scenarios, and with the other alternatives performing poorly. This could help decision makers screen out lower-ranked alternatives from consideration, and focus more detailed analyses on the remaining alternatives. We also see that the alternative A01. no action is always ranked the lowest, indicating that this is an alternative that doesn’t meet the objectives of decision makers, and can be disregarded from further analysis.

Inspecting Table 8, it can be seen that three scenarios are tied for most disruptive. These scenarios are S3 (difficult to upgrade), S4 (support terminated), and S6 (workforce leaving), all scoring a τ value of 0.733. This indicates that each of these scenarios caused the same number of changes in priorities relative to the baseline. Scenario S5 (flaw in key part) was ranked next-most disruptive, with a score of 0.867, and finally S1 (parts unavailable) and S2 (software incompatible) had τ values of 1, indicating that they did not change the priorities relative to the baseline, and are therefore low-disruption scenarios. The number of scenarios with tied τ values is likely due to the low resolution of the case study. With only six alternatives, it is less likely that pairs of ranked alternatives will change relative to the baseline prioritization, limiting the utility of the Kendall’s Tau statistic. However, given a larger set of ranked alternatives, more granularity in the results would become apparent, and it would become clearer which scenarios were the most and least disruptive. For example, ^{Collier et al. (2018)} used the τ statistic on six project management scenarios affecting 16 activities and found good granularity (no ties) in the results.

The goal of this analysis is not necessarily to identify a single best alternative, but rather to guide subsequent modeling and data collection activities. This might include application of more detailed technical and business case analysis on a promising subset of alternatives, and the modeling of particular scenarios of interest (^{Karvetski and Lambert, 2012}).

Scenarios, as they have been described and utilized here, are distinct from the concept of risk. Risk is a measure of expected loss or harm associated with the answers to the questions “what can go wrong?”, “how likely is it?”, and “what are the consequences?” (^{Kaplan and Garrick, 1981}). The scenarios used in this paper represent situations in which stakeholders do not possess enough fundamental knowledge about the future environment to assign probabilities to various events. Thus, scenarios do not represent events in a probability space and cannot be assigned a likelihood, and so should not be confused with risks (^{Karvetski and Lambert, 2012}).

Nevertheless, scenarios can be used to facilitate risk management decision making. For instance, ^{Collier and Lambert (2019)} used a similar scenario framework to identify the most and least disruptive scenarios to an e-commerce system integration project, and then compared the risk-reduction benefits to the costs of implementing various mitigation plans. Understanding the potential negative impacts of scenarios to a system or product can strategically guide management’s allocation of resources and selection of risk responses.

Conclusions

Obsolescence can be caused by many technological, economic, organizational, and social factors over the life cycle of products and systems. As the service lives of certain systems extend into several decades, and the rapid pace of technological change shortens the life cycles of the components within them, obsolescence will continue to be a problem. While novel technologies such as 3D printing can provide a means for mitigation of obsolescence and result in less dependence on suppliers of parts, challenges still remain in adoption and implementation (^{Bromberger and Kelly, 2017}). In addition, there is still the need to be able to proactively anticipate when parts might fail and to optimize levels spare component inventory.

In this paper, we demonstrated a scenario-based approach related to obsolescence management. The approach provides a platform for identifying the most and least disruptive scenarios to a baseline ranking of alternatives, as well as provides insights into the performance of alternatives across scenarios. It can be used as a screening-level tool to eliminate poorly performing alternatives from consideration, and focus more detailed analyses on a subset of promising alternatives.

While this example focused on the selection of particular energy systems subject to obsolescence scenarios, the analysis could easily be modified to investigate the impacts of scenarios on particular components or subsystems within a larger system. For instance, a microgrid consists of various subsystems, including energy generation, storage, distribution, and control systems which may be impacted by various obsolescence scenarios differently. By decomposing a product or system into its subsystems, insights can be gained which could inform operations, inventory management, and risk management decisions.

References

[1] Abdi H (2007). The Kendall rank correlation coefficient. In: Salkind N, ed. Encyclopedia of Measurement and Statistics. Thousand Oaks: Sage

[2] Belton V, Stewart T J (2002). Multiple Criteria Decision Analysis: An Integrated Approach. Boston: Kluwer Academic Publishers

[3] Bromberger J, Kelly R (2017). Additive manufacturing: a long-term game changer for manufacturers. New York: McKinsey & Company

[4] Collier Z A, Hendrickson D, Polmateer T L, Lambert J H (2018). Scenario analysis and PERT/CPM applied to strategic investment at an automated container port. ASCE-ASME Journal of Risk and Uncertainty in Engineering Systems, Part A: Civil Engineering, 4(3): 04018026

[5] Collier Z A, Lambert J H (2018). Time management of infrastructure recovery schedules by anticipation and valuation of disruptions. ASCE-ASME Journal of Risk and Uncertainty in Engineering Systems, Part A: Civil Engineering, 4(2): 04018012

[6] Collier Z A, Lambert J H (2019). Evaluating management actions to mitigate disruptive scenario impacts in an e-commerce system integration project. IEEE Systems Journal, 13(1): 593–602

[7] Collier Z A, Walters S, DiMase D, Keisler J M, Linkov I (2014). A semi-quantitative risk assessment standard for counterfeit electronics detection. SAE International Journal of Aerospace, 7(1): 171–181

[8] Connelly E B, Colosi L M, Clarens A F, Lambert J H (2015). Risk analysis of biofuels industry for aviation with scenario-based expert elicitation. Systems Engineering, 18(2): 178–191

[9] Davis J, Sullivan J (2017). Supply chain risk—what is it? Defense AT&L, 46(2): 15–18

[10] de Grip A (2006). Evaluating human capital obsolesce. In: Proceedings of the Joint EC-OECD Seminar on Human Capital and Labour Market Performance, Brussels, Belgium

[11] de Siqueira Santos S, Takahashi D Y, Nakata A, Fujita A (2014). A comparative study of statistical methods used to identify dependencies between gene expression signals. Briefings in Bioinformatics, 15(6): 906–918

[12] DiMase D, Collier Z A, Carlson J, Gray R B Jr, Linkov I (2016). Traceability and risk analysis strategies for addressing counterfeit electronics in supply chains for complex systems. Risk Analysis, 36(10): 1834–1843

[13] DoD (2015a). Defense Acquisition University Glossary. 16th ed. Fort Belvoir: Defense Acquisition University Press

[14] DoD (2015b). Diminishing Manufacturing Sources and Material Shortages: Cost Metrics. Fort Belvoir: Defense Standardization Program Office

[15] DoD (2016). SD-22—Diminishing Manufacturing Sources and Material Shortages (DMSMS): a Guidebook of Best Practices for Implementing a Robust DMSMS Management Program. Fort Belvoir: Defense Standardization Program Office

[16] Hamilton M C, Lambert J H, Connelly E B, Barker K (2016). Resilience analytics with disruption of preferences and lifecycle cost analysis for energy microgrids. Reliability Engineering & System Safety, 150: 11–21

[17] Hamilton M C, Lambert J H, Valverde L J (2015). Climate and related uncertainties influencing research and development priorities. ASCE-ASME Journal of Risk and Uncertainty in Engineering Systems, Part A: Civil Engineering, 1(2): 04015005

[18] Hirsch A, Parag Y, Guerrero J (2018). Microgrids: a review of technologies, key drivers, and outstanding issues. Renewable & Sustainable Energy Reviews, 90: 402–411

[19] Kaplan S, Garrick B J (1981). On the quantitative definition of risk. Risk Analysis, 1(1): 11–27

[20] Karvetski C W, Lambert J H (2012). Evaluating deep uncertainties in strategic priority-setting with an application to facility energy investments. Systems Engineering, 15(4): 483–493

[21] Karvetski C W, Lambert J H, Keisler J M, Linkov I (2011a). Integration of decision analysis and scenario planning for coastal engineering and climate change. IEEE Transactions on Systems, Man, and Cybernetics, Part A: Systems and Humans, 41(1): 63–73

[22] Karvetski C W, Lambert J H, Linkov I (2009). Emergent conditions and multiple criteria analysis in infrastructure prioritization for developing countries. Journal of Multi-Criteria Decision Analysis, 16(5‒6): 125–137

[23] Karvetski C W, Lambert J H, Linkov I (2011b). Scenario and multiple criteria decision analysis for energy and environmental security of military and industrial installations. Integrated Environmental Assessment and Management, 7(2): 228–236

[24] Kendall M (1938). A new measure of rank correlation. Biometrika, 30(1‒2): 81–93

[25] Lambert J H, Karvetski C W, Spencer D K, Sotirin B J, Liberi D M, Zaghloul H H, Koogler J B, Hunter S L, Goran W D, Ditmer R D, Linkov I (2012). Prioritizing infrastructure investments in Afghanistan with multiagency stakeholders and deep uncertainty of emergent conditions. Journal of Infrastructure Systems, 18(2): 155–166

[26] Lambert J H, Wu Y J, You H, Clarens A, Smith B (2013). Climate change influence on priority setting for transportation infrastructure assets. Journal of Infrastructure Systems, 19(1): 36–46

[27] Morgan M G (2014). Use (and abuse) of expert elicitation in support of decision making for public policy. Proceedings of the National Academy of Sciences of the United States of America, 111(20): 7176–7184

[28] Porter G Z (1998). An Economic Method for Evaluating Electronic Component Obsolescence Solutions. Boeing Company White Paper

[29] Reilly R F (2013). Consideration of functional and economic obsolescence in the assessment of industrial or commercial property. Journal of Property Tax Assessment & Administration, 10(1): 45–58

[30] Romero Rojo F J, Roy R, Kelly S (2012). Obsolescence risk assessment process best practice. Journal of Physics: Conference Series, 364: 012095

[31] Romero F J Rojo R, Roy E, Shehab (2010). Obsolescence management for long-life contracts: state of the art and future trends. International Journal of Advanced Manufacturing Technology, 49(9‒12): 1235–1250

[32] Sandborn P (2007). Software obsolescence—complicating the part and technology obsolescence management problem. IEEE Transactions on Components and Packaging Technologies, 30(4): 886–888

[33] Sandborn P (2013). Design for obsolescence risk management. Procedia CIRP, 11: 15–22

[34] Sandborn P, Prabhakar V, Ahmad O (2011). Forecasting electronic part procurement lifetimes to enable the management of DMSMS obsolescence. Microelectronics and Reliability, 51(2): 392–399

[35] Singh P, Sandborn P (2006). Obsolescence driven design refresh planning for sustainment-dominated systems. Engineering Economist, 51(2): 115–139

[36] Solomon R, Sandborn P A, Pecht M G (2000). Electronic part life cycle concepts and obsolescence forecasting. IEEE Transactions on Components and Packaging Technologies, 23(4): 707–717

[37] Song J S, Zipkin P H (1996). Managing inventory with the prospect of obsolescence. Operations Research, 44(1): 215–222

[38] Wnuk K, Gorschek T, Zahda S (2013). Obsolete software requirements. Information and Software Technology, 55(6): 921–940

RIGHTS & PERMISSIONS

Higher Education Press