An overview of the reliability metrics for power grids and telecommunication networks

Abstract

Power grids deliver energy, and telecommunication networks transmit information. These two facilities are critical to human society. In this study, we conduct a comprehensive overview of the development of reliability metrics for power grids and telecommunication networks. The main purpose of this review is to promote and support the formulation of communication network reliability metrics with reference to the development of power grid reliability. We classify the metrics of power grid into the reliability of power distribution and generation/transmission and the metrics of telecommunication network into connectivity-based, performance-based, and state-based metrics. Then, we exhibit and discuss the difference between the situations of the reliability metrics of the two systems. To conclude this study, we conceive a few topics for future research and development for telecommunication network reliability metrics.

Key words： reliability; metrics; power grids; telecommunication networks

1 Introduction

Critical infrastructures, e.g., power grids and telecommunication networks, are important driving forces of the sustainability of society and economic development. Generally, these facilities need to satisfy certain basic demands of humans, e.g., power grids deliver electrical energy, while telecommunication networks transmit information. With the continuous progress of human society, people’s demand for such services continues to rise. As a result, electricity and communication services have penetrated into almost every aspect of daily life in most parts of the world. Moreover, these services are playing an increasingly important role in our society.

Given the significance of energy and information, these two networks undoubtedly serve as lifelines to humans. The failure of power grids or telecommunication networks can incur tremendous losses. Unfortunately, these essential networks are constantly facing inevitable threats now-adays, including natural disasters, e.g., earthquakes, tsunamis, and volcanic eruptions, and manmade failures, e.g., malicious cyberattacks, terrorist attacks, and human-caused accidents. For instance, in the 2008 ice disaster in Southern China, ice on overhead transmission lines and facilities caused outages in some cities and counties with a total population of approximately 30 million people affected (Lu et al., 2015). In the same year, the Wenchuan Earthquake hit Southwest China. Eight counties suffered a complete outage of communication connection for a certain period, with 28765 kilometers of optic cables damaged and 142078 telecommunication poles fallen (Ran, 2011). In 2013, two transmission lines in the Arkansas Power Grid were cut, and one was pulled down by an individual, thus causing outage for 10000 customers (William, 2003; Zhu et al., 2014). In 2015, Black Energy malware intruded the Ukrainian power grid, thus leaving 700000 households with a power outage (Lee et al., 2016).

The examples above imply the necessity to ensure the reliable operations of power and telecommunication networks (systems). To implement quantitative management for the prevention and mitigation of potential losses or damages, the measurement of reliability is the first and the key issue. Without the metric, reliability cannot be quantified. Thus, quantitative management cannot be realized. With a clearly defined metric for the network reliability, the probability of malfunction can be quantified, comprehended, and communicated. Moreover, the system has a target for its operation and maintenance.

The research works for the reliability metrics of both networks were initiated several decades ago. For power grids, academic research has started earlier, and the utilizations of the metrics are commonly accepted in practice. The system of reliability metrics and the standards for the metrics have been established. In contrast, to the best of the authors’ knowledge, the industry has yet to establish a commonly accepted telecommunication network reliability metric, let alone standards for reliability metrics. Given the importance of telecommunication networks in our society and the great meaning of its reliability metrics, this work aims to provide an overview and a comparison of the developments of the reliability metrics in both fields. Then, it points out the directions for future research efforts for telecommunication network reliability metrics.

The remainder of this study is organized as follows. Section 2 summarizes two types of power grid reliability metrics and provides an overview of their development history. Section 3 summarizes four types of telecommunication network reliability metrics and briefly discusses their development history. Section 4 presents the com-parisons between the situations of power grids and telecommunication networks. Section 5 concludes this work and points out future research directions.

2 Reliability metrics for power grids

The reliability metrics of power girds can be divided into two groups: Distribution reliability and adequacy of generation/transmission systems. In general, a complete power grid is divided into three subsystems: Generation, transmission, and distribution. The former group of metrics is focused on a distribution system. Given that the distribution system is directly connected with power grid users, the distribution reliability was studied first. The latter group of metrics is focused on generation and transmission reliability. They appeared after the distribution reliability metrics, but they are also crucial for the power grids. Detailed information about the developments of the two groups are presented in the following subsections.

2.1 Distribution reliability

In the early 20th century, the reliability of power grids was not taken into account. Only when a failure occurred or certain users lost power would operators be called for maintenance. Then, the staff would manually assign maintenance personnel to repair the problem. However, if multiple failures occurred in a short period, this manual assignment became ineffective. After the US northeastern blackout in 1965, power grid reliability assessments started attracting research attention. Capra et al. (1969) studied the underground distribution systems and discussed the ability of the system from the perspective of distribution. They proposed a metric called the interruption time to average customers, which is perhaps the first published metric for power grid reliability.

In the early 1970s, the power industry began to focus on power grid reliability as it promoted the development and implementation of reliability metrics (Brown, 2007). The most commonly used ones included the System Average Interruption Frequency Index (SAIFI), System Average Interruption Duration Index (SAIDI), Customer Average Interruption Frequency Index (CAIFI), Customer Average Interruption Duration Index (CAIDI), and Average Service Availability/Unavailability Index (ASAI/ASUI).

In the 1980s, more researchers began to study distribution system reliability. Two seminal books were written by Billinton and Allan (1984; 1988). In these two works, various reliability metrics, such as SAIFI, SAIDI, CAIFI, CAIDI, and ASAI, were formally defined, as shown below.

(1)SAIFI={\displaystyle \frac{CI}{N_{\mathrm{T}}}}\mathrm{}\left(\mathrm{interruption/customer}\right),

(2)SAIDI={\displaystyle \frac{CMI}{N_{\mathrm{T}}}}\mathrm{}\left(\mathrm{hour/customer}\right),

(3)CAIFI={\displaystyle \frac{CI}{CN}}\left(\mathrm{interruption/customer\; affected}\right),

(4)CAIDI={\displaystyle \frac{SAIDI}{SAIFI}}\left(\mathrm{hour/interruption}\right),

(5)ASAI={\displaystyle \frac{\mathrm{8760}{N}_{\mathrm{T}}-CMI}{\mathrm{8760}{N}_{\mathrm{T}}}},

where CI is the total number of customers interrupted, N_{\mathrm{T}} is the total number of customers served, CMI represents the total time of customers’ interruption, and CN represents the total number of distinct customers who have experienced a sustained interruption during the reporting period. In ASAI, customer hours demanded are determined as the 12-month average number of customers served times 8760.

In addition, Warren (1996) summarized other distribution reliability metrics, such as the Average System Interruption Frequency Index (ASIFI), Average System Interruption Duration Index (ASIDI), Customer Total Average Interruption Duration Index (CTAIDI), Customers Experiencing Multiple Interruptions (CEMI_n), and Momen-tary Average Interruption Frequency Index (MAIFI), which are expressed as follows.

(6)ASIFI={\displaystyle \frac{{\displaystyle {\sum }_e{L}_e}}{L_{\mathrm{T}}}},

(7)ASIDI={\displaystyle \frac{{\displaystyle {\sum }_e{r}_e{L}_e}}{L_{\mathrm{T}}}},

(8)CTAIDI={\displaystyle \frac{CMI}{CN}}\mathrm{}\left(\mathrm{hour/customer\; affected}\right),

(9)CEM{I}_n={\displaystyle \frac{C{N}_{\left(k\ge n\right)}}{N_{\mathrm{T}}}},

(10)MAIFI={\displaystyle \frac{TMI}{N_{\mathrm{T}}}},

where L_e is the connected kVA load interrupted due to each interruption event e, L_{\mathrm{T}} is the total connected kVA load served, r_e is the restoration time for each interruption event, C{N}_{\left(k\ge n\right)} is the total number of customers that experienced n or more sustained interruptions, and TMI is the total number of customer momentary interruptions.

Later on, Brooks et al. (1998) observed that numerous electricity consumers are adversely affected by more subtle voltage disturbances, such as sags and swells. They proposed four metrics to assess root-mean-square (rms) variation magnitude and the combination of magnitude and duration: System Average RMS Variation Frequency Index (SARFI_x), System Instantaneous Average RMS Variation Frequency Index (SIARFI_x), System Momentary Average RMS Variation Frequency Index (SMARFI_x), and System Temporary Average RMS Variation Frequency Index (STARFI_x).

(11)SARF{I}_x={\displaystyle \frac{S_x}{N_{\mathrm{T}}}},

(12)SIARF{I}_x={\displaystyle \frac{S_{x\left(\mathrm{0.5}\mathrm{c}-\mathrm{30}\mathrm{c}\right)}}{N_{\mathrm{T}}}},

(13)SMARF{I}_x={\displaystyle \frac{S_{x\left(\mathrm{30}\mathrm{c}-\mathrm{3}\mathrm{s}\right)}}{N_{\mathrm{T}}}},

(14)STARF{I}_x={\displaystyle \frac{S_{x\left(\mathrm{3}\mathrm{s}-\mathrm{60}\mathrm{s}\right)}}{N_{\mathrm{T}}}},

where x is rms voltage threshold, S_x represents the total number of customers experiencing short-duration voltage deviations, S_{x\left(\mathrm{0.5}\mathrm{c}-\mathrm{30}\mathrm{c}\right)} represents the total number of customers experiencing instantaneous voltage deviations within a duration of 0.5–30 cycles, S_{x\left(\mathrm{30}\mathrm{c}-\mathrm{3}\mathrm{s}\right)} denotes the number of customers experiencing momentary voltage deviations within a duration of 30 cycles to 3 seconds, and S_{x\left(\mathrm{3}\mathrm{s}-\mathrm{60}\mathrm{s}\right)} represents the total number of customers experiencing temporary voltage deviations within a duration of 3–60 seconds.

In a summary work, Brown (2007) divided the above metrics into three categories, namely, customer-based, load-based, and power quality metrics. The first category contained SAIFI, SAIDI, CAIFI, CAIDI, ASAI, CTAIDI, CEMI_n, and MAIFI. The second category contained ASIFI and ASIDI. The third category contained SARFI_x, SIARFI_x, SMARFI_x, and STARFI_x.

Along with the continuous development of reliability metrics in academia, power companies and professional societies are on the road for the standardization of the metrics, in close collaboration with academic researchers. In the late 1980s, the power engineering society of IEEE established a working group under the Distribution Subcommittee on Performance Records for Optimizing System Design. The group was later renamed the Working Group on Distribution Reliability (WGDR). In 1998, WGDR published a trial user guide on distribution reliability indices (IEEE, 1999), which can be seen as the predecessor to the standard IEEE Guide for Electric Power Distribution Reliability Indices (IEEE, 2001). IEEE continues to update this standard. The latest version was released in 2012 (IEEE, 2012). It contains numerous metrics, including SAIFI, SAIDI, CAIFI, CAIDI, ASAI, and CTAIDI. At present, the standard is used by a large number of grid operators in several countries.

To clarify the descriptions of the above metrics, we use the examples described by Billinton and Allan (1996) to explain the application of some metrics briefly. The metrics can be illustrated by considering a portion of a distribution system having six load-point busbars. This bare bones system shows only the data necessary to illustrate metric calculation. The number of customers and average load connected to these busbars are shown in Table 1, which serves 4000 customers for a total load of 8 MW. Assuming that four system failures occur in one given calendar year of interest, the interruption effects are shown in Table 2.

Tab.1 Details of the distribution system

Load point	Number of customers	Average load connected
1	1000	5000
2	800	3600
3	600	2800
4	800	3400
5	500	2400
6	300	1800
Total	4000	19000

Tab.2 Interruption effects in a given calendar year

Interruption case	Probability	Load point affected	Number of customers disconnected	Load curtailed (kW)	Duration of interruption (hour)	Customer hours curtailed	Energy not supplied (kWh)
1	0.3	2	800	3600	3	2400	10800
		3	600	1800	3	1800	8400
2	0.4	6	300	1800	2	600	3600
3	0.2	3	600	2800	1	600	2800
4	0.3	5	500	2400	1.5	750	3600
		6	300	1800	1.5	450	2700
Total			3100	14200	12	6600	31900

Based on the above data, some of the above metrics can be calculated directly as shown below:

SAIFI={\displaystyle \frac{CI}{N_{\mathrm{T}}}}={\displaystyle \frac{\mathrm{3100}}{\mathrm{4000}}}=\mathrm{0.775}\mathrm{}\left(\mathrm{interruption/customer}\right),

CAIFI={\displaystyle \frac{CI}{CN}}={\displaystyle \frac{\mathrm{3100}}{\mathrm{2200}}}=\mathrm{1.409}\mathrm{}\left(\mathrm{interruption/customer\; affected}\right),

\mathrm{where}CN=\mathrm{800}+\mathrm{600}+\mathrm{300}+\mathrm{500}=\mathrm{2200},

SAIDI={\displaystyle \frac{CMI}{N_{\mathrm{T}}}}={\displaystyle \frac{\mathrm{6600}}{\mathrm{4000}}}=\mathrm{1.650}\mathrm{}\left(\mathrm{hour/customer}\right),

CAIDI={\displaystyle \frac{SAIDI}{SAIFI}}=\mathrm{2.129}\mathrm{}\left(\mathrm{hour/interruption}\right),

ASAI={\displaystyle \frac{\mathrm{8760}{N}_{\mathrm{T}}-CMI}{\mathrm{8760}{N}_{\mathrm{T}}}}=\mathrm{0.999812},

CTAIDI={\displaystyle \frac{CMI}{CN}}=\mathrm{3.000}\mathrm{}\left(\mathrm{hour/customer\; affected}\right),

ASIFI={\displaystyle \frac{{\displaystyle {\sum }_e{L}_e}}{L_{\mathrm{T}}}}={\displaystyle \frac{\mathrm{14200}}{\mathrm{8000}}}=\mathrm{1.775},

ASIDI={\displaystyle \frac{{\displaystyle {\sum }_e{r}_e{L}_e}}{L_{\mathrm{T}}}}={\displaystyle \frac{\mathrm{31900}}{\mathrm{8000}}}=\mathrm{3.988.}

These simple numerical examples are used to illustrate the application of reliability metrics. For more detailed introduction and application, readers are encouraged to refer to Billinton and Allan (1996) and IEEE (2012).

In summary, the development of distribution reliability metrics can be divided into three stages. The first stage is the pre-1980s, where the reliability metrics appeared in industry and large power companies that began to consider reliability. The second stage is the academia promotion starting from the 1980s. Since then, an increasing number of researchers have studied power grid reliability and defined reliability metrics. The third stage is the standardization beginning in the 2000s. Professional groups, such as WGDR, started releasing standards, which are constantly updated.

With the development of power electronics and the popularity of renewable energy, power grids are developing from central generation to distributed generation and from direct current (DC) to alternating current (AC). Moreover, microgrids have been gradually developed. At the same time, the above distribution network metrics are still applicable. Chittum and Relf (2019) used the SAIFI, SAIDI, and other metrics in distributed energy systems, such as combined heat and power (CHP). Meanwhile, Zhao et al. (2019) used SAIDI and ASAI in an AC/DC distribution network and established the reliability model of a converter station, a DC transformer, and a DC circuit breaker. Alahmed et al. (2019) used the SAIFI, SAIDI, and CAIDI to evaluate microgrids, which are distribution networks that include different distributed generators (DGs) and energy storage systems (ESSs) and have the ability to work in off-grid mode.

2.2 Adequacy of generation/transmission system

The distribution reliability described above mainly evaluates the reliability of the power grid from the perspective of customers, such as the number of interruptions and the duration of interruptions. However, it is not suitable for assessing whether the supply of the overall grid is adequate. At the same time in the field of power grid, the adequacy of the power grid is also a very important reliability. The term “adequacy” is used to describe the ability of the power system to fulfill the customer demands. In theory, it can cover the entire power grid, but power engineers intend to use it with a focus on generation and transmission systems. Billinton and Allan (1984; 1988) first defined the adequacy metrics, including Energy Not Supplied (ENS), Expected Energy Not Supplied (EENS), Average Energy Not Supplied (AENS), and Loss of Load Expectation (LOLE).

(15)ENS={\displaystyle {\sum }_j{L}_{a\left(j\right)}{U}_j}\mathrm{}\left(\mathrm{kWh}\right),

(16)AENS={\displaystyle \frac{ENS}{N_{\mathrm{T}}}}\mathrm{}\left(\mathrm{kWh/customer}\right),

(17)EENS={\displaystyle {\sum }_{k}EN{S}_k{p}_k}\mathrm{}\left(\mathrm{kWh}\right),

(18)

where L_{a\left(j\right)} is the average load connected to load point j, U_j is the annual outage time, p_k is the probability of system state k, and is the probability of loss of load on day i or during hour i, in which C is the capacity and L_i represents the load on day i or during hour i.

On the basis of ENS, other similar metrics were developed later on, such as the Average Customer Curtailment Index (ACCI) (Billinton and Allan, 1996) and the Cost of Energy Not Supplied (CENS) (Gooi et al., 1999).

(19)ACCI={\displaystyle \frac{ENS}{N_{\mathrm{a}}}}\mathrm{}\left(\mathrm{kWh/customer\; affected}\right),

(20)CENS=EENS\times v\mathrm{}(\$),

where N_{\mathrm{a}} denotes total number of customers affected (namely, CN), and v represents the value of lost load.

On the basis of LOLE, few loss-of-load metrics were developed, such as Loss of Load Frequency (LOLF) (Capra et al., 1969; Melo et al., 1993; Kim and Singh, 1993; Samaan and Singh, 2002), Loss of Load Duration (LOLD) (Dobakhshari and Fotuhi-Firuzabad, 2009; Lin et al., 2014), and Loss of Load Hours (LOLH) (Rudkevich et al., 2012). The formal definitions are presented as

(21)LOLF={\displaystyle {\sum }_{j}LOL{F}_j}\mathrm{}\left(\mathrm{occurrence/year}\right),

(22)LOLD={\displaystyle \frac{LOLE}{LOLF}}\mathrm{}\left(\mathrm{hour/occurrence}\right),

(23)LOLH=\mathrm{Hours\; of\; shortage\; per\; year}\left(\mathrm{hour/year}\right),

where LOL{F}_j represents the contribution of load point j to the failure frequency of the system.

In addition to the above metrics, other adequacy indicators have been developed (Billinton and Li, 1994), including the Probability of Load Curtailments (PLC), Expected Load Curtailments (ELC), and Bulk Power Interruption Index (BPII).

(24)PLC={\displaystyle {\sum }_{k\in S^l}{p}_k},

(25)ELC={\displaystyle {\sum }_{k\in S^l}{C}_k{F}_k}\mathrm{}\left(\mathrm{MW/year}\right),

(26)BPII={\displaystyle \frac{ELC}{L}},

where S^l is the set of all system states associated with load curtailment, C_k is the load curtailment under system state k, F_k is the frequency of system state k, and L is the system annual peak load.

The capacity margin (CM) is another important adequacy metric that has been widely used by the power industry and regulatory commissions. It was first defined by McCalley et al. (1991) as the difference between a set of specific system conditions and the actual system constraints. Thus, it can describe the adequacy of power grids as well. This definition is qualitative, and has been used throughout the power industry mainly because the CM of power grids is high. Hence, practitioners have little motivation to quantify it. Later on, the quantitative expression of CM has appeared in the literature (European Commission, 2016), which is shown as follows.

CM=\mathrm{Available\; generation\; capacity}

(27)-\mathrm{Peak\; demand}\left(\mathrm{MW}\right).

To explain the above metrics better, we use the data in Tables 1 and 2 to describe the application of some metrics briefly.

ENS={\displaystyle {\sum }_j{L}_{a\left(j\right)}{U}_j}=\mathrm{31900}\mathrm{}\left(\mathrm{kWh}\right),

EENS={\displaystyle {\sum }_{k}EN{S}_k{p}_k}=\mathrm{9650}\mathrm{}\left(\mathrm{kWh}\right),

AENS={\displaystyle \frac{ENS}{N_{\mathrm{T}}}}=\mathrm{7.975}\mathrm{}\left(\mathrm{kWh/customer}\right),

ACCI={\displaystyle \frac{ENS}{N_{\mathrm{a}}}}=\mathrm{14.500}\mathrm{}\left(\mathrm{kWh/customer\; affected}\right).

Another 100 MW system can be used to calculate LOLE and other metrics. The load data for a period of 365 days is shown in Table 3.

Tab.3 Load data

Daily peak load (MW)	57	52	46	41	34
No. of occurrences	12	83	107	116	47

LOLE=\mathrm{12}\mathrm{Pr}\left\{\mathrm{100}-\mathrm{57}\right\}+\mathrm{83}\mathrm{Pr}\left\{\mathrm{100}-\mathrm{52}\right\}+\mathrm{107}\mathrm{Pr}\left\{\mathrm{100}-\mathrm{46}\right\}+\mathrm{116}\mathrm{Pr}\left\{\mathrm{100}-\mathrm{41}\right\}+\mathrm{47}\mathrm{Pr}\left\{\mathrm{100}-\mathrm{34}\right\}=\mathrm{2.151}\mathrm{}\left(\mathrm{day/year}\right),

CM=\mathrm{100}-\mathrm{57}=\mathrm{43}\mathrm{}\left(\mathrm{MW}\right).

These simple numerical examples are used to illustrate the application of reliability metrics. For more detailed introduction and application, readers are encouraged to refer to Billinton and Li (1994) and Billinton and Allan (1996).

Among the metrics above, researchers frequently use EENS and LOLE. The following are some recent cases. Wu et al. (2009) used the two-state model to model the wind turbine generators (WTGs) system and used EENS and LOLE to assess the WTG. Li and Zio (2012) considered distribution networks with renewable generators, and computed LOLE and EENS from the DG system. Dehghanian et al. (2013a; 2013b) presented a reliability-centered maintenance framework to power distribution systems, and used LOLE and EENS as the system metrics to optimize maintenance. Božič and Pantoš (2015) assessed the impact of electric-drive vehicles on power system reliability, where the reliability metrics EENS and LOLE are incorporated in objective function to optimize.

Prior to the 1990s, adequacy metrics did not receive much attention from the power industry, which might because they were not the direct concerns of the customers compared with distribution reliability. In 1990s, the industry started to attract the concerns from the regulatory aspect, as the reliability of the entire power grid became a relevant issue. In 1993, the North American Electric Reliability Corporation (NERC) introduced CM for reliability assessment. Since then, it has been releasing reliability assessment reports every year. As a supplement of CM, NERC added the reserve margin (RM) after 2000 (Altiparmak et al., 2009). RM is based on the limit of how close the load should be allowed to available capacity, which can be expressed as:

(28)RM={\displaystyle \frac{A_{\mathrm{c}}}{M_{\mathrm{L}}}}-\mathrm{1},

where A_{\mathrm{c}} represents the available capacity and M_{\mathrm{L}} represents the maximum annual load.

Since then, CM and RM have been the main metrics for evaluating power grid reliability. Although RM can measure the capability of the system to provide services on the basis of the existing resources and planned resources, it does not reflect the probabilistic characteristics of reliability assessment. Since 2012, NERC has enhanced its reliability assessment process by supplementing its resource adequacy metrics with probabilistic metrics, i.e., LOLH and Expected Unserved Energy (EUE), where EUE has the same definition as EENS.

In Europe, different reliability/adequacy metrics were initially used in different countries. Given the creation of the single electricity market for the entire European Union (Commission) (EU), the EU needs unified reliability/adequacy metrics for the management of the single market. In 2016, the European Commission released a final report named Identification of Appropriate Generation and System Adequacy Standards for the Internal Electricity Market. One of the recommendations of this report was to “establish EENS as a preferred metric, as it alone proves appropriate for the calculation of the socially optimal levels of reserve”.

Similarly, with the continuous development of power grids, these metrics are still used in the current smart grid. Adefarati and Bansal (2019) focused on the evaluation of the reliability benefits of renewable energy resources and used EENS and LOLE in a microgrid system that consists of the photovoltaic, WTG, ESS, and diesel generator. Li and Zio (2012) and Hariri et al. (2020) took EENS as a metric of smart grids including renewable and non-renewable distributed generations. Chamandoust et al. (2020) used LOLE to evaluate grids with renewable energy sources (RESs) and electrical energy storage. Meanwhile, Memari et al. (2021) also used EENS as a metric of renewable distributed generation grids.

3 Reliability metrics for telecommunication networks

Modern telecommunication networks appeared in the late 18th century, when the telegraph and the telephone were invented and popularized. However, the requirement of high quality of service (QoS) was not generally considered by majority of the customers. Thus, reliability research has not attracted much attention. Starting from the 1960s, several telecommunication network operators began to rely on the stable and non-stop service to fulfill the customers’ ever-rising demands for high-quality communication services. Telecommunication network reliability has then become an important research topic as well as a practical concern.

Concerning the traditional definition of network reliability, the publications in this field can be mainly classified into three main groups. The first group is connectivity-based metrics, which are focused on the connectivity between network nodes. The second group includes performance-based metrics, which describe the reliability of the network according to one character of the network and the preset threshold. The last group consists of state-based metrics, which are related to a certain state metric of the network. The subgroups and the relevant publications within each group are presented in Table 4.

Tab.4 Different groups of network reliability metrics

Main groups	Subgroups	Renferences
Connectivity-based metrics	2-terminal connectivity K-terminal connectivity All-terminal connectivity	Jereb (1998); Wilkov (1972); Hwang et al. (1981); Locks (1985); Cook and Ramirez-Marquez (2007); Migov and Shakhov (2014); Xiang and Yang (2020)
Performance-based metrics	Demand reliability	Aggarwal (1985); Rushdi (1988); Bienstock and Günlük (1995); Lee (1980); Aggarwal et al. (1982a; 1982b); Ramirez-Marquez and Coit (2005); Zuo et al. (2007)
	Time reliability	Park and Tanaka (1979); Chiou and Li (1986); Levitin (2003); Wu et al. (2015); Li et al. (2017); Shi et al. (2012)
	SINR reliability	Capra et al. (1969); Miyoshi and Shirai (2014); Pocovi et al. (2015); Zhong et al. (2017); Xiang and Yang (2020)
State-based metrics	Effectiveness reliability	Trstensky and Bowron (1984); Fan and Sun (2010)
	Expected lost traffic	Sanso et al. (1991); Carlier et al. (1997)

3.1 Connectivity-based metrics

In the 1970s and early 1980s, the reliability of the telecommunication network was considered the S-T (source-terminal) connectivity probability of the network (Jereb, 1998). The most common measure was K-terminal connectivity probability (Hwang et al., 1981; Locks, 1985; Wilkov, 1972), which indicated the probability that certain K vertices in all vertices were connected by working edges. For K-terminal connectivity probability, two important special cases were those of 1) 2-terminal (S-T) connec-tivity reliability, in which K = 2 and one of the nodes was designated as a source while the other was a sink node, and 2) overall reliability (all-terminal reliability).

On the basis of the classical definition of K-terminal connectivity probability, some scholars have extended the research works to solve practical problems. Cook and Ramirez-Marquez (2007) studied the reliability of a mobile ad hoc wireless network. The definition of K-terminal reliability is generalized to the probability that k nodes are connected to each other in the network. Migov and Shakhov (2014) presented a new network reliability measure of ad hoc networks with imperfect nodes and perfectly reliable links. The introduced reliability index is the probability that sink nodes are connected with each other and k nodes are at least workable and connected to any sink node. According to the generalized K-terminal reliability, Xiang and Yang (2020) analyzed the reliability of ad hoc networks incorporating the impacts of node failures and interference.

3.2 Performance-based metrics

In the late 1980s, given the rapid development of communication technology and network scale, network load continued to increase. Moreover, network delay and other problems continued to occur. Hence, network performance reliability became an important direction in network reliability research.

From the perspective of transmission, the telecommunication network can be characterized by the flow network. The flow can be seen as information transported in the network, and the maximum possible flow between the source point and the end point is used as the performance index (Rushdi, 1988). On the basis of single-flow network, some researchers used multi-commodity flow networks to describe telecommunication networks to make the problem more practical (Bienstock and Günlük, 1995).

Traditionally, each of the performance indices of connectivity and capacity is used independently; i.e., whenever one is considered, the other is disregarded (Aggarwal, 1985; Rushdi, 1988). In the telecommunication network, only the connectivity of the topological structure to analyze the reliability of the network has drawbacks. For example, the failure of some important components (servers and core switches) in the network will greatly affect the performance of data flow, but the network still has good connectivity. Network reliability should not only be measured by connectivity but also be combined with network flow or other performance.

Lee (1980) and Aggarwal et al. (1982a; 1982b) attempted to integrate two performance indices and considered network reliability as the probability of successfully transmitting the required amount (d) of information from the source node to the terminal node, i.e., service reliability. The network reliability can be described as follows:

(29)R_{\mathrm{s}}=\mathrm{Pr}\left\{\varphi \left(x\right)\ge d\right\},

where x represents the state of the network, and \varphi \left(x\right) represents the flow performance of the network when the network is in state x. Ramirez-Marquez and Coit (2005) and Zuo et al. (2007) studied the evaluation algorithm for such reliability metrics. This metric can be seen as throughput (demand) in a telecommunication network. We use the example network in Fig. 1 to illustrate the metric. Each edge in the network has a certain probability of failure, and the data that each edge can transmit is limited. In this case, the reliability represents the probability that at least d units of data can be transmitted from point s to point t.

Fig.1 Example network.

Full size|PPT slide

In addition to service reliability based on network flow, performance-based reliability also includes time reliability with time as the research object. Park and Tanaka (1979) and Chiou and Li (1986) considered the delay during transmission. In this research, the mean message delay of the network is defined as the performance measure to describe reliability, which can be expressed as follows:

(30)R_{\mathrm{s}}=\mathrm{Pr}\left\{\varphi \left(x\right)\le \tau \right\},

where \varphi \left(x\right) means the network mean message delay and \tau represents a time threshold. Similarly, we take the network in Fig. 1 as an example. At this time, each edge in the network has a certain probability of failure, and it takes a certain amount of time for data to pass through each edge. The reliability at this time represents the probability that the time needed from point s to point t is less than \tau.

Levitin (2003) applied this metric in acyclic transmission networks (ATNs) and suggested an algorithm based on universal generating function. Shi et al. (2012) divided the actual network traffic into two different parts—a-traffic with burst characteristic, and b-traffic with steady characteristic, and then selected the above network time delay as the metric, thus optimizing the metric threshold. Li et al. (2017) and Wu et al. (2015) also used this time reliability metric on the Avionics Full Duplex Switched Ethernet (AFDX). This metric can be seen as latency in telecommunication network. Combining the demand and time, Lin (2010) proposed another reliability metric:

(31)R_{\mathrm{s}}=\mathrm{Pr}\left\{\varphi \left(d,\mathrm{}x\right)\le \tau \right\},

where \varphi \left(d,\mathrm{}x\right) denotes the minimum time for sending d units of data under state x. Taking the network in Fig. 1 as an example, the reliability means the probability that the network can transmit d units of data from s to t within the time threshold \tau.

Another important performance factor of a telecommunication network is the Signal-to-Interference-plus-Noise Ratio (SINR). The signal quality outage performance is of key importance to satisfy stringent reliability requirements. This performance metric is expressed as follows:

(32)R_{\mathrm{s}}=\mathrm{Pr}\left\{SINR>T\right\},

where T is the threshold of SINR. This metric has been widely used in telecommunication networks (Capra et al., 1969; Miyoshi and Shirai, 2014; Pocovi et al., 2015; Zhong et al., 2017).

3.3 State-based metrics

In addition, several new metrics classify the states of a certain characteristic in the network. Trstensky and Bowron (1984) proposed a new index combining effectiveness and connectivity. The reliability of the telecommunication network is expressed as follows:

(33)R_{\mathrm{s}}={\displaystyle \sum \mathrm{Pr}\left\{x\right\}}\cdot \varphi \left(x\right),

where \mathrm{Pr}\left\{x\right\} denotes the probability of occurrence of state x and \varphi \left(x\right) represents the effectiveness of the system. In this research, effectiveness means the maximum number of channels in state x. Similarly, Fan and Sun (2010) used the number of nodes that a given node can communicate with as the effectiveness.

Similar to energy loss of power grid, i.e., EENS, the loss of traffic in the telecommunication network is also valued. Sanso et al. (1991) and Carlier et al. (1997) perceived reliability as the probability that the network is operating at any given time. The measure related to reliability is the expected lost traffic, which represents the demands that cannot be transmitted because of failures. In addition, they take into account routing and rerouting policies after failures. Their model consists of two layers. The first layer is called the physical network, which represents the physical link. The second layer is called the logical network, which represents the traffic demand of the source and the sink node. The metric can be expressed as follows:

(34)Expected\underline{\square }lost\underline{\square }traffic={\displaystyle \sum \mathrm{Pr}\left\{x\right\}}\cdot \varphi \left(x\right),

where \varphi \left(x\right) here means the lost traffic in state x.

These network reliability parameters make the measurement of network reliability more user-oriented and intuitive, thus greatly advancing the research work of network reliability.

3.4 Practical metrics related to reliability

However, the reliability index in practical application has a gap with that in academic circles. The first commercial 1G network was launched in Japan in 1979. It was an analog system and offered no data capabilities. In 1991, the first 2G network was launched in Finland based on the emerging GSM (Global System for Mobile Communications) standard. Before the 2000s, the telecommunication network was mainly limited to the wired network, and the function of the network was relatively single, mainly for calls or telegraph businesses. The related metrics are the Call Setup Success Rate (CSSR) and Call Drop Rate (CDR).

(35)CSSR={\displaystyle \frac{S_{\mathrm{c}}}{T_{\mathrm{c}}}},

(36)CDR={\displaystyle \frac{N_{\mathrm{d}}}{N_{\mathrm{s}}}},

where S_{\mathrm{c}} represents the number of successful call connections, T_{\mathrm{c}} represents the total number of call attempts, N_{\mathrm{d}} denotes the number of call drop times, and N_{\mathrm{s}} is the number of call setup success times. However, in the early days, data records and user complaints were used to infer these indicators.

After the 2000s, 3G and 4G networks developed rapidly, and telecommunication networks became increasingly important in daily life. Continuing the previous study, all-terminal reliability is still used in telecommunication networks (Prempraycon and Wardkein, 2004; Altiparmak et al., 2009). With the development of wireless technology, telecommunication networks have gradually become wireless and wired heterogeneous networks. In addition to the traditional metrics, the social requirements for the network have increased, and metrics, such as single connectivity, no longer meet the needs. The concept of QoS has been gradually applied to telecommunication network services (Tian et al., 2003).

According to previous studies (Capra et al., 1969; Tian et al., 2003; Charilas et al., 2008; Kostanic et al., 2009; Asokan, 2010; Lemeshko et al., 2015), several main QoS metrics exist: Delay, reliability, throughput, jitter, and loss probability. Reliability is only a low-level metric, and it is different from the overall network-level reliability. In some studies, reliability is represented by the bit error rate (Popovski, 2013).

Based on QoS, more specific key performance indicators (KPIs) have emerged, which are more widely used. In addition, many companies use KPI warnings in their actual operations, such as Huawei. Network KPIs are categorized into the following subcategories: Accessibility, retain-ability, mobility, integrity, and availability (3GPP, 2011; Lemeshko et al., 2015; Pocovi et al., 2015; Darlington, 2016; Krasniqi et al., 2018; 2019). In the METIS project, which was co-funded by the European Commission, Popovski (2013) divided KPIs into the following categories: Traffic volume density, experienced user throughput, latency, reliability, and energy consumption. KPIs are summarized in Table 5.

Tab.5 KPIs for LTE (Long Term Evolution) network

Category	Description	Example
Accessibility	A KPI enables the network operator to know whether the service required by a user can be accessed	RRC (radio resource control) setup success rate (signaling) E-RAB (evolved radio access bearer) setup success rate
Retainability	A KPI measures the capacity of the system to ensure the services without interruption	Call drop rate Service call drop rate
Mobility	A measure with the ability to provide continuous services to mobile users in the network	Intra-frequency handover out success rate Inter-RAT (radio access technology) handover success rate (LTE to WCDMA (wideband code division multiple access))
Integrity	A KPI that shows the service quality provided to an end-user (experienced user throughput and reliability)	Service uplink/downlink average throughput Bit error rate SINR Packet error rate
Latency	A KPI that shows the delay experienced by an end-user	User plane latency Control plane latency End-to-end latency One-trip time latency
Availability	A KPI that shows availability of a cell	Radio network unavailability rate Cell availability
Traffic	Traffic KPIs are used to measure the traffic volumes on LTE RAN	Radio bearers Downlink/Uplink traffic volume Area traffic capacity Peak data rate
Energy efficiency	A KPI that shows data energy efficiency in operational Evolved Universal Terrestrial Radio Access Network (E-UTRAN)	Spectral efficiency E-UTRAN data energy efficiency

As the 5G era is approaching, the telecommunication network that carries this advanced technology has already become a behemoth. Moreover, it has been growing and expanding repeatedly, from simple calls/communication to increasingly complex information transmission functions. While enjoying the convenience brought by the telecommunication network, each user cannot help asking whether the network used is reliable and stable, whether the data are complete, and whether the transmission is timely. Toward 5G, society has become increasingly dependent on the telecommunication network.

The International Telecommunications Union (ITU) defined the following main usage scenarios in the 5G era: Enhanced mobile broadband (eMBB), which is mainly driven by the need for high data rates and large capacity; massive machine-type communication (mMTC), which asks for energy-efficient communications and wide coverage; and ultra-reliable and low latency communication (URLLC), where high reliability and low latency are crucial. The 3GPP radio access network (RAN) (3GPP, 2018) working group has specified the design targets for the Requirements for the Next Generation, as summarized in Table 6. The details can be found in TR 38.913 (Benjebbour et al., 2018; 3GPP, 2018; ITU, 2017).

Tab.6 5G KPIs proposed by 3GPP

Usage scenarios	KPIs	Target
		Download	Upload
eMBB	Peak data rate	20 Gbps	10 Gbps
	Peak spectral efficiency	30 bps/Hz	15 bps/Hz
	Control plane latency (same as URLLC)	10 ms
	User plane latency	4 ms
	Average spectral efficiency (bps/Hz)	Three times higher than IMT (International Mobile Telecommunications)-advanced
	Area traffic capacity	10 Mbps/m2
	User experienced data rate	100 Mbps	50 Mbps
	5% user spectrum efficiency (bps/Hz/user)	Three times higher than IMT-advanced
	Target maximum mobility speed (same as URLLC and mMTC)	500 km/h
	Mobility interruption time (same as URLLC and mMTC)	0 ms
	Network energy efficiency (same as URLLC and mMTC)	No quantitative requirement
	User equipment energy efficiency (same as URLLC and mMTC)	No quantitative requirement
	Bandwidth	At least 100 MHz; Up to 1 GHz for operation in higher frequency bands (e.g., above 6 GHz)
mMTC	Coverage	Max coupling loss 140 dB
	User equipment battery life	Beyond 10 years, 15 years is desirable
	Connection density	1000000 device/km2
	Latency of infrequent small packets	10 s
URLLC	User plane latency	0.5 ms
	Reliability	1×10−5 success probability for 32 bytes within 1 ms on user plane delay

The above KPIs are used for network monitoring, and the metric value obtained by monitoring is compared with the target value to judge whether the current network meets the requirements. For example, for eMBB scenarios, the peak rate is more important, and the uploading and downloading processes have clear requirements. The peak download rate should not be less than 20 Gbps, while the peak upload rate should not be less than 10 Gbps. For mMTC and URLLC scenarios, the peak rate has no requirement. In the case of URLLC, the latency is very important, and the latency of the user layer is less than 0.5 ms. However, in practical applications, as mentioned above, these metrics are mainly deterministic values obtained through network testing. They are quite different from the reliability probabilistic indicators studied by the academia.

Amid the numerous KPIs that currently exist, some disadvantages have emerged. A large number of KPIs cause warning storms. Any KPI anomalies are alerted. Hence, operators receive an excessive number of KPI alerts every day. In addition, when facing the alerts of KPIs, operators need to invest more resources for maintenance, which increases the burden of operation and maintenance. As mentioned above, several practical metrics are deterministic monitoring values, which do not have statistical meaning and cannot evaluate the probability that the network meets the demand. Hence, the future update and operation of the communication network is difficult to promote. Compared with the power grid, the telecommunication network urgently needs to investigate a unified reliability metric under academic and industrial contexts.

4 Discussions

In this section, we discuss the differences between the reliability metric developments of the two types of networks, such that the useful experience from the power industry could be applied to the development of reliability metrics for telecommunication networks.

(1) The development process of power grid reliability started from the industry. Then, academic researchers began to study it. The problems studied by the academia has a close relationship with the industrial practice, and the gap between the two fields is relatively small. Academic research has also been successfully applied to industrial practice. However, for telecommunication networks, the academia began to study its reliability in theory at the early stage. The industry took a long time before it began to pay attention to the reliability of the telecommunication network and proposed a number of metrics. At the same time, the metrics that the academia studied earlier have not been combined with and applied to the industry. This missing application has made a large gap between the academia and industry.

(2) The power grid reliability has gradually developed into two major categories of distribution reliability and adequacy of generation/transmission. Among them, the power grid reliability metrics represented by EENS can be applied to the entire network. They can also evaluate the end-to-end power grid reliability for multiple services. For the telecommunication network, the traditional reliability metrics were classified into connection-based reliability and performance-based reliability. In theory, they are global metrics and are able to evaluate the reliability of the entire network. However, in practice, no metric has the capability to evaluate the end-to-end reliability of the entire network for multiple services in telecommunication networks.

(3) The number of power grid reliability metrics are appropriate, as the IEEE standard has been established to define those metrics. Furthermore, several reliability metrics have been frequently used by power grid operators and regulators in various countries. However, for telecommunication networks, too many metrics or KPIs exist, as different communication operators use different KPIs. By far, no set of standard metrics has been established across different operators.

(4) With the continuous development and upgrading of science and technology, power grids have gradually developed from central generation to distributed generation. As mentioned above, some metrics are still applicable to the grid of distributed generation. However, with the popularization of renewable energy, numerous customers in the power grid are not only the power demanders but also the power suppliers. Furthermore, they may belong to different operators. Looking for methods to measure the reliability of such customers and determining whether new index parameters need to be added are also issues that should be considered in the future. For the telecommunication network, with the rapid development of the 5G network and the popularity of network slicing, edge computing, and other technologies, the demand of communication network is also diversified, and a reliability metric standard for telecommunication network must urgently be established.

5 Conclusions

In this research, a systematic review is conducted on the reliability metrics of power grids and telecommunication networks. Reliability metrics of two classes for power grid and three classes for telecommunication are discussed respectively. Among them, the power grid reliability metrics developed earlier are relatively mature, and standard metrics have been formed and widely used. Meanwhile, research on the reliability of communication is far from mature. The reliability metrics of the two major infrastructure networks are further compared, which reflects the insufficient development of the telecommunication network metrics.

At the same time, the reliability of telecommunication networks has become increasingly important. On the basis of the current research status of telecommunication network reliability, we suggest some major topics for future studies.

(1) Researchers should establish a unified metric system amid the numerous indicators.

(2) The demands of different users in various scenarios of the telecommunication network are different. Therefore, researchers should determine how to establish the relationship between network requirements and network reliability metrics and then establish a unified reliability standard.

(3) Telecommunication networks are complex systems composed of multiple subnetworks with different properties. Each subnetwork deals with different components. In the future, researchers should explore the methods needed to assess the reliability of the whole network based on the reliability of subnetwork’s components.

(4) The scale of telecommunication networks is growing, with the number of nodes ranging from hundreds to millions. Future research should determine how to evaluate network reliability quickly and efficiently.

(5) The establishment of metrics is for management and supervision. Future research should explore ways to apply unified reliability indicators widely to daily supervision, operation, and maintenance.

References

1 3GPP (2011). TS 32.450: Key Performance Indicators (KPI) for Evolved Universal Terrestrial Radio Access Network (E-UTRAN): Definitions (Release 9)

2 3GPP (2018). TR 38.913: Study on Scenarios and Requirements for Next Generation Access Technologies (Release 15)

3 Adefarati T, Bansal R C (2019). Reliability, economic and environmental analysis of a microgrid system in the presence of renewable energy resources. Applied Energy, 236: 1089–1114

4 Aggarwal K K (1985). Integration of reliability and capacity in performance measure of a telecommunication network. IEEE Transactions on Reliability, R-34(2): 184–186

5 Aggarwal K K, Chopra Y C, Bajwa J S (1982a). Capacity consideration in reliability analysis of communication systems. IEEE Transactions on Reliability, R-31(2): 177–181

6 Aggarwal K K, Chopra Y C, Bajwa J S (1982b). Modification of cutsets for reliability evaluation of communication systems. Microelectronics and Reliability, 22(3): 337–340

7 Alahmed A S, Taiwo S U, Abido M A, Almuhaini M M (2019). Intelligent flexible priority list for reconfiguration of microgrid demands using deep neural network. In: IEEE Innovative Smart Grid Technologies–Asia. Chengdu: 3490–3495

8 Altiparmak F, Dengiz B, Smith A E (2009). A general neural network model for estimating telecommunications network reliability. IEEE Transactions on Reliability, 58(1): 2–9

9 Asokan R (2010). A review of Quality of Service (QoS) routing protocols for mobile Ad hoc networks. In: International Conference on Wireless Communication and Sensor Computing (ICWCSC). Chennai: IEEE, 1–6

10 Benjebbour A, Kitao K, Kakishima Y, Na C (2018). 3GPP defined 5G requirements and evaluation conditions. NTT DOCOMO Technical Journal, 19(3): 13–23

11 Bienstock D, Günlük O (1995). Computational experience with a difficult mixed integer multicommodity flow problem. Mathematical Programming, 68(1–3): 213–237

12 Billinton R, Allan R N (1984). Reliability Evaluation of Power Systems. Springer Science & Business Media

13 Billinton R, Allan R N (1988). Reliability Assessment of Large Electric Power Systems. Springer Science & Business Media

14 Billinton R, Allan R N (1996). Reliability Evaluation of Power Systems, 2nd ed. Boston: Springer

15 Billinton R, Li W (1994). Reliability Assessment of Electric Power Systems Using Monte Carlo Methods. Boston: Springer

16 Božič D, Pantoš M (2015). Impact of electric-drive vehicles on power system reliability. Energy, 83: 511–520

17 Brooks D L, Dugan R C, Waclawiak M, Sundaram A (1998). Indices for assessing utility distribution system RMS variation performance. IEEE Transactions on Power Delivery, 13(1): 254–259

18 Brown R E (2007). Electric Power Distribution Reliability. Boca Raton: CRC press

19 Capra R L, Gangel M W, Lyon S V (1969). Underground distribution system design for reliability. IEEE Transactions on Power Apparatus and Systems, PAS-88(6): 834–842

20 Carlier J, Li Y, Lutton J L (1997). Reliability evaluation of large telecommunication networks. Discrete Applied Mathematics, 76(1–3): 61–80

21 Chamandoust H, Derakhshan G, Hakimi S M, Bahramara S (2020). Tri-objective scheduling of residential smart electrical distribution grids with optimal joint of responsive loads with renewable energy sources. Journal of Energy Storage, 27: 101112

22 Charilas D, Markaki O, Nikitopoulos D, Theologou M (2008). Packet-switched network selection with the highest QoS in 4G networks. Computer Networks, 52(1): 248–258

23 Chiou S, Li V (1986). Reliability analysis of a communication network with multimode components. IEEE Journal on Selected Areas in Communications, 4(7): 1156–1161

24 Chittum A, Relf G (2019). Valuing distributed energy resources: Combined heat and power and the modern grid. Electricity Journal, 32(1): 52–57

25 Cook J L, Ramirez-Marquez J E (2007). Reliability of capacitated mobile Ad hoc networks. Proceedings of the Institution of Mechanical Engineers, Part O: Journal of Risk and Reliability, 221(4): 307–318

26 Darlington M (2016). Evolving 4G KPIs to Improve End User QoE for 4G LTE Broadband Systems. Gweru: Midlands State University

27 Dehghanian P, Fotuhi-Firuzabad M, Aminifar F, Billinton R (2013a). A comprehensive scheme for reliability centered maintenance in power distribution systems—Part I: Methodology. IEEE Transactions on Power Delivery, 28(2): 761–770

28 Dehghanian P, Fotuhi-Firuzabad M, Aminifar F, Billinton R (2013b). A comprehensive scheme for reliability-centered maintenance in power distribution systems—Part II: Numerical analysis. IEEE Transactions on Power Delivery, 28(2): 771–778

29 Dobakhshari A S, Fotuhi-Firuzabad M (2009). A reliability model of large wind farms for power system adequacy studies. IEEE Transactions on Energy Conversion, 24(3): 792–801

30 European Commission (2016). Identification of appropriate generation and system adequacy standards for the internal electricity market. Technical Report

31 Fan H, Sun X (2010). A multi-state reliability evaluation model for P2P networks. Reliability Engineering & System Safety, 95(4): 402–411

32 Gooi H B, Mendes D P, Bell K R W, Kirschen D S (1999). Optimal scheduling of spinning reserve. IEEE Transactions on Power Systems, 14(4): 1485–1492

33 Hariri A M, Hashemi-Dezaki H, Hejazi M A (2020). A novel generalized analytical reliability assessment method of smart grids including renewable and non-renewable distributed generations and plug-in hybrid electric vehicles. Reliability Engineering & System Safety, 196: 106746

34 Hwang C L, Tillman F A, Lee M H (1981). System-reliability evaluation techniques for complex/large systems: A review. IEEE Transactions on Reliability, R-30(5): 416–423

35 IEEE (1999). IEEE Trial-Use Guide for Electric Power Distribution Reliability Indices. IEEE Std 1366–1998

36 IEEE (2001). IEEE Guide for Electric Power Distribution Reliability Indices. IEEE Std 1366–2001

37 IEEE (2012). IEEE Guide for Electric Power Distribution Reliability Indices. IEEE Std 1366–2012

38 International Telecommunication Union (ITU) (2017). Minimum requirements related to technical performance for IMT-2020 radio interface(s). Report ITU-R M.2410-0. Radiocommunication Sector of ITU

39 Jereb L (1998). Network reliability: Models, measures and analysis. In: Proceedings of the 6th IFIP Workshop on Performance Modelling and Evaluation of ATM Networks. Ilkley, T02

40 Kim J O, Singh C (1993). A frequency and duration approach for generation reliability evaluation using the method of stages. IEEE Transactions on Power Systems, 8(1): 207–215

41 Kostanic I, Mijatovic N, Vest S D (2009). Measurement based QoS comparison of cellular communication networks. In: IEEE International Workshop Technical Committee on Communications Quality and Reliability. Naples, FL, 1–5

42 Krasniqi F, Gavrilovska L, Maraj A (2019). The analysis of key performance indicators (KPI) in 4G/LTE networks. In: Poulkov V, ed. Future Access Enablers for Ubiquitous and Intelligent Infrastructures. Proceedings of the 4th EAI International Conference. Sofia: Springer International Publishing, 285–296

43 Krasniqi F, Maraj A, Blaka E (2018). Performance analysis of mobile 4G/LTE networks. In: South-Eastern European Design Automation, Computer Engineering, Computer Networks and Society Media Conference (SEEDA_CECNSM). Kastoria: IEEE, 1–5

44 Lee R M, Assante M J, Conway T (2016). Analysis of the cyber attack on the Ukrainian power grid. Electricity Information Sharing and Analysis Center

45 Lee S H (1980). Reliability evaluation of a flow network. IEEE Transactions on Reliability, R-29(1): 24–26

46 Lemeshko O V, Garkusha S V, Yeremenko O S, Hailan A M (2015). Policy-based QoS management model for multiservice networks. In: International Siberian Conference on Control and Communications (SIBCON). Omsk: IEEE, 1–4

47 Levitin G (2003). Reliability evaluation for acyclic transmission networks of multi-state elements with delays. IEEE Transactions on Reliability, 52(2): 231–237

48 Li R, Li M, Liao H, Huang N (2017). An efficient method for evaluating the end-to-end transmission time reliability of a switched Ethernet. Journal of Network and Computer Applications, 88: 124–133

49 Li Y F, Zio E (2012). A multi-state model for the reliability assessment of a distributed generation system via universal generating function. Reliability Engineering & System Safety, 106: 28–36

50 Lin J, Cheng L, Chang Y, Zhang K, Shu B, Liu G (2014). Reliability based power systems planning and operation with wind power integration: A review to models, algorithms and applications. Renewable & Sustainable Energy Reviews, 31: 921–934

51 Lin Y K (2010). Reliability evaluation of a revised stochastic flow network with uncertain minimum time. Physica A, 389(6): 1253–1258

52 Locks M O (1985). Recent developments in computing of system-reliability. IEEE Transactions on Reliability, R-34(5): 425–436

53 Lu J, Zeng M, Zeng X, Fang Z, Yuan J (2015). Analysis of ice-covering characteristics of China Hunan Power Grid. IEEE Transactions on Industry Applications, 51(3): 1997–2002

54 McCalley J D, Dorsey J F, Qu Z, Luini J F, Filippi J L (1991). A new methodology for determining transmission capacity margin in electric power systems. IEEE Transactions on Power Systems, 6(3): 944–951

55 Melo A C G, Pereira M V F, Leite da Silva A M (1993). A conditional probability approach to the calculation of frequency and duration indices in composite reliability evaluation. IEEE Transactions on Power Systems, 8(3): 1118–1125

56 Memari M, Karimi A, Hashemi-Dezaki H (2021). Reliability evaluation of smart grid using various classic and metaheuristic clustering algorithms considering system uncertainties. International Transactions on Electrical Energy Systems, 31(6): e12902

57 Migov D A, Shakhov V (2014). Reliability of Ad Hoc networks with imperfect nodes. In: Jonsson M, Vinel A, Bellalta B, Belyaev E, eds. Multiple Access Communications. Cham: Springer International Publishing, 49–58

58 Miyoshi N, Shirai T (2014). A cellular network model with Ginibre configured base stations. Advances in Applied Probability, 46(3): 832–845

59 Park Y J, Tanaka S (1979). Reliability evaluation of a network with delay. IEEE Transactions on Reliability, R-28(4): 320–324

60 Pocovi G, Soret B, Lauridsen M, Pedersen K I, Mogensen P (2015). Signal quality outage analysis for ultra-reliable communications in cellular networks. In: IEEE Globecom Workshops. San Diego, CA, 1–6

61 Popovski P, Braun V, Mayer H P, Fertl P, Ren Z, Gonzales-Serrano D, Ström E, Svensson T, Taoka H, Agyapong P, Benjebbour A, Zimmermann G, Meinilä J, Ylitalo J, Jämsä T, Kyösti P, Dimou K, Fallgren M, Selén Y, Timus B, Tullberg H, Schellmann M, Wu Y, Schubert M, Kang D H, Markendahl J I, Beckman C, Uusitalo M, Yilmaz O, Wijting C, Li Z, Marsch P, Pawlak K, Vihriala J, Gouraud A, Jeux S, Boldi M, Dell’aera G M, Melis B, Schotten H, Spapis P, Kaloxylos A, Chatzikokolakis K (2013). Scenarios, requirements and KPIs for 5G mobile and wireless system. ICT-317669-METIS/D1.1. Mobile and wireless communications Enablers for the Twenty-twenty Information Society (METIS), Deliverable D1.1

62 Prempraycon P, Wardkein P (2004). Economic and reliability of telecommunication network design using multiple tabu search algorithm. In: 6th International Conference on Advanced Communication Technology. Pyeongchang: IEEE, 898–902

63 Ramirez-Marquez J E, Coit D W (2005). A Monte-Carlo simulation approach for approximating multi-state two-terminal reliability. Reliability Engineering & System Safety, 87(2): 253–264

64 Ran Y (2011). Considerations and suggestions on improvement of communication network disaster countermeasures after the Wen-chuan earthquake. IEEE Communications Magazine, 49(1): 44–47

65 Rudkevich A M, Lazebnik A I, Sorokin I S (2012). Economically justified locational criteria of the security of supply. In: 9th International Conference on the European Energy Market. Florence: IEEE, 1–9

66 Rushdi A M (1988). Indexes of a telecommunication network. IEEE Transactions on Reliability, 37(1): 57–64

67 Samaan N, Singh C (2002). Adequacy assessment of power system generation using a modified simple genetic algorithm. IEEE Transactions on Power Systems, 17(4): 974–981

68 Sanso B, Soumis F, Gendreau M (1991). On the evaluation of telecommunications network reliability using routing models. IEEE Transactions on Communications, 39(10): 1494–1501

69 Shi J, Wang S, Wang K, Sullivan J P, Chen C Y (2012). Performance-related reliability analysis methodology of computer networks. Communications in Statistics, Theory and Methods, 41(21): 4000–4022

70 Tian M, Gramm A, Naumowicz T, Ritter H, Freie J S (2003). A concept for QoS integration in Web services. In: 4th International Conference on Web Information Systems Engineering Workshops. Rome: IEEE, 149–155

71 Trstensky D, Bowron P (1984). An alternative index for the reliability of telecommunication networks. IEEE Transactions on Reliability, R-33(4): 343–345

72 Warren C A (1996). Distribution reliability: What is it? IEEE Industry Applications Magazine, 2(4): 32–37

73 Wilkov R (1972). Analysis and design of reliable computer networks. IEEE Transactions on Communications, 20(3): 660–678

74 William P (2003). FBI, joint terrorism task force arrest suspect in Arkansas power grid attacks. Available at: forbes.com/sites/williampentland/2013/10/14/fbi-joint-terrorism-task-force-arrest-suspect-in-arkansas-power-grid-attacks/#25a562ae7ea4

75 Wu L, Park J, Choi J, El-Keib A A, Shahidehpour M, Billinton R (2009). Probabilistic reliability evaluation of power systems including wind turbine generators using a simplified multi-state model: A case study. In: IEEE Power & Energy Society General Meeting. Calgary, AB, 1–6

76 Wu Z T, Huang N, Li R Y, Zhang Y (2015). A delay reliability estimation method for Avionics Full Duplex Switched Ethernet based on stochastic network calculus. Maintenance and Reliability, 17(2): 288–296

77 Xiang S, Yang J (2020). K-terminal reliability of Ad Hoc networks considering the impacts of node failures and interference. IEEE Transactions on Reliability, 69(2): 725–739

78 Zhao H, Hu E, Zhang Z, Shen Z, Fu M, Wang R (2019). Reliability assessment of AC/DC hybrid distribution network considering renewable energy. In: 4th International Conference on Power and Renewable Energy (ICPRE). Chengdu: IEEE, 46–50

79 Zhong Y, Quek T Q S, Ge X (2017). Heterogeneous cellular networks with spatio-temporal traffic: Delay analysis and scheduling. IEEE Journal on Selected Areas in Communications, 35(6): 1373–1386

80 Zhu Y H, Yan J, Tang Y F, Sun Y, He H B (2014). Coordinated attacks against substations and transmission lines in power grids. In: IEEE Global Communications Conference. Austin, TX: 655–661

81 Zuo M J, Tian Z, Huang H Z (2007). An efficient method for reliability evaluation of multistate networks given all minimal path vectors. IIE Transactions, 39(8): 811–817