10.2.1 Software‐Defined Networking

While networks are based on distributed CP solutions, there is a huge interest around SDN orchestration mechanisms that enable not only the separation of UP and CP, but also the automation of the management and service deployment process. Current SDN approaches are mainly focused on single‐domain and single‐vendor scenarios (e.g., data centers). However, there is a need of SDN architectures for heterogeneous networks with different technologies (e.g., IP, MPLS, Ethernet, and optical), which are extended to cover multi‐domain scenarios.

The SDN architecture, as defined by the Open Networking Foundation (ONF) in [3], is composed of an application layer, a control layer, and an infrastructure layer, as depicted in Figure 10‐1. User or provider‐controlled applications communicate with the SDN controller via an Application‐Controller Plane Interface (A‐CPI), also known as northbound interface (NBI). The controller is in charge of orchestrating the access of the applications to the physical infrastructure (i.e., the NEs), using a Data‐Controller Plane Interface (D‐CPI), also known as southbound interface (SBI).

Figure 10‐2 presents a more descriptive view of a typical SDN architecture, where a management plane is also included, to carry out tasks such as registration, authentication, service discovery, equipment inventory, fault isolation, etc. In addition, Figure 10‐3 shows the situation where the infrastructure owner gives away control of part of its infrastructure to a number of external entities. This is relevant to scenarios where a network provider gives controlled access to equipment (or a slice of equipment through virtualization mechanisms) to some other service providers.

ONF also describes the possibilities of implementing hierarchical controllers, primarily for scalability, modularity or security reasons. Such hierarchical control structure introduces a new interface, the Intermediate‐Controller Plane Interface (I‐CPI), as shown in Figure 10‐3. This hierarchical structure allows for recursiveness and to assure scalability, while maintaining the control of each domain in separate controllers.

In terms of functionalities, there are four main capabilities in this kind of interfaces enabling the flexible control and orchestration of different resources. Such capabilities are: (1) network topology extraction and composition, (2) connectivity service management, (3) path computation, and (4) network virtualization.

The need of network topology extraction and composition is to export the topological information with unique identifiers. Such network identifiers (such as IPv4 addresses or datapath‐IDs) are required for the other functionalities. To compose the topology, it is required to export the nodes and the links in a given domain, which can be physical or virtual, as well as some parameters like the link utilization or even information about physical characteristics of the link if the operator requires the deployment of very detailed services.

The second functionality is to manage connectivity services. The operations on these services are the setup, tear down, and the modification of connections. Such services can be as basic as a point‐to‐point connection between two locations. Nonetheless, there are scenarios where the orchestration requires more sophistication like (a) exclusion or inclusion of nodes and links, (b) definition of the protection level, (c) definition of traffic‐engineering (TE) parameters, like delay or bandwidth, or (d) definition of disjointness from another connection.

The third function is the path computation, which is fundamental as it provides the capability of properly defining an E2E service. For instance, when different controllers in a multi‐domain environment are considered (e.g., in situations where multiple network segments are under a single administration, such as backhaul, metro and core networks), this permits to interact with individual controllers in each domain that are only able to share abstracted information that is local to their domain. The orchestrator with its global end‐to‐end view can improve end‐to‐end connections that individual controllers cannot configure. Without a path computation interface, the orchestrator is limited to carrying out a crank‐back process that would not find proper results. This can be exploited as well when multiple technologies are considered, following a multi‐layer decision approach.

Lastly, a network virtualization service allows to expose a subset of the network resources to different tenants. This advances in the direction of network slicing, where resources and capabilities of the underlying physical transport network can be offered to different users or tenants to appear as dedicated in its global network slice composition, as detailed in Chapter 8.

The ONF architecture presented here illustrates the general enablers for the objective of network programming. However, several other organizations are working on the standardization of NBIs and SBIs. In terms of maturity, there is not yet a complete solution for each model, but multiple candidate technologies for some interfaces. This is commented later on in this chapter.

10.2.2 Network Function Virtualization

European Telecommunications Standards Institute (ETSI) NFV is the most relevant standardization initiative arisen in the NF virtualization arena. It was incepted at the end of 2012 by a group of top telecommunication operators, and has rapidly grown up to incorporating other operators, network vendors, information and communications technology (ICT) vendors, and service providers. To date, the ETSI NFV industry specification group (ISG) can count on over 270 member companies. It represents a significant case of collaboration among heterogeneous and complementary kinds of expertise, in order to seek a common foundation for the multi‐facet challenges related to NFV towards a solution as open and scalable as possible.

The ETSI NFV roadmap initially foresaw two major phases: The first one was completed at the end of 2014, where a number of specification documents were issued [4], covering functional specification, data models, proof‐of‐concept (PoC) description, etc. The second phase released a new version of the ETSI NFV specification documents. A third phase is ongoing at the time of writing, progressing the work on architectural and evolutionary aspects. The work of the ISG is further articulated into dedicated working groups (WGs). In phase 1, three WGs have been created, dealing with NFVI, Management and Orchestration (MANO), and Software Architectures (SWA). In phase 2, two additional WGs were spawned, dealing with Interfaces and Architecture (IFA) and Evolution and Ecosystem (EVE).

The currently acting specification of the ETSI NFV architecture was finalized in December 2014 [5], and its high‐level picture is shown in Figure 10‐4.

The ETSI NFV specification defines the functional characteristics of each module, their respective interfaces, and the underlying data model. The data model is basically made up by static and dynamic descriptors for both virtual network functions (VNFs) and network services (NSs). The latter are defined as compositions of individual VNFs, interconnected by a specified network forwarding graph, and wrapped inside a service.

The ETSI NFV framework specifies the architectural characteristics common to all the VNFs. It does, though, not rule out which specific network functions can or should be virtualized, leaving this decision up to the NF provider (apart from the use cases advised for the proofs of concept).

The ETSI NFV architecture supports multi‐point‐of‐presence (PoP) configurations, where a PoP is defined as the physical location where a NF is instantiated. A PoP can be mapped to a data center or a data center segmentation isolated from the rest of world.

A summary description of the modules in the ETSI NFV architecture is given in Table 10‐1.

As it can be observed in Figure 10‐4, the ETSI NFV framework assumes the existence of an outside operations support system (OSS) or business support system (BSS) layer in charge of the basic equipment and service management.

It is worth to mention that starting 2016, ETSI has launched the Open Source Mano (OSM) initiative [6]. OSM intends to develop an open‐source NFV Management and Orchestration (MANO) software stack aligned with ETSI NFV specifications. This kind of open‐source software initiative can facilitate the implementation of NFV architectures aligned to ETSI NFV specifications, increasing and ensuring the interoperability among NFV implementations.

10.3.1 Open and Standardized Interfaces

Through the existence of controllers allowing the programmability of the network, the operational goal is to facilitate the creation and definition of new services to be configured in the network and automatically, via OSS or directly by means of the interaction with tailored applications. The SDN controller will in this context take care of performing all the tasks needed to set up the configuration in the network (i.e., calculate the route from source to destination, check the resource availability, set up the configuration to apply in the equipment, etc.). For example, the inventory system can be better synchronized with the network, so that the provisioning can be done based on the real status of the network, avoiding any misalignment between the planning process and the deployment process.

One of the expected benefits of SDN is to speed‐up the process to integrate a new vendor, or a new OSS system or application in the network. To do so, it is necessary to have standard NBI interfaces towards the OSS systems (e.g., network planning tools, inventory data bases, and configuration tools), and standard SBI interfaces towards the network element that depend only on the technology (e.g., microwave wireless transport, Metro‐Ethernet/IP, or optical) and not on the vendor.

Nowadays, even for a single transport technology, the particularities per vendor implementation force a constant customization of the service constructs. This affects not only the provisioning phase, but also the operation and maintenance of the services. Activation tools (as part of current OSS and BSS) are in some cases present, being in charge of the automated configuration of network services. However, the configuration is provided by vendor‐dependent interfaces, and when a service needs to be extended by configuring different network segments, the configuration process needs to be done in each network separately, and usually by means of specific or dedicated systems. For the same reason, integrating a new vendor or new equipment (or a new release of an existing vendor or equipment) is time‐consuming, and needs upgrades of the interfaces and changes in the OSS tools already deployed. It delays the introduction of new technologies, de facto blocking the transformation process towards 5G with the agility and flexibility needed by the operator. All of this renders the adoption of open and extensible interfaces, for both NBI and SBI, necessary.

Currently, there is no real progress about the definition of NBIs from the orchestrator perspective that could facilitate the smooth integration referred to before with respect to OSS and BSS. All the available NBIs are platform‐dependent; in consequence, there is not a common or general approach in the industry by now. However, for the SBI there is some consensus.

For the programmability and management of the network, both NETCONF and YANG, as introduced in the following, are being recognized as future‐proof options.

NETCONF [7] provides a number of powerful capabilities for a uniform configuration and management of network elements. It is transport protocol independent, meaning that it does not impose restrictions for getting access towards the devices. With NETCONF, it is possible to have a separation of the configuration data from the operational data in such a way that the administrator can set some variables from features like statistics, alarms, notifications, etc. In addition to that, thanks to the support of transactional operations, it is possible to ensure the completion of configuration tasks even on a network basis. Since NETCONF supports an automated ordering of operations, the sequential actions on the network can be defined, facilitating straightforward rollback operations if needed. NETCONF is then foreseen as the manner of managing and orchestrating multi‐vendor infrastructures. However, NETCONF only defines the mechanisms to access and configure the network elements, but not the configuration information to be applied.

In this sense, YANG [8], as a data modeling language, complements NETCONF by defining the way in which the information applicable to a node can be read and written. It provides well‐defined abstractions of the network resources that can be configured or manipulated by a network administrator, including both devices and services. The YANG language simplifies the configuration management as it supports capabilities like the validation of the input data, and data model elements are grouped and can be used in a transaction, etc. Nowadays, there is an intensive work in the definition of general and standard YANG models especially in the Internet Engineering Task Force (IETF), but not only. Figure 10‐5 presents the evolution in the number of YANG models being proposed.

Similar to NETCONF, the RESTCONF protocol [9] provides a programmatic interface for create, read, update and delete (CRUD) operations accessing data defined in YANG based on Hypertext Transfer Protocol (HTTP) transactions, allowing Web‐based applications to access the configuration data, state data, data‐model‐specific remote procedure call (RPC) operations, and event notifications within a networking device, in a modular and extensible manner. The purpose is then similar to the one described for NETCONF.

Regarding the orchestration of services and the management of VNF lifecycles, Topology and Orchestration Specification for Cloud Applications (TOSCA) emerges as the more solid option. ETSI NFV ISG is considering it as a description language, and recently started the specification of TOSCA‐based descriptors [11], not yet being released at the time of writing. Nevertheless, a TOSCA template [12] is available, which is specifically designed to support describing both NS descriptors (NSDs) and VNF descriptors (VNFDs).

TOSCA is a service‐oriented description language to describe a topology of cloud‐based web services, their components, relationships, and the processes that manage them, all by the usage of templates. TOSCA covers the complete spectrum of service configurations, like resource requirements and VNF lifecycle management, including the definition of workflows and FCAPS management of VNFs. By this way, an orchestration engine can invoke the implementation of a given behavior when instantiating a service template.

A topology template defines the structure of a service as a set of node templates and relationships that together define the topology model as a (not necessarily connected) directed graph. Node and relationship templates specify the properties and the operations (via interfaces) available to manipulate components. The orchestrator will interpret the relationship template to derive the order in which the components of the service should be instantiated. TOSCA templates could also be used for later lifecycle management operations like scaling or software update.

From the point of view of communication method, TOSCA uses a simple REST application programming interface (API).

NETCONF/YANG and TOSCA can complement each other. Basically, the lifecycle management of the VNFs can be performed by means of TOSCA, while the VNFs can be dynamically configured at runtime by means of NETCONF/YANG. This interplay is facilitated by architectural propositions like the integrated SDN control for tenant‐oriented and infrastructure‐oriented actions in the framework of NFV, as described in [13]. Figure 10‐6 shows the positioning of the two different levels of SDN control.

The SDN controller in the tenant domain can configure on‐demand of NETCONF/YANG the functionality of the VNFs deployed by using TOSCA.

Furthermore, this architecture facilitates the integration of control and orchestration actions with a SDN controller at the infrastructure level for coordinating actions allowing cross‐layer coordination. Both controllers manage and control their underlying resources via programmable southbound interfaces, each of them providing a different, but complementary, level of abstraction. This concept is leveraged from [14].

10.3.2 Modeling of Services and Devices

The same need for standardization as highlighted before would be also necessary for services and devices. By expressing a service to be deployed in a standard manner, it is possible to make it independent or agnostic of the actual underlying technology in which it is engineered. This provides more degrees of freedom for the decisions about how to implement a given service, and also allows for portability of such service across platforms.

Via those models, a unique entity can process all the service requests, later on triggering actions in the network for service delivery and deployment. Such an entity can be seen as a service orchestrator, which can maintain a common view across all the services deployed, instead of the legacy approach of siloed services, which renders a combined planning difficult. With such service orchestrator, dependencies can be detected in advance, allowing to improve the design by means of a coordinated usage of resources.

Similarly, the definition of common models for the same type of device simplifies the management, operation and control of the nodes in the network. A common representation of node capabilities and parametrization produce homogeneous environments removing the particularities that motivate onerous integration efforts as happens today to handle per‐vendor specificities.

A generic reference about service models can be found in [15] and [16].

10.4.1 Multi‐Domain Scenarios

When talking about multi‐domain, different meanings can be associated to the term domain. For instance, this can refer to different technologies, like packet, optical, microwave, etc., or different network segments. Finally, multi‐domain can be understood as a multi‐operator environment, with the interaction of different players for the E2E provision of a service. We use the term multi‐domain for multi‐operator environments and multiple administrative scenarios in this section. The importance of analyzing such scenarios was firstly raised in [17].

5G is expected to operate in highly heterogeneous environments using multiple types of access technologies, leveraging multi‐layer capabilities, supporting multiple kinds of devices, and serving different types of users. The great challenge is to port these ideas to the multi‐domain case, where the infrastructure (considered as network, computing, and storage resources), or even some of the necessary network functions, are provided by different players, each of them constituting a separate administrative domain.

Multi‐operator orchestration requires the implementation of an E2E orchestration plane able to deal with the interaction of multiple administrative domains (i.e., different service and/or infrastructure providers) at different levels, providing both resource orchestration and service orchestration. An example would be the case of service providers offering their NFVI PoPs to host service functions of other providers, or even offering VNFs to be consumed by other service providers. However, existing interconnection approaches are insufficient to address the complexity of deploying full services across administrative domains. For instance, evolved interconnection services demanding, e.g., computing capabilities for the deployment of network building blocks as VNFs, or even inserting VNFs in the UP, cannot be satisfied with existing solutions for multi‐domain environments.

An inter‐provider environment imposes additional needs to be offered and served between providers like service‐level agreement (SLA) negotiation and enforcement, service mapping mechanisms (in order to assign proper sliced resources to the service instance), reporting of assigned resource and service metrics, and allocation of proper control and management interfaces, to mention a few.

From the architecture perspective, an orchestration approach assuming a hierarchical top‐level orchestrator playing the role of broker, with total visibility of the all providers' networks, and with the capability of orchestrating services across domains is certainly impractical, due to issues like scalability, trustiness between providers, responsibilities, etc. Instead, a peer‐to‐peer architecture seems to be more adequate for this kind of scenarios, as it already exists nowadays in the form of the pure interconnection for IP transit and peering.

From the point of view of SDN architecture, a primary approach to such a peer‐to‐peer relationship is provided by ONF in [18], which introduces an initial idea about the interaction of peer controllers, as reflected in Figure 10‐7. Here, basically, each of the controllers may act as client to invoke services from the other as server, whereby A‐CPI is the Application‐Controller Plane Interface, and D‐CPI the Data‐Controller Plane Interface. The relationship among controllers is then proposed to be equivalent to a hierarchical provider/customer relationship.

For more complex orchestration scenarios, involving the provision of NFV‐related services across providers, some other initiatives are in progress. To this respect, ETSI has produced a report on the description of architectural options for multi‐domain [19], taking as basis for the analysis some use cases like NFVI‐as‐a‐Service.

The Metro Ethernet Forum (MEF) Lifecycle Service Orchestration (LSO) is another initiative in the standardization arena, with a reference architecture defined in [20]. The MEF LSO architecture oversees the E2E orchestration of services where all the network domains require coordinated management and control. A shared information model for connectivity services is under definition, including the service attributes defined in MEF service specifications. Specifically, two inter‐provider reference points are being proposed:

• LSO Sonata, which facilitates the interconnection of the BSS functions of different providers, addressing the business interactions between those providers. This includes aspects such as ordering, billing, trouble ticketing, etc.;
• LSO Interlude, which instead facilitates the interconnection of the OSS functions of different providers. Interlude supports control‐related management interactions between two service providers and is responsible for the creation and configuration of connectivity services as permitted by service policies. It also covers notifications and queries on the operational state of services and their performance.

Co‐operation between providers then takes place at the higher level, based on exchanging information, functions and control. These interfaces serve for the business‐to‐business and operations‐to‐operations relations between providers.

In addition, the 5G PPP 5G‐Exchange (5GEx) project [21] has developed a multi‐domain orchestration framework enabling the trading of NFs and resources in a multi‐provider environment, and targeting a Slice‐as‐a‐Service (SaaS) approach. The envisioned 5G service model is an evolution of the ETSI NFV model, proposing extensions to it. The original NFV paradigm foresees that resources used inside a service (for instance, for different VNF components) can be distributed over distinct PoPs (i.e., physical infrastructure units, typically data centres). However, the PoPs are supposed to be under a unique administration. Furthermore, the level of control is quite limited outside the perimeter of the data centers as for instance in wide area networks (WANs). The project addressed these limitations, aiming at functionally overcoming them (i.e., enabling the integration of multiple administrative domains) and at least assessing the non‐functional enablers needed to make actual business out of the technology.

5GEx has built on top of the concept of logical exchange for a global and automatic orchestration of multi‐domain 5G services. A number of interfaces implement such kind of exchange for the CP perspective. This ecosystem allows the resources such as networking, connectivity, computing and storage in one provider’s authority to be traded among federated providers using this exchange concept, thus enabling service provisioning on a global basis.

Figure 10‐8 presents a high‐level overview of the 5GEx architecture. Different providers participate in this ecosystem, each of them representing a distinct administrative domain interworking through multi‐domain orchestrators (MdOs) for the provision of services in a multi‐provider environment. This architecture extends the ETSI MANO NFV management and orchestration framework for facilitating the orchestration of services across multiple administrative domains. Each MdO handles the orchestration of resources and services from different providers, coordinating resource and/or service orchestration at multi‐provider level, and orchestrating resources and/or services using domain orchestrators belonging to each of the multiple administrative domains.

The domain orchestrators are responsible of performing virtualization service orchestration and/or resource orchestration exploiting the abstractions exposed by the underlying resource domains that cover a variety of technologies hosting the actual resources.

There are three main interworking interfaces and APIs identified in the 5GEx architecture framework. The MdO exposes service specification APIs that allow business customers to specify their requirements for a service on business‐to‐customer (B2C) interface I1. The MdO interacts with other MdOs via business‐to‐business (B2B) interface I2 to request and orchestrate resources and services across administrative domains. Finally, the MdO interacts with domain orchestrators via interface I3 APIs to orchestrate resources and services within the same administrative domains.

Figure 10‐9 presents the functional detail of the proposed architecture, showing different components identified as necessary for multi‐domain service provision. In this case, all the providers are considered to contain the same components and modules, although in Figure 10‐9 the complete view is only shown for the provider on the left for simplicity.

We briefly describe some of the components in the figure, particular to 5GEx.

• The inter‐provider NFVO is the NFVO implements multi‐provider service decomposition, responsible of performing the end‐to‐end network service orchestration. The network services operator (NSO) and resource orchestration (RO) capabilities are contained here;
• The topology abstraction module performs topology abstraction, elaborating the information stored in the resource repository and topology distribution modules;
• The topology distribution module exchanges topology information with its peer MdOs;
• The resource repository that keeps an abstracted view of the resources at the disposal of each one of the domains reachable by the MdO;
• The SLA manager is responsible for reporting on the performance of its own partial service graph (its piece of the multi‐domain service);
• The policy database which contains policy information;
• The resource monitoring module dynamically instantiates monitoring probes on the resources of each technological domain involved in the implementation of a given service instance;
• The service catalog in charge of exposing available services to customers and to other MdOs from other providers;
• The MD‐PCE (multi‐domain path computation element) devoted to make the necessary path computations and to set up the connection between domains.

From the interfaces perspective, the functional split considered is related to service management (‐S functionality), VNF lifecycle management (‐F), catalogs (‐C), resource topology (‐RT), resource control (‐RC) and monitoring (‐Mon). Table 10‐2 summarizes the functional needs for the mentioned interfaces as well as potential candidate solutions for their implementation. At the time of writing, the identification and specification of these interfaces is currently being defined and will be fully described in future deliverables of the project.

Figure 10‐9 shows the interconnection of MdOs for three different domains. As mentioned, the left MdO is shown with full details, while the other two are skipped for simplicity. The 5GEx interfaces are presented with the corresponding functional split. The interfaces have to be considered as symmetric, since the consumer‐provider role is situational in an exchange.

The left MdO is the entry point for the service request coming from the customer, through the I1 interface. Using I1‐C, I1‐S and I1‐F, the customer (e.g., an infotainment company) will be able to request VNF instantiation and configuration, apart from expressing the way in which they are interconnected by means of a service graph.

The service will be decomposed by the NFVO of the provider A. If the service cannot be honoured by the sole use of its own resources, the NFVO will make use of resources offered by other providers in the exchange. The availability of resources from other parties is collected via I2‐RT, and the availability of services offered by such parties is obtained through I2‐C. Once the decision about using resources from other providers is taken, the left MdO will make use of I2‐S and I2‐RC for requesting and controlling the necessary resources and services. The same MdO will make use of the I3 interface for governing the own resources accordingly, in a similar manner.

In order to accomplish the negotiated SLA between the parties (i.e., both the customer and the entry provider, and the providers participating in the E2E service provision), convenient monitoring capabilities are deployed, using I1‐Mon, I2‐Mon and I3‐Mon for the respective capabilities.

As a reference of the different roles in the exchange, note that the provider B in Figure 10‐9 (i.e., the one in the middle) participates on the E2E service only for providing UP connectivity between providers A and C.

10.4.2 Multi‐Technology Scenarios

Nowadays, the automatic establishment of E2E connections is complex in a network composed of heterogeneous technological domains (that is, domains constituted by specific technologies like IP, optics, microwave, etc.). The complete process not only requires long time and high operational costs for configuration (including manual interventions), but also the adaptation to each particular technology implementation. The capability to operate and manage the network automatically and E2E is the main requirement for multi‐technology scenarios. This facilitates as well the multi‐vendor interworking, which is another dimension of the multi‐technology issue, as already described in Section 10.2.1 in the context of the relevance of SDN. The target is to move towards a service‐driven configuration management scheme that facilitates and improves the completion of configuration tasks by using global configuration procedures.

Typically, the transport network is referred to as a wide area network (WAN) in the ETSI NFV model, regardless of the complexity and diversity of the underlying infrastructure. The idea of the ETSI model is that the service orchestrator can easily interact with control capabilities that could permit the configuration and manipulations of the WAN resources to create E2E services without considering the transport domains’ heterogeneity. However, this is yet far from existing capabilities and solutions.

Network operators have built their production networks based on multi‐layer architectures. However, the different technologies in current transport networks are rarely jointly operated and optimized, i.e., the implications of a planning and configuration decision for different layers at the same time are typically not considered. Instead, they are usually conceived as isolated silos from a deployment and operation point of view.

This can be even more burdening across multiple domains as described before. A service deployed across domains will require actions in different networks using different technologies, inherently multiplying the intricate complexity of the E2E network provision and configuration.

A logically centralized orchestration element can have a complete and comprehensive network view independently of the technologies employed in each technological domain, and propose optimal solutions to improve the overall resource utilization. Such orchestrator, by maintaining a multi‐layer view of the controlled network, can determine which resources are available to serve any connectivity request in an optimal manner, considering not only partial information (per technology domain), but the entire network resources, in a comprehensive manner. Aspects like global utilization, protection, congestion avoidance or energy saving can be optimized with such an approach. For getting the information per technology and building the multi‐layer view (i.e., underlying topology, per‐layer capabilities, border ports, etc.), the orchestrator could rely on lower‐level controllers, for instance one per layer. In [22], an overview of the benefits obtained through a multi‐layer approach is provided.

Network programmability, as enabled by SDN and already touched with relation to the radio access network (RAN) in Section 6.8, permits new ways of resource optimization by implementing sophisticated traffic engineering algorithms that go beyond the capabilities of contemporary distributed shortest path routing. Multi‐layer coordination can help to rationalize the usage of technologically diverse resources. This new way of planning and operating networks requires a comprehensive view of the network resources and planning tools capable for handling this multi‐layer problem.

10.5.1 Xhaul Software‐Defined Networking

10.5.1.2 Possible Hierarchical SDN Controller Approaches for Xhaul

A possible solution to manage and control such diversity of heterogeneous technologies is to focus on a deployment model in which a SDN controller is deployed for a given technology domain (considering it as a child controller), while the whole system is orchestrated by a parent controller, relying on the main concept of network abstraction [25].

The proposed SDN architecture by ONF foresees the introduction of different levels of hierarchy, allowing for network resource abstraction and control. A level is understood as a stratum of hierarchical SDN abstraction. In the past, the need of hierarchical SDN orchestration has been justified by two purposes: a) Scaling and modularity: each successively higher level has the potential for greater abstraction and broader scope (e.g., RAN, transport, and data center network abstraction); and b) Security: each level may exist in a different trust domain, where the level interface might be used as a standard reference point for inter‐domain security enforcement. The benefits of hierarchical SDN orchestration become clear in the scope of the described Xhaul with technology heterogeneity.

The Applications‐Based Network Operations (ABNO) framework has been standardized by the IETF, based on standard protocols and components to efficiently provide a solution to the network orchestration of different CP technologies. An ABNO‐based network orchestrator has been validated for E2E multi‐layer and multi‐domain provisioning across heterogeneous control domains employing dynamic domain abstraction based on virtual node aggregation [26].

Figure 10‐10 shows the proposed hierarchical architecture for a future Xhaul network. It takes into account the different network segments and network technologies which are expected to be present. In the RAN segment, we observe several SDN‐enabled controllers for wireless networks, which tackle their complexities. In a transport network, the aggregation segments and core network are taken into account. SDN‐enabled Multiprotocol Label Switching ‐ Transport Profile (MPLS‐TP) can be used in the aggregation network, while a core network might use an optical SDN controller, such as an active stateful path computation element (AS‐PCE) on top of an optical network. Finally, several SDN‐enabled controllers are responsible for intra‐data center networks, which typically run at layer‐2.

Within the hierarchy, an SDN orchestrator may consider itself as the direct control entity of an information model instance that represents a suitably abstracted underlying network. It follows that, with the exception of network domain SDN controllers (which are directly related to NEs), a given SDN orchestrator might provide an abstracted network view and be present at any hierarchy level and act as parent or child SDN orchestrator. At any level of the recursive hierarchy, a resource is understood to be subject to only one controlling entity.

In the proposed architecture, several child ABNOs (cABNO) are proposed. Each cABNO is responsible for a single network segment. A recursive hierarchy could be based on technological, SDN controller type, geographical/administrative domains or network segment basis (each corresponding to a certain hierarchical level). Further, parent ABNO (pABNO) are introduced, responsible for the provisioning of E2E connections through different network segments.

For both the pABNO and the cABNO, the internal system architecture is similar, based on a set of components that are displayed in Figure 10‐10 and detailed in [26]. The network orchestration controller is the component responsible for handling the workflow of all the processes involved (e.g., the provisioning of E2E connectivity services). It also exposes a NBI to offer its services to applications. For the cABNO, the NBI of the network orchestrator controller is extended to offer a REST‐based interface for topology recovery and connection provisioning based on the Control Orchestration Protocol [27], which has evolved in ONF T‐API and IETF TE models.

Figure 10‐10 also provides the different topological views at different hierarchical levels (top hierarchical level for the pABNO, and lower hierarchical level for the different segments). The provided topological views correspond with the proposed experimental validation, where a pABNO and cABNO‐T and cABNO‐DC are deployed. The cABNO‐T is responsible for SDN orchestration of two SDN aggregation domains and an SDN core network domain. The cABNO‐DC is responsible for two intra‐DC network domains.

The hierarchical SDN approach benefits single operator scenarios, where multi‐layer, multi‐vendor, and multi‐technology SDN controllers are needed. For multi‐operator scenarios, where centralized elements may be impractical, a peering model as presented in Section 10.4.1 may be the preferred option [21].

10.5.2 Core Transport Networks

The evolution towards fully operational 5G networks imposes a number of challenges that are usually perceived as impacting only the access networks, although this is not actually the case. Network functions, as integral parts of the services offered to the end‐users, have to be composed in a flexible manner to satisfy variable and stringent demands, including not only dynamic instantiation but also deployment and activation. In addition to that, and as a complement of it, the whole network should be programmable to accomplish such expected flexibility, allowing for interconnecting the network functions across several NFVI‐PoPs and scaling the connections according to the traffic demand. The versatile consumption of resources and the distinct nature of the functions running on them can produce very variable traffic patterns on the networks, changing both the overlay service topology and the corresponding traffic demand. The location of the services is not tightly bound to a small number of nodes any more, but to distributed resources that are topologically and temporally changing. The network utilization then becomes time‐varying and less predictable. In order to adapt the network to the emergence of 5G services, the provision of capacity on demand through automatic elastic connectivity services in a scalable and cost‐efficient way is required. The backbone or core transport networks then become a key component for E2E 5G systems.

The transformation objectives of the core transport networks have been traditionally focused towards more affordable and cost‐effective technologies, being able to cope with the huge increase in traffic experienced in the latest years, at a reduced cost per bit. 5G networks, however, present innovative requirements to be faced by the transport networks, like the need to accommodate a large number of simultaneous devices, provide transport and service resources in a flexible and dynamic manner, and reduce the provisioning time to make such flexibility functional. Specifically, 5G transport networks will have to support high traffic volumes and ultra‐low latency services. The variety in service requirements and the necessity to create network slices on demand will also require an unprecedented flexibility in the transport networks, which will need to dynamically create connections between sites, network functions or even users, providing resource sharing and isolation. Key aspects on the concept of network slicing are [31]: (i) resource manageability and control, (ii) virtualization through abstraction of the underlying resources, (iii) orchestration of disparate systems, and (iv) isolation of the offered compound assets in the form of slice. 5G transport shares all those goals. Moreover, the flexibility required by 5G transport, such as the dynamic creation and reconfiguration of network slices, makes some of the requirements even more stringent.

The programmability of the transport networks will be performed through open, extensible APIs and standard interfaces that permit agile E2E service creation in a rapid and reliable way. The goal is to evolve towards E2E automated, dynamically reconfigurable and vendor‐agnostic solutions based on service and device abstractions, with standard APIs able to interoperate with each other, and facilitating a smooth integration with the OSS and BSS deployed by network operators.

From a complementary angle, transport networks will also have a very relevant role in the optimization of RAN resources by enabling flexible fronthaul and backhaul systems, maximizing the benefits provided by distributed and virtualized RAN environments, tailored to the needs of a variety of vertical customers. The support of different functional splits in the radio part, the packetized transport of such signals, and the dynamic location of the processing units will render a full programmability and dynamicity in the transport part necessary.

Network management and orchestration mechanisms at transport level are needed in order to create the programmable environment required for 5G networks. The purpose is to integrate this programmable transport infrastructure with the overall 5G orchestration system, creating, managing and operating slices for different customers.

10.7.1 Motivation

To meet the diverse and stringent KPI requirements specified in ITU‐R IMT‐2020, the 5G system will necessarily become more complex [32], which can be mainly characterized by the following technical features: 1) a heterogeneous network consisting of marco cells, small cells, relays, and device‐to‐device (D2D) links; 2) new spectrum paradigms, e.g., dynamic spectrum access, licensed‐assisted access, and higher frequency at mmWave bands, as elaborated in Chapter 3; 3) cutting‐edge air‐interface technologies, such as massive antenna arrays and advanced multi‐carrier transmission, as detailed in Chapter 11; and 4) a novel E2E architecture for flexible and quick service provision in a cost‐ and energy‐efficient manner, as introduced in Chapter 5.

The system's complexity imposes a high pressure on today's manual and semi‐automatic network management that is already costly, vulnerable, and time‐consuming. However, mobile networks’ troubleshooting (related to systems failures, cyber‐attacks, and performance degradations, etc.) still cannot avoid manually reconfiguring software, repairing hardware or installing new equipment. A mobile operator has to keep an operational group with a large number of network administrators, leading to a high operational expenditure (OPEX) that is currently three times that of capital expenditure (CAPEX) and keeps rising [33]. Additionally, troubleshooting cannot be performed without an interruption of the network operation, which deteriorates the end user's quality‐of‐experience (QoE) [34]. Without the introduction of new management paradigms, such large‐scale and heterogeneous 5G networks simply become unmanageable and cannot maintain service availability.

Recently, the research community has started to explore artificial intelligence [35] in order to minimize human's intervention in managing networks to lower the OPEX and improve the system’s performance. IETF has initiated a research group called Intelligence‐Defined Networks to specifically study the application of machine learning technologies in networking. Moreover, the 5G PPP projects SELFNET [36] and CogNet [37] have focused on designing and implementing intelligent management for 5G mobile networks. For example, the SELFNET project has been set up to design, prototypically implement, and evaluate an autonomic management framework for 5G mobile networks. Taking advantage of new technologies, in particular SDN [38], NFV [39], self‐organized networks (SON) [40], multi‐access edge computing (MEC) and artificial intelligence, the framework proposed by the SELFNET project can provide the capabilities of self‐healing against network failures, self‐protection against distributed cyber‐attacks, and self‐optimization to improve network performance and end users' QoE [41]. Although the current SON techniques have a self‐managing function, it is limited to static network resources. It does not fully suit 5G scenarios, such as network slicing [42] and multi‐tenancy [43], where dynamic resource utilization and agile service provision are enabled by SDN and NFV technologies. Currently, existing SON can only reactively respond to detected network events, while the intelligent framework is capable of proactively performing preventive actions for predicted problems. The automatic processing in SON is usually limited to simple approaches like triggering, and some operations are still carried out manually. In addition, the self‐x management mainly focuses on the RAN. An extension beyond the RAN segment to provide a self‐organizing function over the E2E network is required. By reactively and more importantly proactively detecting and diagnosing differently network problems, which are currently manually addressed by network administrators, the SELFNET framework could assist network operators to simplify management and maintenance tasks, which in turn can significantly lower OPEX, improve user experience, and shorten time‐to‐market of new services.

In this section, a reference architecture for the autonomic management framework [36] will be introduced, including the functional blocks, their capabilities and interactions; the autonomic control loop starting from the SDN/NFV sensor and terminating at the actuators will be provided, as well as a brief exemplary loop so as to illustrate how the autonomic system may mitigate a network problem. Furthermore, several classical artificial intelligence algorithms that can be applied to implement the network intelligence are briefly shown.

10.7.2 Architecture of Autonomic Management

In addition to the software‐defined and virtualized network infrastructure [44], the autonomic management framework mainly consists of: 1) SDN/NFV sensors that can collect the network metrics; 2) monitoring modules that can derive the symptoms from the collected metrics; 3) network intelligence that is in charge of diagnosing network problems and making tactical decisions; and 4) SDN/NFV actuators and an orchestrator that perform corrective and preventive actions. As shown in Figure 10‐11, the potential architecture for autonomic management can be split into several layers, which are explained as follows:

• Infrastructure layer: All NFs managed autonomously by the framework rely on physical and virtualized resources in this layer. It encompasses physical and virtualization sublayers. The former provides an access to physical resources (networking, computing, storage, etc.), while the latter instantiates virtual infrastructures on top of the physical sublayer. It represents the NFVI as defined by the ETSI NFV terminology;
• Data network layer: This implies an architectural evolution towards the SDN paradigm by decoupling the CP from the UP. In this framework, the data layer represents a simple data‐forwarding, which can be either a non‐virtualized or virtualized NF;
• SON control layer: This layer includes two internal sublayers: SDN controllers and SON CP sublayer. SDN/NFV sensors and actuators, which are capable of collecting data from the entire system and enforcing actions, respectively, are also contained. The SON control layer and data network layer have associated CPs and UPs of the network that are decoupled in the SDN paradigm;
• SON autonomic layer: To realize the network intelligence, this layer consists of three modules, i.e., monitor, aggregator and analyzer, autonomic manager, and orchestrator. The monitor and analyzer extract metrics related to network behavior, aggregate the collected metrics into health of network (HoN) metrics, and use these to infer the network status. The autonomic manager is in charge of diagnosing the root cause of any existing or potential network problems, and deciding which countermeasure should be conducted. Following the tactical decisions from the autonomic manager, the orchestrator coordinates the physical and virtualized resources, and manages the SDN/NFV actuators to execute the decided actions;
• NFV orchestration and management layer: This layer is responsible for orchestrating and managing VNFs via the VNF manager, as well as virtualized resources through VIM. It conforms to the NFV MANO specified by ETSI [5];
• SON access layer: This is the external interface that is exposed by the framework. Despite the fact that internal components may have specific interfaces for the particular scope of their functions, these components contribute to a general SON API, managed by the SELFNET API broker that exposes all aspects of the autonomic framework to external systems, such as BSS or OSS and administration graphical user interfaces (GUIs). The latter enable network administrators to interact with and configure the SELFNET framework and also observe the complete status of the network.

10.7.3 Autonomic Control Loop

One of the main challenging aspects of the autonomic management is the implementation of network intelligence. Apart from the underlying software‐defined and virtualized network infrastructure, a closed control loop referred to as autonomic control loop, starting from the sensors and terminating at the actuators, is needed to control the processing flow. When the monitor detects or predicts a network problem, an autonomic control loop is initiated. The autonomic manager diagnoses the cause of the problem, decides on a tactic, and plans an action. Once the orchestrator receives an action request from the action enforcer (AE), it coordinates the physical and virtualized resources to enforce this action.

The autonomic manager can be regarded as the brain of the autonomic management framework and plays a vital role in the provision of network intelligence. Taking advantage of cutting‐edge techniques in the field of artificial intelligence, it provides the capabilities of self‐healing, self‐protection and self‐optimization by means of reactively and proactively dealing with detected and predicted network problems. As illustrated in Figure 10‐12, the autonomic manager consists of the following functional blocks:

• A diagnoser is in charge of diagnosing the root cause of network problems. The monitor can derive a symptom for each detected or predicted network problem from the collected sensor data. The diagnoser processes the reported symptom to make clear its reason, and notifies the decision‐maker;
• A decision‐maker (DM) can decide a set of corrective or preventive tactics to deal with the network problems based on incoming diagnostic information. A tactic is a high‐level description of a countermeasure, which needs to be transferred into an implementable action;
• An action enforcer (AE) is responsible for providing a consistent and coherent set of scheduled actions to be enforced in the network infrastructure. For this purpose, this module recognizes and validates these tactics by applying conflict detection and resolution in order to provide implementable actions to be enforced.

Within this control loop, the metrics collected by the sensors are processed by the monitor module first. Subsequent modules extract the required information from the previous module and provide the next‐level results to the next module. The information model associated with the autonomic control loop is explained as follows:

• Sensor data: A range of differentiated data sources can be expected to be identified in the upcoming 5G infrastructure. All monitoring information retrieved from physical devices, UP, SDN controller, SDN/NFV sensors, VIM etc., are uniformly referred to as sensor data. The monitor is the corresponding module that is in charge of collecting sensor data from underlying infrastructures;
• Monitor data: The monitor regularly collects the sensor data and reports the necessary information to the aggregator. Some of the data is periodically collected, which stands for either normal or abnormal network behaviors;
• Aggregated data bundle (ADB): The monitor data related to a network problem may be retrieved from a set of sensors, rather than a single one. For example, in the case of a distributed denial of service (DDoS) attack, the source and destinations are distributed. The raw information contained in monitor data should be processed to produce aggregated and correlated information, which is called aggregated data bundle;
• Symptom: A high‐level health‐of‐network metric that may be derived from a set of correlated alarms, events, KPIs, etc., that can be evaluated to indicate the characteristics of an existing or emerging network problem, together with the additional contextual information such as metadata, is defined as a symptom;
• Performance: The report of achieved performance by an executed action is two‐fold: i) if an action degrades performance rather than solving a problem, a roll‐back mechanism will be triggered to recover the network status to the initial point before the action was performed; ii) the achieved performance, which can be regarded as the benefit or reward of action. If a large extent of operational data can be recorded, the network intelligence can be trained based on machine learning techniques;
• Cause: It is a description of what the reason of a network problem is or why a network problem happens or will happen. Once the diagnoser receives a symptom, it diagnoses the cause of this symptom;
• Tactic: After the cause of a network problem is clarified, a countermeasure that can be applied to tackle this problem needs to be decided by the decision‐maker. A tactic is a high‐level description of a countermeasure, which is required to be transferred into an implementable action;
• Action: This is an implementable version of a countermeasure with a description of how to enforce this, taking into account available physical and virtualized resources. The action provided by the AE contains more implementation details, e.g., the actuator's type, the target deployment location, and configuration information.

To close this section, let us use the following example to show the autonomic control loop and illustrate its main mechanisms. The storyline is depicted as follows: A summer concert is taking place in the city centre, where a large number of spectators gather in a small area. Some of the spectators start to share real‐time videos in their social media. When the number of video users increases, especially if some of them transfer videos in ultra‐high definition, the network suffers from traffic congestion and the perceived QoE deteriorates. The monitoring modules first detect this network's anomaly by means of collecting, aggregating, and analysing the sensor data. A symptom called video QoE decreasing is reported to the diagnoser. After the diagnosis, it is found that the cause of the video QoE decreasing is the increased number of video users in the zone. Then, the possible tactics, for instance, load‐balancing, video coding optimizing, and admission control, are determined by the decision‐maker. The AE transfers these tactics into implementable actions and notifies the orchestrator. Taking into account available resources, the action of load balancing is finally selected and executed by the orchestrator. An actuator acting as a load balancer is instantiated, configured and deployed in the local network surrounding this concert. Afterwards, the congested network is successfully recovered and the perceived QoE of end users is improved.

10.7.4 Enabling Algorithms

We will give a brief introduction about enabling intelligence algorithms. The motivation is to provide a view for the readers how to apply artificial intelligence to implement the network intelligence. Hence, only several classical algorithms are given. For further artificial intelligence technologies, such as neural networks [45], reinforcement learning [46], transfer learning [47], and deep learning [48], the reader is referred to the stated references.