6.2.3.1 Scalability

Scalability is a core requirement, not only for big data management, but also a proper geo-distribution of devices. To state the obvious, [8] proposes to add geo-distributed property as a further data dimension in big data analysis, in order to manage the information distributed nature as a coherent whole. In this scenario, fog plays an important role, thanks to its proximity to the edge and, thus provides information's location-awareness.

By considering scalability referred to big data processes, we feature the characteristic of the framework to scale, depending on the amount of data and, if necessary, being able to manage great amount of data. On the other side, regarding “scalability” of device geo-distribution, we underline the ability of fog computing to manage a large number of nodes in a highly distributed system.

Big data scalability is a fundamental necessity for IoT applications, where a developing number of devices must be interconnected. Geo-distributed scalability is a demand that underlines the paradigm of fog computing to have a capacity of overseeing distributed services and applications, even profoundly distributed frameworks, which falls in stark contrast with the more centralized cloud settings. In exceptionally conveyed systems, fog is dealing with a huge number of nodes across the board in geographic zones, and nodes can likewise be spread out with different degrees of density on the ground. Fog computing is thus dealing with various sorts of topology and distributed configurations systems and have the capacity to scale and adjust so as to meet the demands for every scenario.

6.2.3.2 Interoperability

The IoT is a very heterogeneous setting that is normally found in real-life situations, in light of a wide scope of various devices that gather heterogeneous data from the encompassing geography. Sensors differs when it comes to range coverage, from short- to long-distance coverage. Bonomi et al. [8] lists various heterogeneity inside the fog: (1) fog nodes extend from high-end servers, edge switches, access points, set-top boxes and, even, end devices such as vehicles and cell phones; (2) the different hardware platform has varying levels of RAM memory, secondary storage, and real estate to support new functionalities; (3) the platforms run various kinds of operating systems and software applications resulting in a wide variety of hardware and software capabilities; (4) the fog network infrastructure is also heterogeneous, ranging from high-speed links, connecting enterprise data centers and the core, to multiple wireless access technologies, toward the edge. In addition, inside fog computing, services must be unified on the grounds that they require the participation of various suppliers [9]. Fog computing is a very virtualized setting that needs heterogeneous devices and their running services to be unified under one umbrella in a homogeneous way.

In complex settings, heterogeneity can influence technical interoperability, in addition to semantic interoperability. Technical interoperability concerns communication norms, components executions, or parts interfaces with various information formats or diverse media sorts of data streams. Whereas semantic interoperability is more concerned with the information inside data interleaved and the likelihood that two components comprehend and share similar data in an unexpected way. A standard method to portray and exchange data, together with an abstraction layer that anonymizes physical diversities among components are thus required to make interoperability possible. Diallo et al. [24] explains under which conditions systems are interoperable, proving definitions and classifications and many approaches to address interoperability at different levels.

Fog computing is there for empowering interoperability, so as to make an exceptional information stream to be later handled by sensing and information analytics parts or to host conventional application programming interfaces (APIs) that can be utilized by various applications, without the costly need to move calculations to the cloud layer.

6.2.3.3 Real-Time Responsiveness

Real-time responsiveness is a principle empowering agent for IoT applications and their organization in real-life situations. Fog computing is vital to accomplish low-latency prerequisite in cases where cloud-IoT collaborations are not able to achieve the target latency for several reasons including the distance. (1) IoT and cloud are, in practice, geometrically distant and information requires considerable time in the loop, arriving as an input to cloud and thereafter returning as results (final or intermediate) to IoT devices. Fog is a promising field that alleviates the overhead costs caused by long-distance traveling of information; instead it is clear that performing computations near the edge costs less than sending data all the way downstream to cloud nodes. For some jobs, which do not require high computational power or are less demanding, fog promotes instant computation. (2) Real-time interactions loosely mean processing the unbounded streams of fresh data that arrive continuously in the cloud. Moments where data arrival rates exceed the processing capacities of cloud resources, in addition to Internetwork communication overheads, are not unheard of. Such conditions normally challenge the capacities of fully resource-loaded cloud deployments, where neither reactive nor proactive solutions make a difference. The promising (near) real-time operation of cloud environments is hindered by such facts. To top that off, in a highly dynamic and real-time scenario, such as those in Industry 4.0 (I4.0) or smart cities, data from IoT is fed very quickly to cloud, and because different chunks follow different networking routes, so that it arrives, sometimes, out of order, thus negatively affecting the overall accuracy. Fog diminishes this by sensing information, processing it, and acting in real-time using data that reflect instantly the situation. (3) Sensors accumulate a tremendous amount of continuously arriving data that if sent to the cloud potentially causes system congestion and consequently causing system's performance to hit a wall. On the contrary, fog acts as a front-stage that locates data traffic in a defined space surrounding sensors and preprocesses information before uploading data to the cloud, with lower focused loads and reduced core network load.

6.2.3.4 Data Quality

Data quality is a pertinent demand in real-life scenarios, making it essential for high-performing systems' operations. Also, this element essentially performs initial data filtering so as to early discard unnecessary duplicate and noisy loads, thus improving significantly the overall system quality and performance.

Data quality support is provided by the fog layer with the aim of early discarding useless data, aiming at relieving the burden on subsequent computational stages and consequently decreasing network data traffic by confining traffic near the edge, and reducing the amount of data pushed toward the cloud. Data quality depends on the association of various strategies. The mix of data filtering, aggregation, standardization and analytics, big data and small data analysis is fundamental to understand the surrounding environment, thus performing proactive maintenance and anomaly detection in real time, among a huge amount of data gathered from IoT sensors. In a ubiquitous environment, finding noisy data is challenging, especially in nonstationary settings where data is arriving fast and may thus challenge a front-stage system's resources.

6.2.3.5 Security and Privacy

An essential challenge in ubiquitous settings and fog computing is to harmonize security and reliability of systems, citizens' privacy concerns and personal data control, with the possibility to access data to provide better services. Particularly, with multitenancy support, fog offers policies to specify security, isolation, and privacy management that are required for different applications.

Fog computing is utilized in real applications that appear in critical settings, so reliability and safety are basic requirements. Moreover, it must consider that actuator's operations may be irreversible. Subsequently, the presence of unforeseen conduct, even due to bugs in applications, must be minimized with precautionary measures. Security is a key issue that must be solved to help industrial organizations and it concerns the entire system's architecture, from IoT devices reaching down to cloud. We need to provide important features such as (1) confidentiality, ensuring that the data arrives to the target spot, thus counteracting divulgence of data to unapproved objects with access limitations; (2) integrity, detecting, and preventing unauthorized alteration of the system by steps that control the preserving of consistency, accuracy, and trustworthiness of data over its entire life-cycle; (3) availability, ensuring that services are available when requested by authorized users, and performing repairs, if necessary, to maintain a correct functioning of the system.

A rich arrangement of security features that empowers essential security for every condition for the entire framework is thus required to avoid implementing security mechanisms on a node-by-node basis. Many controls are needed at all levels, including network and communications, from the physical and computational points of view. In fact, intelligence, data processing, analysis, and other computing workloads move toward the edge but, on the downside, many devices will be located in low-security locations, thus protecting devices, and their data becomes a big challenge.

Distributed and internetworked security arrangements are required to ensure complete and superior intelligence and for responsiveness reactions with automated decisions based on M2M communications and M2M security control without human interventions.

What is more, privacy is an undeniably critical issue that is growing in importance with ubiquitous and pervasive settings, where clients are aware concerning the privacy of their private information. Storing encrypted sensitive data in traditional clouds for privacy is not a suitable option as it causes many processing problems when applications have to access these data. In fog computing, personal information is kept in the system for better protection of privacy. It is imperative to characterize the responsibility for information inside the fog, since applications must utilize o information that they have access to [1]. In particular, we must consider the geographic diversity of information and certain data. For example, sensitive military or government data cannot be sent outside of certain geometrical areas. Fog can anonymize and aggregate user data and is thus useful for localizing intelligence and preventing the discovery of protected data. Anyway, it is necessary to introduce additional ways to protect data privacy and thereby to incentive the utilization of fog computing in privacy-critical contexts.

6.2.3.6 Location-Awareness

In dynamic real-world scenario, such as IoT applications, the location-awareness is the property of fog computing, due to its proximity to the edge, to own a widespread knowledge of its subnetwork and to comprehend the outer setting in which it is submerged. Fog improves adaptability because of its behavior adjustability in response to different events, where it adapts itself to better suit certain circumstances, assisted by the awareness of the context.

6.2.3.7 Mobility

Extending the concept of availability and trying to satisfy novel IoT application requirements, fog adapts itself in accordance with geographical distribution of its devices, thus providing mobility support. MIoT (Mobile Internet of Things) is proving itself to be a challenge for distributed supports [25, 26]. The ubiquity of mobile devices raises the need to present mobility support in fog computing, which allows sensing information and reacting while they are moving around the environment. Fog computing, in order to be effective, even with systems that have mobility as a peculiarity, must adjust to oversee high mobility devices. In addition, fog computing supports the likelihood that devices can move between fog nodes without causing issues that may bring the system into halts.

6.4.1.1 Standardization

A standout amongst the most basic focuses for the right coordination and intercommunication between IoT devices and IoT applications is the networking protocol utilized. Those protocols standardize the communication among sensors, actuators, and fog and cloud nodes, allowing programmers to accomplish proper levels of infrastructure interoperability in IoT settings. Various authors [31, 32] partition the infrastructure interoperability into two diverse arrangements of protocols: application protocols and infrastructure protocols. Application protocols are utilized to guarantee communicating messages among applications and their devices (Constrained Application Protocol [CoAP] [33], Message Queuing Telemetry Transport [MQTT] [34], Advanced Message Queuing Protocol (AMQP) [35], HTTP, Data Distribution Service [DDS] [36], ZigBee [37], Universal Plug and Play (UPnP) [38]). The latter are required to launch the underlying interconnection among various networks (RPL [39], 6LoWPAN [IPv6 low-power wireless personal area network] [40], IEEE 802.15.4, BLE [Bluetooth Low Energy] [41], LTE-A [Long-Term Evolution – Advanced] [42], Locator/ID Separation Protocol [LISP] [43]). Each system is to utilize a layered-up stack of protocols relevant to a set of requirements and the sensate traits of every application.

For instance, IEEE 802.15.4 offers a generally very secure wireless personal area network (WPAN), which focuses on low-cost, low-power usage, low-speed ubiquitous communication among devices, and with support to facilitate very large-scale management with a dense number of fixed endpoints [44]. On the top of the IEEE 802.15.4 standard, the ZigBee protocol takes advantage of the lower layers in order to facilitate the interoperability between wireless low-power devices, the optimization for time-critical environment, and the discoverability support for mobile devices. In addition, it is possible to extend the IEEE 802.15.4 standard, with 6LoWPAN network protocol that creates an adaptation layer that fits IPv6 packets, improving header compression with up to 80% compression rate, packet fragmentation, and thus direct end-to-end Internet integration [44].

The smart grid is composed of frameworks made out of an enormous number of distributed and heterogeneous devices spread out in various networks. Wang and Lu [45] represents Smart Grid communication network onto a hierarchical network composed by backbone network and millions of different local-area networks with ad-hoc nodes. Infrastructure interoperability is needed to allow devices and networks to cooperate in order to create a unique vision of the system state or to execute a common task. In smart building applications, in addition to the devices, the integration and communication among buildings allow managers to share the infrastructure and management costs, thus reducing the capital and operational expenses [46]. ZigBee is specifically utilized in smart building and smart grid applications because of its short-range and robustness under noise traits [47]. In [48], ZigBee is utilized in smart grid applications for associating sensors with smart meters, considering the overarching characteristic of its low bandwidth prerequisites and minimal-effort deployment.

ZigBee can also be used in vehicular applications, especially to perform short intra-vehicles communications among all the devices inside a vehicle and, for certain specific applications, to communicate outside the vehicle, as shown in [49], where it is used to improve the safety short-range system requirements. SCV acts in a heterogeneous scenario, with the involvement of a swarm of in-built sensors inside the vehicle that communicate, as well as many types of vehicles, externally seen as macro-endpoints, and access point stations that must cooperate among themselves. Several wireless technologies are used to communicate in the network environment [50]. Hence, infrastructure interoperability must be provided in order to allow V2V communications, access-point-to-vehicle and access-point to access-point.

Another well-known kind of WPAN connection is that of Bluetooth tech, characterized by an exceptionally low transmission range and a poor transmission rate, but on the upside with low power consumption. In IoT situations, BLE has gained momentum. This version extends the Bluetooth technology to face and support connections among constrained devices, optimizing lightweight coverage (about 100 m), latency (about 15 times lower than that of a classic Bluetooth), and energy requirements, with a transmission power range of 0.01–10 mW. BLE provides a good trade-off between energy requirements, communication range and flexibility, and a lower bit rate, combined with a low latency and reduced transmitting power, allows developers to transmit beacons beyond 100 m. For these reasons, BLE has several advantages that make it suitable for V2V applications, and thus could be successfully adopted in SCV systems [51].

Moreover, in heterogeneous environments, OSGi framework is widely adopted as a lightweight application container that defines dynamic managements of software components, allowing deployment of bundles that expose services' discoverability and accessibility. As examples, [52, 53] present platforms for SCV where the middleware is based on OSGi services and modules in order to improve the interaction between devices. In [54, 55] the proposal architecture, utilized in Smart Grid and Smart Building scenarios, capitalizes OSGi as a component platform, which enables to activate, deactivate, or deploy system modules easily, to provide a platform on which new components can be integrated in a plug-and-play fashion, keeping the design as flexible and technology independent as possible.

6.4.1.2 Reliability

Another imperative property of the communication protocols is that of the intercommunication reliability. This property guarantees the gathering of the information transmitted by diverse nodes of the setting. Presently, various systems can be utilized to guarantee the quality of the communications, e.g.: retransmission, handshake, and multicasting.

First, retransmission requires the acknowledgment of every packet and thereby retransmitting each lost packet. Several application protocols, like CoAP, MQTT, AMQP, DDS, concentrate on communication's dependability and are founded on retransmission schemes that are designed to handle the packet loss in lower layers. For example, the scheme per-hop retransmission (often called automatic repeat request [ARQ] at the mandatory access control [MAC] layer) tries to retransmit a packet several times before reaching a defined level before the packet is declared lost. Losses are on-the-spot realized and corrected, and even a few per-link retransmissions can substantially enhance end-to-end dependability [56, 57]. CoAP is based on UDP, an unreliable transport layer protocol, but it promotes the utilization of confirmable messages, that require an acknowledgment message, and non-confirmable messages, that does not need an acknowledgment [58].

Second, the handshake mechanism is designed so that two nodes or devices attempting to communicate can agree on parameters of the interconnection before data transmission. MQ Telemetry Transport (MQTT) and Advanced Message Queuing Protocol (AMQP) support three different layers of reliability that are used based on the domain-specific needs: (1) level 0, where a message is delivered at most once, with no acknowledgment of reception; (2) level 1, where every message is delivered at least once with a confirmation message; and (3) level 2, where the message is delivered exactly once and uses a four-way handshake mechanism.

Third, the publish/subscribe technique permits a device to publish some specific information. Other devices or nodes can be subscribed to that information. To state the obvious, each time the publisher posts new information, it is forwarded to the subscribers. Multiple nodes can subscribe to the same information; therefore, the information would be multicast to all of them. DDS uses multicasting for bringing excellent QoS and high reliability to its applications, with the support of various QoS approaches in connection with a wide scope of adjustable communication paradigms [59]: network scheduling policies (e.g. end-to-end network latency), timeliness policies (e.g. time-based filters to control data delivery rate), temporal aspects for specifying a rate at which recurrent data is refreshed (stated another way, timelines between data sampling), network transport priority policies, and other policies that influence how data is processed alongside the communication in relative to its reliability, urgency, importance, and durability [60].

In SVC and STL case studies, with huge increase in the number of connected vehicles, the number of sensors they incorporate, and with their unprecedented mobility, the support for low latency and unobstructed communication among sensors and fog nodes is crucial to guarantee a correct flow of applications [61]. DDS has been used as basis for building an intra-vehicle communication network. To that end, the vehicle was divided into six modules constituting the intra-vehicle network; the vehicle controller, inverter/motor controller, transmission, battery, brakes, and the driver interface and control panel. Fifty-three signals were shared among the different modules, some of them periodic and some sporadic. The test showed that this protocol improves the reliability and the QoS [60].

6.4.1.3 Low-Latency

As fog computing is implemented at the edge of the network, it is easier to provide low latency response, but it is also necessary to use the right protocol. Distinctive protocols can be utilized to improve the interplay between fog or cloud nodes or among devices and nodes. For instance, [59] utilizes MQTT publish/subscribe protocol to acknowledge continuous and low-latency streams of data, in a real-time setting dependent on fog computing capabilities toward cloud and IoT, utilizing fog layer in the meantime as broker and message interpreter: MQTT conveys data stream among fog and cloud and MQTT-SN, while the lightweight variant transports information from edge devices to the fog layer. CoAP [33] is yet another application protocol specifically utilized in IoT scenarios to provide low-latency cycles. In addition, [58] explains performance differences between MQTT and CoAP, focusing on the response delay variation in association with reliability and the QoS provided for communication: lower packet loss or bigger message size implies that MQTT outperforms CoAP; the opposite holds, reversing the conditions. So, deciding the protocol to use is essential depending on the type of application. In other terms, this means employing MQTT for reliable communications or for communication of large packets, and CoAP elsewhere for decreasing latency and thereby increasing the system's performance accordingly.

Finally, DDS [36] is a brokerless publish/subscribe protocol, recursively used for real-time M2M communication scenarios among resource-constrained devices [59]. Amel et al. [60] consider DDS as a favorable solution for real-time distributed industrial deployments and applies the protocol for improving the performance of vehicular application scenarios, evaluating the performance with tests that encapsulate hard real-time applications benchmark. Hakiri et al. [62] suggests utilizing DDS for enterprise-distributed real-time scenarios and embedded systems, those common in smart grid and smart building applications, for the efficient and predictable circulation of time-critical data.

In smart grid and smart building scenarios, most control capacities have strict latency prerequisites and need instant responses. Low-latency activities are fundamental to improve the framework's adaptability on the two sides of the electricity market. In smart grid scenarios, electricity markets expect to utilize demand-response instance pricing and charge clients for time-fluctuating costs that mirror the time-changing savings for power obtained at the discount level [63]. Wang and Lu [45] highlights the importance of low-latency actions in order to collect correlated data samples from local-area systems to enable a global power signal quality at a particular time instant. All samples must be collected by the phasor measurement unit (PMU) in a timely fashion to estimate the power signal quality for a certain instant and, depending on applications, the frequency of synchronization is usually 15–60 Hz, leading to delay requirements of tens of milliseconds for PMU data delivery. For modern power distribution automation, the IEDs (intelligent electronic devices) that are implanted in substations are sending their measurements to data collectors within 4 ms ranges in row, while intercommunications between data collectors and utility control centers need a network latency that falls roughly within the 8–12 ms range [32]; and for the standard communication protocol IEC 61850, maximum acceptable delay requirements vary from 3 ms; for fault isolation and protection purposes messages, to 500 ms; for less time-critical information exchange, such as monitoring and readings [45].

6.4.1.4 Mobility

One of the characteristics of IoT applications is the high mobility of some of their devices [9]. Currently, there are different protocols applying different techniques to support such mobility, such as routing and resource discovery mechanism.

Routing mechanisms oversee constructing and maintaining paths among remote nodes. Few protocols are responsible for building such routes, despite the need of some nodes for mobility peculiarities. For example, the LISP [43] indicates a design for decoupling host identity from its locational data in the present address scheme. This division is accomplished by supplanting the addresses utilized in the Internet with two separate name spaces: endpoint identifiers (EIDs), and routing locators (RLOCs). Isolating the host identity from its locations allows good improvements to its mobility, by enabling the applications to tie to a perpetual address, which is dubbed as the host's EID. Host location changes commonly amid an active connection. RPL is yet another routing protocol for constrained communications, utilizing insignificant routing necessities through structuring a strong topology over light connections and supporting straightforward and complex traffic models such as multipoint-to-point, point-to-multipoint, and point-to-point [31].

In STL and SCV case studies, mobility and routing support are essential needs for creating dependable applications ready for high rates of mobility of vehicles edges. Vehicles must be managed as macro-endpoints, allowing them to switch from one fog subnetwork to another. SCV must incorporate a routing mechanism for supporting vehicles' mobility externally but it does not require these mechanisms internally, since the swarm of sensors normally act statically inside the vehicle. RPL is adaptable for a multi-hop-to-infrastructure design, as a protocol that enables huge area coverage in real geometries, hosting connected vehicles with minimal deployment of infrastructure [64]. This protocol has practical applications in SCV and STL systems, and it is emerging as the reference Internet-related routing protocol for advanced metering infrastructure applications, since it can meet the requirements of a wide range of monitoring and control applications, such as building automation, smart grid, and environmental monitoring [65].

Resource discovery techniques strategies concentrate on recognizing adjacent nodes when a device relocates, so as to build up new communications. For instance, CoAP is able to discover nodes resources in a subnetwork, through URI that host a rundown of assets given by the server. On the other hand, MQTT does not offer out-of-the-box support resources discovery, and thus clients must understand the message design and associated topics to enable the communication. UPnP is a discovery protocol widely used in many application contexts that enables automatic devices' discovery in distributed environments. Fog solutions use UPnP+ [66], which is complementary to IoT applications. This version encapsulates light-tailed protocols and architectural parts (e.g. REST interface, JSON data format instead of XML) aimed at enhancing communication levels with resource-constrained devices. Moreover, Kim et al. [54] proposes an architectural view encompassing both smart building and smart grid utilizing UPnP for further detecting and adding new devices dynamically with no user intervention, unless the system wants additional information about the user's environment. Likewise, Seo et al. [67] propose using UPnP in vehicular applications, to allow external smart devices to communicate with an in-vehicle network, sharing data over a single network with the services provided by each device.

In other scenarios, like wind farm, smart grid, and smart building, the mobility of devices is not a priority in building the application, due to the static nature of the systems.

6.4.2.1 Safety

Safety is fundamental for critical IoT systems. Safety is most often found in a lawful society and the corresponding business logic of IoT systems. That is, these systems must be designed to maximize the safety of any element, entity, good, or user of the system.

The most widely adopted safety practices are activity coordination for orchestrating actions with a concentration on maximizing the users' safety or even those of goods; activity monitoring for ensuring a streamlined and correct execution of actions; and, action planning for controlling the actions required in hazardous situations by either one of deterministic or stochastic designs [68].

Evaluating the application of these techniques in the different use cases, for instance, in STL, different applications can execute coordinated activities to construct green waves for assisting emergency vehicles in avoiding traffic jams or to reduce noise and fuel consumption [69]. SCV systems also rely on action control techniques for monitoring each and every operation, through acquisition of images or vehicles' movement patterns. More often, all users' actions are traced utilizing targeted surveillances.

As a final consideration, in wind farm scenarios, the framework must face fluctuating weather conditions, compare them with a predefined set of thresholds, and apply a lot of arranged activities, e.g. stopping the turbines in the event of a strong wind, for safety reasons. In [70], the authors survey diverse ways to deal with location vulnerability in wind control generation in the unit commitment issue, with fascinating results that demonstrate the presence of models that can adequately adjust expenses, revenue, and safety in a balanced fashion. What's more, as it is demonstrated in [68], the use of stochastic models for unit commitment, instead of deterministic models, can expand the penetration of wind power without any trade-off of safety.

6.4.2.2 Security

Security is a fundamental perspective that must be confronted by industrial settings. The security of IoT systems is usually supported by at least four basic pillars: the confidentiality of the information, ensuring the arrival of data to safe locations, thus preventing their circulation among unauthorized parties. Data encryption and the use of sandboxes to isolate executions, data, and communications are standard methods of ensuring confidentiality [71, 72]. Data loss, can occur; information can be lost throughout circulation or in the different nodes of the fog environment. Normally, these situations are controlled by various protocols utilized for sending data to fog nodes or to cloud settings, through the channels of version control, and configuration management [73]. The integrity of the data must be ensured, discovering and disallowing nonauthorized dissemination of information throughout the whole cycle. Common ways to ensure integrity are file permissions, user-access controls, checksum, and hashing [74] methods. Intrusion detection allows a system to identify if an nonauthorized client is intending to access protected data [27]. Data analytics techniques are used to detect intrusions (observing the behavior of the system), to check for anomalies, and to discover fault recognitions. Finally, pattern detection techniques can be used to compare the users' behaviors with already known patterns, or with the prediction of the system's behavior. Notwithstanding, adapting intrusion detection mechanisms for each IoT setting is challenging as it requires domain-specific knowledge in addition to the technical aspects.

Some techniques are utilized for the shown case studies. For example, in smart grid and smart building, security levels needs are obviously high, because any potential breach could result in a blackout. In [45], the authors highlight that these attacks commonly come through intercommunication networks, from intruders with the motive and ability to perform malicious attacks, such as the following. (1) Denial-of-service (DoS) attacks can be targeted toward a variety of communication layers (applications, network/transport, MAC, physical) to degrade intercommunication performance and thus obstruct the normal operation of associated electronic devices. (2) Attacks can target integrity and confidentiality of data, trying to acquire/manipulate information. The great number of devices and providers connected to these systems require the implementation of different security policies to prevent and face any possible cyberattack. Those systems have to ensure that devices are protected against physical attacks, utilizing user-access control policies, and also that sensitive data will never be altered throughout their transmission life cycles, using data encryption and sandboxes techniques [32]. Additionally, fault-tolerant and integrity-check mechanisms are normally operated hand-in-hand with power systems for protecting data integrity and for thereby defending and anonymizing user's actions and their associated localizations.

In relation to protocols used in critical applications, the widely adopted solution is to use different protocols for different parts of the system. MQTT benefits in terms of security and privacy from encryption through secure socket layer (SSL)/transport layer security (TLS), but, at the same time, it has a weak authentication phase (e.g. short usernames and passwords) and inappropriate security/privacy policies (e.g. global namespaces that produce global topics). Hence, MQTT in particularly is used to connect local elements in private networks, e.g. connections between a fog node and sensors in the subnetwork. AMQP extends the security and privacy of MQTT with sandboxes for the authentication phase and proxy servers to protect the servers with additional security levels (e.g. firewall protection). In addition, ADQP separates the message and the delivery information, providing metadata management and encrypted message. ADQP is normally used to connect public elements, e.g. connections between fog and cloud. Finally, CoAP enhances the security/privacy match using DTLS (datagram transport layer security) aimed at preventing eavesdropping, tampering, or message forgery toward integrity and confidentiality of exchanged messages.

6.4.2.3 Privacy

In fog computing, personal data are circulated throughout the fog's nervous system and not centralized in some components, which normally improves privacy. Controlling data access remains the top method for ensuring the privacy of personal data that must be protected from unauthorized parties. Few strategies such as user authentication, security token, or air gapping are normally utilized for increasing the privacy of data for privacy-sensitive applications [75]. Moreover, Stojmenovic and Wen [27] emphasize authentication at various stages yet it remains a security issue of high importance in fog settings. Every network device has an IP address and a hacker is able to tamper with a device and send false readings or imitate the IP addresses. To avoid this, a few authentication techniques, based on public key infrastructures or key exchange, are utilized.

In STL, e.g. privacy plays an important role and its concerns are related to the images acquisition of the vehicles approaching the intersection and the close-circuit cameras for vehicle presence detection and monitoring of traffic conditions, which also analyze patterns of vehicles' movements. Likewise, in SCV, the vehicles store information about the drivers' habits, their actions, and their driving patterns. Different user control techniques should be implemented to prevent unauthorized users to access that information.

In smart grid and smart building scenarios, privacy aspects mainly concern a mistaken spread of users' private data (e.g. monetary information and account balance) or information related to energy, from voltage/power readings to device running status to nonauthorized parties. This information is valuable and the system should seek to keep it safe and out of reach of malicious users. Ancillotti et al. [65] classifies the main smart grid vulnerabilities as the following: (1) device vulnerabilities, where attackers tamper with IEDs, which are normally utilized for monitoring electricity production/distribution, with probable not-to-be-underestimated consequences in the language of stealing data or destroying operations, or even aggravating by an unsafe WiFi communications between edge devices; (2) network vulnerabilities, where the adoption of architectures comprising open network is risky and open gates for routing modification attacks, DNS hacking, different DoS; (3) data vulnerabilities, where data is attacked aiming compromising the private data of customers, e.g. man-in-the-middle (MiM) attacks. Out-of-the-box smart grid applications normally ensure stringent access control with minimized levels of abilities performed by every node, which are usually resource-constrained nodes. As fog computing joined the play, smart grid nodes are now off-loading access control to the fog layer, which provides more computational resources and thus boosts more accurate and faster analysis near the edge.

6.4.3.1 Sensors

Sensing is a capacity that normally IoT devices are performing. This element is concerned with two types of sensors: physical sensors, those that get data dynamically as directly connected and implanted within the environment utilizing specific hardware devices, and virtual sensors, those that indirectly acquire data through indirect channels (e.g. using a third-party system web service) [76, 77].

In STL and SCV, sensing is a main element that collects data for helping in accident prevention and maintaining a fluid flow of traffic. Sensing is important to understand the state of the road, and the internal state of the vehicle in SCV. For example, a smart vehicle may have physical sensors to identify wheel pressure, the distance to the nearby vehicles, or the lighting conditions. It can also have different virtual sensors to gather data on forecasting meteorology in its route or different traffic alerts obtained directly from the traffic authority [78]. Regarding wind farm systems, physical sensors gather real-time data related to fluctuations in weather, the wind speed, or the electrical power generated [8]. On the other hand, virtual sensors are fundamental for collecting relevant data and assisting in forecasting weather conditions [79]. Therefore, various reports are composed to confront weather conditions with generated power or even a deviation from forecasted loads.

In smart grid and smart building applications, physical sensor networks are required to sense the whole area pervasively and virtual sensors can provide information on networks incidents and system load. Thus, different statistics and reports are normally generated for comparing data from real-time operations in connection with data related to network incidents, loading, and connectivity. In smart building, sensors sense and share information among different rooms/floors of the building and obtain dynamic information about users' activities and energy supply conditions [46].

Few settings have only a sensing part, since they just watch the environment without concentrating on any actuation act. Business intelligence (BI) applications are a prominent case of this sort of frameworks dependent on sensing. BI applications utilizes procedures such as data discovery [80], data mining [81], business performance, analytics and processing, to convert sensed raw data into valuable insights for subsequent strategic decision-making or for solely visualizing results [82, 83]. In real-world settings, using BI in real-time scenarios in support of knowledge delivery is normally the act [84, 85], thus enhancing the strategic decision-making cycle and maximizing on the cost/profit cycles of the enterprise resources.

6.4.3.2 Actuators

Other systems provide relevant sensing and actuation phases; thus, they actively modify the environment, reacting to current events. Most real use cases are dependent on a sensing phase alongside a robust actuation phase. Henceforth, actuators are normally changing surrounding environments in an automatic or semi-automatic fashion.

This element comprises two types of actuators: physical actuators, which normally produce a physical modification that affects the surrounding entities via specific hardware; and virtual actuators, which are utilized for controlling a group of physical actuators [86], aiming at anonymizing the unnecessarily complicated peculiarities of low-level layers communicating with the physical actuators or otherwise substituting faulty actuators and preventing the system from thereby taking a severe dive [87, 88].

In STL and SCV, the actuation is fundamental in preventing accidents and maintain a fluid traffic flow. In STL, different reaction can be provided in relation of the circumstances. For instance, physical actuators may launch an alarm or alternate a traffic light so that it turns to red so as to slow down near vehicles. On the same side, virtual actuators are utilized for better controlling big areas and thus creating green waves of traffic lights to decrease pollution or for the passage of emergency vehicles. Likewise, SCV provides many virtual and physical reactions for sending warning messages to actuate passively on the driver showing visible or audible signals. Also, it can physically activate a warning or specific piece of hardware for recovering from anomalies as quickly as possible (e.g. slowing down the vehicle or turning some lights on) [89].

In wind farm scenarios, physical actuators are utilized for starting and possibly stopping turbines in response to a forecast and wind speed, thus preventing the system from breaking down [8]. In general we can resume some typical operational scenarios related to weather conditions: (1) in low wind speed, turbines switched off to avoid economic losses; (2) in normal wind speed, normal operation conditions, and blade optimized to maximized the electrical production; (3) in high wind speed, turbines, and power limited in order to avoid exceeding electrical and mechanical load limits; (4) in very high wind speed, turbines switched off to prevent electrical and mechanical breakdowns, due to the high possibility of exceeding allowable values. Also, in the recent literature, researchers of wind turbines replace faulty actuators through corresponding virtual actuators [87].

As a final consideration herein, in smart grid, operational planning and optimization actors employ simulation of network operations, put switching actions on schedule, dispatch repair crews, inform affected customers, and schedule the importing of power [90]. In smart building, the systems handle the energy consumption issue through a strong autonomous and continuous interaction sensing-actuation. Sensors sense and share information among different rooms/floors of the building and then make distributed decisions and activate physical actuators such as lowering the temperature, injecting fresh air, turning lights on and off, opening windows, or removing moisture from the air [91].

6.4.4.1 Data Normalization

As detailed in Section 6.2.1, IoT is an extremely heterogeneous environment that is built upon a comprehensive range of devices that gather information, often heterogeneous, from surrounding environments. Sensors have normally different reachability ranges, fluctuating from weak to robust sensors, with significant differences in power consumption, data rate, and available resources. In the same vein, fog nodes are naturally heterogeneous, with the different levels of services they provide. Hence, all senses are normalized so as to facilitate data dissemination among various devices or between fog nodes. Data normalization is a main data preprocessing procedure for learning data forms [92].

Currently, there are different approaches to normalize all the exchanged data in order to improve the semantic interoperability between devices. In general, interoperability affects the system at different levels. Different approaches tackle this issue using different cross-applications, languages, and data homogenization and serialization mechanisms. Specification languages allow developers to transform all sensed data to a common format. Data homogenization mechanism unify data from different sensors and services. Even though semantic data and open standard middleware can homogenize data, the great variety of data sources leads to the need for new mechanisms. Each IoT application should adopt the technique that best fits their requirements [93]. Data serialization techniques convert and compact data to various formats to transmit them effectively.

Often these mechanics are combined to enhance data normalization. For instance, Zao et al. [59] employ two-level mechanisms for data normalization: first, they utilize a global specification language called Pigi [94] to gather data/metadata fields; secondly, they use the Google Protocol Buffers [95] to perform the data serialization for streaming and archiving the information with a compact binary data encoding.

STL and SCV are systems composed of heterogeneous components that should work appropriately together. Though multiple providers give different specifications, they must exhibit common interfaces and interoperability abilities. Data normalization is highly important in these environments. For example, in SCV, all sensed data are converted to a common language and serialized in order to be efficiently transmitted to a local fog node in the vehicle or to other vehicles or road infrastructures requiring that information.

Smart Grid systems usually comprise a huge mass of distributed and naturally heterogeneous devices, thus demanding the use of protocols so as to achieve the inter-communication between them. Greer et al. [90] indicates eight layers of the degrees of interoperation necessary to enable different transactions on the smart grid, divided among three drivers: (1) an informational driver, which focuses on what information is exchanged and its meaning (using specific languages); (2) an organizational driver, which emphasizes the pragmatic (business and policy) aspects of interoperation, especially those pertaining to the management of electricity; and (3) a technical driver, which involves technical aspects, such as mechanisms to establish physical and logical connections and to exchange messages between multiple systems across various networks. In addition, [90] promotes a few guidelines for identifying standards and protocol in support for interoperability throughout the smart grid, coupled with definitions of architectures for incorporating and supporting a wide spectrum of technologies, spanning from legacy to novel. Also, standard languages are common for performing interoperability between smart meters, smart devices, charging interfaces, and in exchanging information for all smart grid scenarios [96].

In smart building applications, data normalization is a critical activity due to the huge number of devices that must communicate [46]. In addition, these applications could be integrated with smart grid systems with the intention to manage consumption in response to supply conditions by selectively turning off appliances or reducing nonessential or non-time-critical services, with many benefits for suppliers (e.g. avoiding costly capital investments) and customers (e.g. sharing the savings resulting from the lower operational cost of energy production) [46]. In this situation, the language used to transmit the information between different systems is essential in order to achieve the required coordination. These iterations issues are exacerbated by the range of producers that build devices and smart building systems that generally use different type of implementations and with different policies. Hence, smart building applications must define and/or enforce standardized data normalization techniques and guidelines. In this scenario, Chen et al. [46] propose reference semantic models (RSMs) implementing the smart-building related industry standards to facilitate the exchange of information among different subsystems, through a variety of functions including measurements, planning/scheduling, and life cycle management.

Along these lines, a wind farm system is mainly responsible for sensing weather conditions, wind speed, and the turbine power and react interactively with a number of different actuators, thus, data normalization can be achieved. Nevertheless, the most appropriate techniques for data homogenization and serialization are selected so as to exchange information efficiently.

6.4.4.2 Data Filtering

IoT frameworks with several sensors continually sending data to the fog nodes and to the cloud could quickly clog the system and overwhelm the nodes' constrained assets. Data filtering is in part responsible for decreasing the quantity of data transmitted by taking out any information that is repetitive, mistaken, or broken [97]. Data filtering procedures ought to be actualized as close to the edge as possible to decrease information traffic. In spite of the fact that sensors may execute lightweight filtering to evacuate some noise at information accumulation stage, increasingly vigorous and complex data filtering is as yet required at the fog layer. The principal data-filtering techniques consists of duplicate detection, errors detection, and data prioritization. Normally, due to its location near to the edge, fog nodes do not have a widespread knowledge of the whole system, thus, they cannot perform advanced data-filtering operations, so they are left to subsequent stages.

Duplicate detection techniques focus on analyzing received data, from either a specific sensor or a set of sensors, so as to detect duplicated data that can be safely discarded. The Bloom algorithm [98], for instance, detects the redundant data via a buffer that stores the received data and periodically controls whether newly arrival data is present in the buffer. Different algorithms utilize the locality property of IoT systems. They indicate that there is a strong spatial-temporal interconnection between various sensors, with an elevated probability that they collect same data at times and that the same information will probably be gathered by those sensors soon.

Error detection mechanisms are used for identifying flaw data generated by incorrect measures obtained through sensors or noise. [99]. The knowledge of the system, the relation input-output, and the use of secondary data features of data gathered to compare several additional information are basic to detect faulty data. To detect these data, different techniques can be applied. For instance, Statistical model approaches can assist in creating models for predicting data distribution and thus detecting those data which do not fit the model (known as outliers).

Finally, data prioritization techniques filter time-critical data aiming at prioritizing and forwarding them to be processed early by the data analytics component. This is in stark contrast to the aim of reducing data size that is of the other filtering techniques mentioned earlier.

In the STL and SCV case studies, the variance and standard deviation for the vehicles observed speed compared to the average data received at same contexts, with same location, time, weather, and type of vehicle, metrics are normally utilized for identifying outliers and detecting an accurate speed of an approaching vehicle. Consequently, with the definition of data thresholds, it is possible to discard outliers and refine the information collected.

In addition, it is possible to introduce various management control system (e.g. Six Sigma, Lean, etc.) usually adopted in manufacturing or business processing, adapting them to be effective to face the specific problem of data quality improvements. These control systems extend variance/standard deviation measures and are based on a set of advanced statistical methods and techniques, trying to identify and remove all the causes of errors and waste in order to maximize the quality level in relation to the specific application where they act.

In smart building, there are works that focus on reducing the inaccurate sensed information by identifying outliers. The authors indicate that outlier's detection is important in application processing since these erroneous data can lead to abnormal behavior of an application, e.g. turning on the air conditioning when an erroneous measure of the building temperature has been identified. In [47], the authors use the Hodrick-Prescott and moving average techniques to identify the outliers. The Hodrick-Prescott filter is a mathematical tool used to remove the cyclical components of a time series from raw data. The moving average technique is a calculation to analyze data points by creating a series of averages of different subsets of the full data set [100]. The authors applied those techniques to minimize the fluctuation of the temperature/humidity parameters in smart buildings.

Finally, in smart grid, the signals are typically sampled and communicated at high rates – several tens or hundreds of times per second – to augment or even replace the conventional supervisory control and data acquisition system. In [101], the authors indicate that this situation creates some congestion problems, preventing the most critical information to be received at the right time at the nodes where decisions are made and hindering a real-time response. Therefore, they propose the technique of distributed execution with filtered data forwarding to assign more importance to time-critical applications compared to other, less time-critical monitoring applications, prioritizing data for applications having more stringent timeliness requirements compared to ones having lenient QoS.

6.4.4.3 Data Aggregation

The data aggregation element further focuses on minimizing collected information by means of aggregating them into groups based on some keys and constraints for some variables of interest. To such an end, it combines all the gathered information in order to form a unified picture and a unique flow of data. In addition, this component also improves the system interoperability, since it aggregates and merges data from various sensors, ensuring reliable data collected from sensors [102].

Data aggregation can be done using diverse integral perspectives focused on: merging large data sets, hierarchically aggregating data in different nodes and improving the degree of safety via data aggregation. Data fusion methods attempt to combine diverse sorts of information to diminish data size and acquire a novel data stream. Keeping that in mind, a distinctive number of arithmetic operations can be used to get increasingly steady and representative estimations from a big data sample. Spatial methods can be utilized to aggregate data relying upon the sample's geographical location. Hierarchical data aggregation promotes the idea of identifying whether the information can be aggregated on a single node or simultaneously on many nodes. In fog environments that incorporate middle nodes with diverse capacities, this technique normally allows the consequent application of aggregation techniques to enforce location-awareness property for every node. Safety data aggregation centers around gathering similar data from various sensors to have alternate points of view of a similar context with the objective of improving the security and safety of the IoT applications.

The application of these techniques is evaluated in the different cases of uses. For instance, both SCV and STL systems are highly distributed applications with numerous geo-distributed information gatherers that must impart and collect data so as to make productive traffic strategies to draw paths for vehicles.

In SCV, some approaches use data aggregation to monitor the roughness of road surfaces. In [14], the authors propose to collect and fuse different information from accelerometer and GPS sensors and progressively thin it at different levels through sampling and spatial/temporal aggregation techniques. Each collected point is mapped onto a map database and aggregated according to specific geometric constraints. This makes it possible to consistently map the sensed physical quantities of several adjacent points into a single aggregate data. Then, a temporal aggregation is performed. The stored points are kept updated with the last significant changes (incrementally down-weighting older points). Finally, by applying a linear predictive coding (LPC) algorithm to the collected samples, the roughness of the road surface upon which the vehicle travels are estimated as their arithmetic average. This estimate provides significant information on the quality of road surface, given the capability of the LPC algorithm of filtering out (to a certain degree) spurious components of the acceleration signals (engine vibrations, gravitation, inertial forces, etc.).

In STL, the control of the diverse states of the vehicles and the streets is basic to give the ideal safety to the drivers. Currently, different approaches rely on safety data aggregation methods to identify and trace vehicles via observation cameras and diverse sensors (for instance, BOLO Vehicle Tracking Algorithm [103]).

In smart grid, the data produced by the diverse elements of smart grid network is huge. Therefore, in [97], the author proposes the hierarchical aggregation of the data. Power consumption data from different meters are collected and aggregated at prespecified aggregator meters hierarchically. All collected data is aggregated at data aggregation points of power generation plants, distributed energy resource stations, substations, transmission and distribution grids, control centers, and so on. The aggregator meters perform the aggregation by using some arithmetic and temporal operations. The aggregator meter waits for the arrival of data packets associated with the same timestamp from lower meters. For time-aligning the packets from multiple nodes, it temporarily buffers the extracted data, and once all the data with the same timestamp arrive, they are aggregated to create a single data packet. Then, the data are transmitted to the next aggregator meter.

In wind farms, wind turbines transmit data to the base station using multiple hops. Intermediate fog nodes can operate as aggregators reducing the number of direct links from the wind turbines to the base station. In [79], authors apply different data fusion techniques to aggregate the sensed data in the intermediate nodes depending on the physical components, timestamp of the sensed information, sampling rate, and sensor type.

6.4.5.1 Virtualization

Virtualization permits fog nodes to make VMs. Virtualization considers two principle attributes. Tho=ese are the technology utilized to epitomize the IoT framework and how the virtual images are relocated between fog nodes, supporting the clients and framework mobility needs. Currently, the main technologies for creating virtual images are hypervisor and container. Hypervisor is a virtualization approach, where since a virtual image not only contains a final output but, rather, it delivers an operating system required for executing it. OpenStack and OpenNebula [104] are some of the frameworks supporting this virtualization technique. Instead, containers is a lightweight solution for the deployment of isolated execution context in top of an already defined operating system (i.e. it contains the applications, but not the operating systems). Not supporting the emulation of various operating platforms, it improves the execution and the relocation of the containerization parts. LinuX Containers (LXC) [105] or Docker [106] are some representative examples of this technology.

Migration is yet another key characteristic in virtualization. VM migration is essential in fog computing to address the mobility requirement. When a user leaves the area covered by the current fog node, the VMs or the container may need to be migrated to another node covering the user's destination. This process should be fast enough for maintaining real-time and location-awareness needs of IoT settings. Two principal migration approaches are used; a complete migration, in which the entire virtual image is migrated from one node to another (using Internet Suspend/Resume [ISR] [107] or Xen live migration [108], for instance) or partial migration, in which only specific pieces of the virtual image are transferred (utilizing, for instance, alternatives for the previous methods [109]).

Currently, there are many approaches employing those virtualization methods in fog computing settings. Cloudlet was proposed before fog computing, but both share the same concept. Actually, as it is indicated in Section 6.2.2, the Open Edge Computing Consortium promotes its use. Cloudlet comprises a three-layered-up stack, with the base layer containing a working OS, such as Linux, in addition to the information cache from the cloud. The center layer incorporates a hypervisor to isolate the transient guest software environment from the cloudlet infrastructure. Concretely, this layer is based on OpenStack++ [110], which is a specific extension of OpenStack including a set of cloudlet-specific APIs. The third upper layer constitutes applications separated by various VMs. Finally, cloudlets again deploys a specific method for the subtotal relocation of VM instances, dubbed as dynamic VM synthesis. Each cloudlet node comprises a basic VM and every mobile device constitutes a tiny VM overlay. Hence, when a mobile device relocates, a source node stops the overlay and saves it in the mobile device. When the mobile device hits the destination, it transmits the VM overlay into the destination node, subsequently applying it to its base, and thereby beginning its running in a precise state where it left off.

In [71, 72], the authors characterize an exploratory fog processing platform. This stage utilizes a hypervisor virtualization method so as to give adaptability. Solidly, they utilize OpenStack together with the Glance module for the administration of VM images. Likewise, they additionally execute two diverse migration schemes to encourage the meander of VM images among fog nodes. In the main technique, they take a snapshot of the VM to be relocated, compress it, and afterward exchange the compacted data to the target fog. In the second strategy, the VM has a base snapshot entrenched on the two fogs with the goal that they just exchange the incremental part of the VM's snapshot. IOx [111] is the Cisco implementation of fog computing, providing uniform and consistent hosting capabilities for fog applications across its own network infrastructure. IOx works by hosting applications in a guest operating system running a hypervisor directly on the connected grid router. The platform also supports programmers in running applications embedded on Docker and LXC, packaged as a VM, compiled and run as Java or Python bytecodes.

In addition, different platforms have been developed specifically focused on supporting some of the case studies evaluated in this paper.

Truong et al. [50] proposes a platform using software-defined networking (SDN) and fog computing to reduce the latency and improve the responsiveness of SCV applications. Concretely, the authors deploy a fog infrastructure on the roadside-unit controller (RSUC) and on the cellular base station (BS), allowing the computation and storage of some information from the vehicles in these elements. For supporting the virtualization, they use the hypervisor technique. Then, different vehicular and traffic services can be hosted in the VMs allowing service migration and replication.

In smart building, [112] proposes the platform ParaDrop to exploit the underused resources of WiFi access point or cable set-top box, provided by the network operators, with the aim of making them smarter and reducing the information transmitted to the cloud. To that end, ParaDrop uses the LXC abstraction for providing resource isolation and allowing third-party developers to deploy their services using this container. The containers retain user state and can move with the users as the latter changes their points of attachment. This platform has been used to support the control of environmental sensors (humidity and temperature) and security cameras, improving the privacy and the latency and providing local networking context to the system, since all the information is computed in the local node and only the data requested by end-users are transmitted through Internet.

6.4.5.2 Storage

Given the huge amount of data generated by smartphones and IoT devices, data must be allocated as close as possible to users. Information is first stored on fog nodes so as to accelerate its processing, decreasing thereby the information exchange latency and increasing the systems QoS.

Various approaches include storing that data on fog nodes or on different components of the network.

From one viewpoint, the fog system is able to handle local storage for keeping data in a disk-residence fashion. Based on priority, data are stored locally in a compressed manner, thus improving their security and privacy. Typically, these OSs store information on a single node (similar to a semi-centralized mode) or on various nodes (thus similar to a distributed mode).

Every virtualization mechanism and solid framework used by the operating system can implement one or both models. As detailed above, one of the widely used Hypervisor frameworks is OpenStack. The basic implementation of OpenStack can complements different modules providing different storage capabilities. The most important modules are Cinder and Swift. Cinder gives persistent block storage to guest VMs. This module encourages the storage of information on a given fog node, utilizing a centralized mode. Cinder virtualizes the administration of block storage devices and gives end clients a self-managed API to ask for and expend those assets. Swift functions as a distributed, API-open storage system that can be incorporated straightforwardly into IoT applications or used for storing VM images.

The fog platform presented in [89] is based on OpenStack, including the module Cinder for data storage. Therefore, this platform allows the centralized data storage in a specific fog node. Cloudlet is built on OpenStack++, which is mainly focused on improving the cloudlet deployment. Nevertheless, as OpenStack++ is plugging on top of OpenStack, it supports the addition of Cinder or Swift components. In addition, in [113], the authors present CoSMiC, a cloudlet-based implementation of a hierarchical cloud storage system for mobile devices based on multiple I/O caching layers. The solution relies on Memcached as a cache system, preserving its powerful capacities, such as performance, scalability, and quick and portable deployment. The solution aims to be hierarchical by deploying memcached-based I/O cache servers across all the I/O infrastructure data paths.

Containers, such as Docker or LXC, again provide various functionalities for caching and storing data. For instance, in [114], the authors evaluate Dockers as an edge computing platform. Every Docker container is separated and comprises its network subsystem in addition to memory and file systems. For data storage, Docker utilizes a lightweight file system dubbed as UnionFS to provide the building blocks for containers, allowing the caching of data to decrease user local access and to improve overall application performance. Concretely, Flocker is a container data-volume manager. It allows state-full container, e.g. production database data, to be protected. Any container regardless of its location can use Flocker.

On the other hand, other relevant works consider caching users' information on network infrastructure. Suchinformation is typically stored in a reactive fashion, thus caching data as users request it; or proactively, analyzing users' requirements on data and precaching accordingly.

The content delivery network (CDN) [115] serves as a representative for caching networks. CDN acts as an Internet-based cache network by disseminating cache servers at Internet edge so as to reduce upstream lateness caused by contents arriving from remote sites.

Information centric network (ICN) [116] is a wireless cache infrastructure that provides content distribution services to mobile users with distributed cache servers. Different from the cache servers in ICN, the fog servers are intelligent computing units. The fog servers can be connected to the cloud and leverage the scalable computing power and big data tools for rich applications other than content distribution, such as IoT, vehicular communications, and smart grid applications [9, 117].

Bastug et al. [118] detail that the information demand patterns of mobile users are predictable to an extent and propose to proactively precache the desirable information before users request it. The social relations and device-to-device communications are leveraged. The proactive caching framework described by Bastug et al. can be applied in fog computing.

6.4.6.1 Data Analytics

Data analytics is an advanced analytics techniques application to data loads, aiming basically at gaining deep insights [61]. Concentrating basically on the location where data is analyzed, this component is normally divided into big data analytics and small data analytics. Big data analytics relies completely on cloud settings in order to perform complex operations for huge avalanches of fast-arriving big data loads, being able to identify complex associations, patterns, and trends [119]. On the contrary, small data analytics are performed near the edge, and thereby close to the IoT devices, suited to be handled by fog nodes. Small data analytics can handle only small amounts of fine-grained data that naturally provide beneficial information for a system, thus enabling real-time decisions near the edge in an as-fast-as-possible manner. The centralized model is perfectly suitable for orchestrating data generated by many applications with an appropriate response time. Likewise, a decentralized model provides real-time information for fast and short-term decision-making. Nevertheless, when many geographically distributed devices produce data requiring analysis alongside on-the-fly decisions, hierarchical data analytics models seem to respond better to different scenarios. Instead of sending all gathered data to the cloud, information can be stored, and small data analytics can be executed in the fog nodes. Then, relevant or complex data can be aggregated and posted to the cloud environment. In doing so, more complex and resource-demanding analytical techniques can be performed, and medium or long-term decisions can follow, depending on their outcomes [5].

Methods herein are employed for our aforementioned case studies. For instance, in STL and SCV systems, small data analytics are used for constructing a nearly exact picture of the surrounding current situation and thus assisting decision-maker components in reacting spontaneously. Both STL and SCV act in a critical context where the quickness of the decisions plays a key role in these systems than can even save many lives. These systems collect data regarding traffic density, vehicle-related data (for instance, speed, traces, internal condition), movements of different vehicles or bicyclists in lanes and emergency routing. The fog node should store this data and execute fast analysis to recognize certain events on the street, e.g. which vehicles are coming closer, crossing, or traveling along the road, and foresee where they will likely move. In this way, a basic setting for actuation must be able to prevent systems from crashing and improve safety. In this sense, Hong et al. [103] promoted the MCEP system for overseeing traffic via several patterns.

In the case of a wind farm, big data analytics are more often used, because not only are instant actions significant but also insights into future events are useful. A common issue in wind darm applications is to gather enough real-time data of good quality and, therefore, some ISO/RTO introduce mandatory data reporting requirements for wind power producers, with penalties for noncompliance [70]. As a general consideration, it is of great importance to ensure a specific exactness level when applying wind forecasting methods, especially those that are short-lived, so as to improve the quality of a wind power generator that the system offers to markets and to schedule appropriate levels of operating reserves needed to perform the different regulation tasks [68].

In smart grid and smart building use cases, layered-up data analytics are essential to guarantee that systems are working effectively and to accurately oversee dynamic end-client requests and distributed producing sources, supporting instant responses if an occurrence of unforeseen occasions should arise. Chen et al.'s [46] stacked-up data analytics are central to confront sustainable power source supply capriciousness that might be a important factor in connection to weather fluctuations, since each middle node serves as a functioning control unit, processing and responding to passing information [96]. In this scenario, many utility companies, with the intention to improve coordination and control techniques, have introduced smart meter readings to remotely meter-gathered data. As an example, consider consumption records, electricity production, and alarms [32].

6.4.6.2 Decision-Making

As the speed at which the collected information must be transmitted and processed in IoT scenarios increases, the readiness to settle on decisions to trigger explicit business procedures and standards in the correct minute is pivotal and it plainly influences the asset usage and consumer loyalty levels essentially [120]. Depending on how fast a decision is to be considered, the decision-making element is thus divided into predictive and reactive.

The predictive model stores all the data gathered by the sensors, the system's behaviors, its performance, etc., to get a profound learning of the surroundings and the framework so as to trigger the most suitable responses for every context and to surmise conceivable system advancement. These models focus on data calculation and gaining insights, thus constructing clear and predictive models from extensive volumes of data gathered so as to predict the relationship between settings' data. Despite what might be expected, reactive models react in the briefest conceivable time to various occasions that occur in the environment so as to endeavor to create corrections as early as possible. These models serve in a way that enables it to accomplish an ideal objective while interfacing with an environment.

More often, however, systems have diverse dimensions in the decision-making process consolidating at various degree the two models, yet our proposal depends on which part is progressively created and on which functionality the system is increasingly cantered around.

Few systems work in settings where predictions are vital, and they perform comprehensive data calculation and analytics.

6.5.1.1 Standardization

The first step is selecting a protocol for communicating different system devices. This allow developers to standardize the communication between the different elements of the system and, at the same time, to choose the one that best suits the system's requirements. The selection of communication protocols is very dependent on the specific requirements of the application. Right now, various protocols can be chosen to improve the communication at network level, or between various devices or parts of the system. To state the obvious, numerous protocols at the application level depend on explicit protocols at the foundation level. Hence, certain attributes of the latter are acquired by the former. For instance, the ZigBee protocol depends on the IEEE 802.15.4 standard.

Specifically, by analyzing our case studies, we have noticed that ZigBee is especially recognized in IoT scenarios such as smart grid, smart building, and smart vehicle. This may be attributed in part to its short range and robustness with present noise conditions. In addition, especially in smart building and smart vehicle environments, the BLE has also gained momentum recently, in part due to its low-power energy consumption, short-range communication, and elasticity.

6.5.1.2 Reliability

The communication standardization is important for the correct exchange of information, but its reliability is also critical. This requires a system that confirms that appropriate data levels are being received instead of being lost, thus ensuring a correct flow of the system in most circumstances. Therefore, the protocols for communicating the different parts of the applications must contain techniques ensuring such reliability. As can be seen in Table 6.1, protocols such as CoAP, MQTT, AMQP, and DDS are based on retransmission techniques, allowing message retransmission a certain number of times until it is received. Algorithms such as MQTT and AMPQ also encompass handshake methods to establish the most appropriate parameters to ensure such communication. In addition, in order to further increase the reliability of the communications, other protocols implement multicasting techniques, so that different nodes can get the transmitted information.

Those techniques are commonly found in cases studies encompassing critical environments, such as smart vehicle and STL, since reliability is an essential requirement.

Nevertheless, as Table 6.1 shows, all the reliability methods found in the surveyed proposals are adopted by application protocols.

6.5.1.3 Low-latency Communication

A low-latency communication is important in IoT applications so that the system can respond adequately to certain situations. Low-latency must be accomplished among fog nodes, in addition to fog nodes and those in the cloud. CoAP, MQTT y DDS are protocols in support for low-latency of communication among various nodes. In addition, MQTT proposes the usage of various versions, a version for the communication between devices and fog nodes, and the other one is for the exchanging information between fog and cloud nodes. In the case studies evaluated, DDS is promoted in different situations associated with smart grid, smart building, and smart vehicle as it provides efficient and predictable distributions for time-critical data.

Similar to reliability, in the surveyed methods, low latency is focally accounted for by protocols of the applications (Table 6.1).

6.5.1.4 Mobility

The protocols figured beforehand normally encompass various techniques, such as routing and discovery mechanisms to improve and facilitate the mobility of the different nodes of the system. As indicated in Section 6.2, this is a fundamental requirement of IoT systems.

According to the analyzed solutions, mobility support is essential for infrastructure and application protocols in row. Two of the protocols that support this feature are RPL and LISP, making it easy for a device to migrate from one subnetwork to another. In addition, when a device changes from one context to another, it must facilitate the device in discovering services and resources disponible in that context. Normally, this characteristic is provided by application protocols. CoAP and UPnP are two of the analyzed protocols supporting this type of mobility.

Again, STL and SCV are the case studies where a correct implementation of the mobility techniques is crucial, since vehicles may be constantly moving from one fog node to another and the system operations should not be diminished by it. RTL is being adopted in other settings in addition to those described herein. In these same environments, the resource discovery techniques are used to identify and interact with new devices automatically.

6.5.2.1 Security

IoT systems are environments where data is normally scattered between different elements. Thus, the communication between them is constant. Therefore, techniques ensuring the security of such communications are essential.

This security is fundamentally given by the communication protocols. As can be seen in Table 6.2, some of them implement different techniques to encrypt the data transmitted between the different elements of the system. In addition, other protocols, such as CoAP, encapsulate techniques for improving integrity of data, ensuring that the exchanged messages have not been manipulated during the communication.

On the other hand, security has also to be provided by every system. As Table 6.2 shows, the methodologies and case studies assessed basically focus on privacy and integrity of the information to guarantee that all communications are made by approved people and to ensure that the exchanged data is not altered.

Nonetheless, distributed and interlinked security mechanisms are urgent to integrate supreme solutions with a superior intelligence for responsiveness reactions and automated decisions based on M2M communications and M2M security control, without human interventions.

6.5.2.2 Privacy

For all case studies, privacy is considered a focal job. They store and analyze privacy-sensitive data and any unapproved access involves an incredible hazard for the IoT frameworks and for their clients. Nevertheless, as can be seen in Table 6.2, few of the analyzed solutions include mechanisms for the data privacy control, concentrating mainly on applications security and safety.

6.5.2.3 Safety

Many IoT applications work in critical environments where safety techniques must exist for ensuring the correct operation in suspected situations.

While analyzing the reviewed methodologies, we have distinguished that those case studies including an incredible number of devices and requiring an instant actuation more often than not employ one of two methods: activity monitoring and coordination. The former is utilized to precisely know the condition of every component, while the previous is applied to trigger facilitated activities to act over the environment and meet the system's objectives.

But, on the case studies where the reaction ought not be continuous such as those cases found in wind farm, there is a more noteworthy emphasis on the utilization of planned action techniques, since this permits a fine planning of each activity so as to augment the system benefits and safety.

6.5.3.1 Sensors

Sensing is a capacity performed by all those case studies. They all gather sense the surroundings gathering that assist in obtaining credible results, analyze the circumstance and make decisions accordingly. Both virtual and physical sensors are used herein.

Few systems just require a sensing element, as an actuation phase is simply not there serving no purpose. More often than not, those systems acquire such data solely for analyzing the surrounding environment, or visualizing reports, or for conducting executive-level strategic decision making, such as establishing a company strategy for upcoming years.

6.5.3.2 Actuators

Real-time responsiveness requires more than sensing. Here is where actuators come into play. They work normally by changing the environments aiming at automatically or semi-automatically achieving desired goals. All case study systems clearly demand physical actuators. However, few of them use virtual actuators.

6.5.4.1 Data Normalization

Normalization collected is essential in unifying all heterogeneous data coming from different sources under one umbrella. In the case studies analyzed, we have recognized that present solutions are typically founded on proposing normal dialects improving the communication between various devices or nodes or some serialization instruments diminishing the asset consumption that is caused by exchanging data.

6.5.4.2 Data Filtering

Data filtering is a widely spread approach for reducing the sizes of data loads arriving in avalanches. Currently, many solutions employ methods for specifying duplicates and, detecting outliers. On the other part of the story, data prioritization is less famous and receives less attention specifically used in settings where different components of a system receive diverse priorities.

However, filtering critical data can be off-loaded to fog nodes, thus, prioritizing them on the fog environment level.

6.5.4.3 Data Aggregation

Data aggregation can normally be performed by fog nodes with certain computing and storage capacities. The vast majority of the techniques analyzed focus on fusing data, because it is the focal constituent part of this element. Various techniques also exploit a layered-up aggregation, on the price that this implies that common aggregation methods are deployed in each fog node or systems' coordinators have to perfectly know the resources for every fog node and its location so as to establish the aggregation method that is to be employed.

Also, data aggregation safety is yet another aspect widely accepted in critical scenarios. Nevertheless, they also require to know the different fog nodes in order to establish specific encryption and access control mechanisms.

6.5.5.1 Virtualization

To create a fog platform capable of supporting different IoT applications, designers must analyze and clearly define the virtualization technology to use. As can be seen in Table 6.5, Hypervisor is the most extended technology, because is flexible and a wider range of IoT applications are exploiting it for their deployments, because the required OS is emerged in the VM. However, for settings where fog nodes have lower computing resources, or the type of deployed applications, and their requirements, are known and more or less fixed, containers can provide additional advantages, because it is a lightweight solution. Specifically, analyzing the surveyed solutions, those approaches proposing general fog platforms usually implement the hypervisor technology or both. More specific approaches, such as ParaDrop, use containers.

Also, all platforms allow the relocation of VMs. Often, containerized applications are totally relocatable, while platforms depending on the hypervisor technology permit a complete move of the VM images or complete and partial relocation. This is in part due to a large size of hypervisor-based VM images. It worth noting that some approaches propose few efficient algorithms for effectively migrating VMs images using IoT devices for transmitting information without overburdening network shoulders.

6.5.5.2 Storage

When it comes to storage, all fog nodes are welcoming. Naturally, data storage respects a centralized model, storing data in the same fog node that receives it. However, different approaches also implement the distributed storage of information, thus enhancing users' mobility alongside network and capacity of storage but increasing network overload.

Moreover, other approaches are working on directly storing data on the network infrastructure. These solutions allow the deployment of a large number of servers to cache the information in a distributed way. In addition, some proposals are working on predicting the information that will be required by users in order to store it in the network infrastructure close to the users that will request it. These approaches usually do not implement any method for the deployment of IoT applications on these servers, but they can be used to store the information produced by the IoT applications deployed on near fog nodes or incorporated in the fog platform.

6.5.6.1 Data Analytics

IoT systems aim mainly at analyzing massive amounts of sensor-collected datasets from diverse contextual conditions, and consecutively extracting a knowledge-base that serves top tiers in stacked-up architectures mainly for visualization and reporting in support for strategic decision making. Adopted data deep analytical techniques depend highly on the volume of the data in addition to its being time-sensitive, safety-sensitive or alike. Those techniques, often, consider the geo-locality of the arrived data so as to decide keeping it in fog or turning it on to a remote resource-rich cloud. Thus, the stringent the temporal constrains and the smaller the data set to analyze, the greater the number of techniques that should be relegated to the fog nodes. Instead, if the data set is very large, the analytics techniques are resource consuming, and the temporal requirements are not very strict, a higher number of these techniques should be executed in the cloud.

Referring to our case studies, we know intuitively that most of them require instant decision making, thus are relying highly on processing some tasks near the edge, in fog specifically. However, some systems, such as those of wind farms, which normally require long-haul forecasting analytics more than often delegate such compute-intensive tasks to a resourceful cloud. It worth mentioning however that hybridizing capacities of the both worlds, small data analytics and big data analytics, is capable of providing the system with a performance boost.

6.5.6.2 Decision-Making

Upon analyzing collected data, the resulting information is normally used for making different decisions. If the decision-making process is automated or semi-automated, it can be performed in both the cloud and the fog nodes. As mentioned earlier, this depends, for the most parts, on the target response time, in such a way that those that are time-sensitive can perform some parts of their jobs near the edge (in fog), whereas those with more relaxed time constraints can wait normally for the whole cycle to complete, from sending collected data remotely to cloud all the way back to visualization and reporting In this sense, the data analytics and the decision-making elements are mostly related, since their distribution between fog and cloud nodes is usually similar.

Wind Farm, smart building, and smart grid are those scenarios such that the temporal requirements are normally eased, so many of their decision-making functionalities can be relegated to the cloud.

On the other hand, in critical systems, such as STL and SCV, the low-latency reaction is a key necessity and is pivotal for ensuring safety. These environments are predominantly reactive.

Nevertheless, most systems comprise various levels constituting the decision-making process hybridizing both models at different degrees.