An agent can be regarded as an autonomous, problem-solving, and goal-driven computational entity with social abilities that is capable of effective, maybe even proactive, behavior in an open and dynamic environment in the sense that it is observing and acting upon it in order to achieve its goals (cf., e.g., Wooldridge and Jennings, 1995; Wooldridge, 2002). There are a number of definitions of intelligent agents that need to be extended in the light of long successful research in this area (cf., e.g., Weiss, 1999; Object Management Group, 2004). The set of features that is to be supported when the term (advanced) agent is used encompasses the properties listed in Table 1.1.

1.2.2 Types of Agents

Agent research defines deliberative and reactive agents as the extreme points within the spectrum for the smartness of agents.

Depending on the point of view, a deliberative, respectively (cognitive) intentional agent is either a synonym for a proactive agent or a specialization of it. Its behavior and architecture is reasonably sophisticated (i.e., the internal processes and computation is comparatively complex and, thus, time- and resource-consuming. However, in contrast to human beings, an agent “understands” at most only a small, abstracted portion of the real world, although it has always been intended to equip it with comprehensive real-world knowledge. This goal was in the mind of researchers from the beginning, but up to now has turned out to be too ambitious. Wooldridge (1995) defines a deliberative agent as one “that possesses an explicitly represented, symbolic model of the world, and in which decisions (e.g., about what actions to perform) are made via symbolic reasoning.” The most popular architecture for the implementation of such agents is the belief-desire-intention architecture (BDI) (cf. Bratman, 1987). The beliefs reflect the agent’s abstract understanding of that comparatively small part of the real world it is an expert in. This understanding is subjective to the agent, and thus may vary from agent to agent. The desires represent the goals of the agent (i.e., describe what the agent wants to achieve). It can be distinguished between short-term goals and long-term goals. The long-term goals are those that actually drive the behavior of an agent, and thus are comparatively stable and abstract. They form the underlying decision base for all (re)actions of the agent. Short-term goals only reflect goals that the agent wants to achieve in a specific situation. They may only express what the agent can do in this specific situation at most, and so usually only have a temporary character. As Logan and Scheutz (2001) state, deliberativeness is often realized by applying the concept of symbolic representation with compositional semantics (e.g., data tree) in all major functions, for an agent’s deliberation is not limited to presenting facts, but to construe hypotheses about possible future states, and in doing so, potentially offer information about the past. These hypothetical states involve goals, plans, partial solutions, hypothetical states of the agent’s beliefs, etc. On top of its symbolic representation, a deliberative agent has methods to interpret and predict the outside world in order to compare its state to the agent’s desired state (goal). On the basis of these interpretations and assumptions, it develops the best possible plan (from its point of view) and executes it. Intelligent planning is a complex process, especially if the resulting plan is comparatively sophisticated and spans a large exponentially growing solution space. During this planning time, the environment may change in a way that makes the execution of the actual plan (partially) obsolete or suboptimal. Thus, an immediate re-planning may be necessary. Vlahavas and Vrakas (2004) believe that deliberative agents are especially useful when a reasonable reaction to a sophisticated situation is required (however, not a real-time one), because of their ability to produce high-quality, domain-independent solutions.

While deliberative agents are comparatively flexible in acting upon their environment, they may, on the other hand, become considerably complex and grow slow in their reactions. The architecture and behavior of a reactive agent are simpler because the agent doesn’t have to deal with a representation of a symbolic world model, nor does it utilize complex symbolic reasoning. Instead, reactive behavior implies that the agent responds comparatively quickly to relevant stimuli from its environment. Based on this input, it produces output by simple situation-action associations, usually implemented via pattern matching. Reactive agents need few resources, and so can react much faster. On the negative side, they are not as flexible and dynamic as deliberative agents, and usually are not able to behave proactively. Nevertheless, Knight (1993) and other researchers believe the results of reactive agents are normally not (much) worse than the results of deliberative agents. In many cases, it may even be possible to replace one deliberative agent with several reactive ones without a loss of quality. This, however, seems to be more a reflection of the current state-of-the-art in deliberative agent concepts and their inherent complexity. In the future, it can be expected that deliberative agents will be much more powerful than reactive ones.

As a rule of thumb, it can be said that purely reactive agent systems can reveal little smartness, can hardly exhibit goal-directed behavior, and usually come with very limited learning capabilities. On the positive side, their implementation is relatively easy to achieve, their reaction to relevant real-world incidents can be extremely fast, and their explanation capabilities for their behavior usually work very well. Deliberative agents have their strengths where reactive ones have their weaknesses. Because they are based on general-purpose reasoning mechanisms, their behavior is neither fully explainable nor deterministic. The analysis of real-world incidents and their influence on the agent’s goals may need a lot of computing power, which results in slow reaction times. On the other hand, their behavior can be regarded as being comparatively smart and flexible. Moreover, in principle, deliberative agents can learn very well.

In reality, often MASs use agents that do not belong to one of the preceding extremes but realize an architecture somewhere in between. Such agents are called hybrid agents. The main idea is to structure the reasoning capabilities of a hybrid agent into two or more parts that interact with each other to achieve a coherent behavior of the agent as a whole. One part may produce a fast reaction of the agent, which is then fine-tuned by its deliberative capabilities. Whenever real-time requirements of the environment require it, intermediate planning results of the agent’s reasoning can be executed.

1.2.3 Multi-Agent Systems and Their Properties

Due to the limited capabilities of a single agent, more complex real-world problems require the common and cooperative effort of a number of agents in order to get the problem at hand solved. A multi-agent system (MAS) is a federation of fully or semi-autonomous problem solvers that join forces to work positively toward a symbiosis of their individual goals, as well as the overall goals of the federation or the involved set of agents. In order to succeed, they rely on communication, collaboration, negotiation, and responsibility delegation, all of which are based on the individual rationality and social intelligence of the involved agents (cf. Marík et al., 2002). The global/macro behavior of a MAS is defined by the emergent interactions among its agents, which implies that the capabilities of a MAS surpass those of each individual agent. The reduction of complexity is achieved by recursively decomposing a complex task into well-defined subtasks until each subtask can be dealt with by a single agent. However, unlike hardwired federations, a MAS may be highly dynamic and flexible. Depending on the organizational rules, agents may join or leave the coalition whenever they feel like it, provided their commitments are fulfilled. Such MASs are usually referred to as open MASs. In general, if agents can be heterogeneous in their structure and their communication skills and languages, and if they, nevertheless, live in an environment in which they can arbitrarily join and leave arbitrary institutions, such institutions are called open institutions, respectively open MASs. In order for such an environment with possibly many different types of institutions (with different rules and architectures) and heterogeneous agents to function, many issues need to be resolved, such as the heterogeneity of agents, the communication languages and behavior, trust and accountability, finding and joining issues of institutions, or exception handling in case of failures that may jeopardize the global operation of the system. In such an environment, standards are fundamental but cannot be assumed. In reality, most existing MASs are built with homogenous agents and may only support a restricted admission policy. Or as Dignum et al. (2008):1 state: “Currently, in practice, agents are designed so as to be able to operate exclusively with a single given institution, thus, basically defying the open nature of the institution.” Instead, homogenous MASs with often static structures, called closed MASs, still dominate most “real” agent-based applications.

Table 1.2 presents essential properties of (advanced) MASs.

1.2.4 Agent Communication

Agents need a means for communication in order to be able to cooperate. Communication can be direct or indirect.

Direct communication usually translates to an exchange of messages. Like letters, messages consist of an envelope and its actual contents. Still, the most important, yet slightly outdated, communication languages KQML (Knowledge Query and Manipulation Language, standardized by DARPA) and ACL (agent communication language, a Foundation for Intelligent Physical Agents’ (2014) (FIPA) standard, (cf. Dale (2005)) are based on so-called speech acts that were introduced by Searle (1975) and enhanced by Winograd and Flores (1986). Speech acts are defined by a set of performatives and their meanings, such as agree, propose, refuse, request, and query-if. A performative can be seen as an envelope that contains an enhanced set of syntactical information. Only in rare cases will the envelope cover the complete communication act (maybe when a refuse answer occurs). In most cases, the actual content of the communication—its semantic part—is contained within the envelope. While the performatives of a communication language are standardized, its actual content is not. The reason for this is that the envelope is syntax, while its content is a message that needs to be understood by the receiver. Computer systems still can only “understand” semantics in rare, well-defined and closely limited situations. Thus, the underlying content language is usually application-specific and heavily limited in its expressiveness.

Swarm intelligence approaches (see later) and, especially, the pheromone trails of ant colonies are known examples of simple but effective kinds of indirect communication. A different popular form of indirect communication between agents is the well-known concept of a blackboard on which agents can post their messages, which can then be read by other agents. Especially relevant in the e-business area is the contract net protocol. It is the digital version of the procedure that leads to a contract in normal life when, for example, a company asks suppliers to submit offers for a (public) announcement it made (task announcement, bidding, awarding, solution providing, and rewarding). Finally, agents may also be involved in electronic marketplaces and auctions, which is also a form of indirect communication through bidding.

1.2.6 Development Support for Agent-Based Systems

When it comes to the realization of agent-based systems, at least three principal support options are available: development methodologies, agent-oriented programming methodologies and languages, and development toolkits or frameworks.

1.2.7 MAS-Based Simulation Environments

A simulation studies the resource consumption, behavior, and output of a physical or conceptual system over time. Agent-based systems have always been an excellent candidate for the design and implementation of simulations in many application areas (cf. Uhrmacher and Weyns, 2009). The most obvious advantage is that they provide an intuitive and direct way for the modeling of a simulation study, because real-world entities can one by one directly be realized by agents in the simulation environment. This means that not only can real-world types be modeled, but, especially, non-typed real-world instances with individual behaviors. The big advantage here is that the coarse grained modeling level of many other simulation techniques—by only allowing types to be defined and their instances to be treated equally—is replaced by a much more flexible approach where entities may still inherit properties from a type but can act individually. Given that industrial systems usually rely on a heterarchy or a hierarchy, which means a set of components, this makes them ideal candidates for an agent-based simulation. Each component is represented by its own agent, and all the communication among, and intelligence of, the individual agents comes for free. Such a simulation has the advantage that, if the agents were modeled and implemented properly, the agents used in the simulation environment can directly be transferred and used in a real-world application. There are two approaches possible. On the one hand, an intense simulation study can be executed before the deployment of the industrial application. On the other hand, the simulation tool can be integrated into the application system in order to be used whenever a reliable prediction of the (future) behavior of the overall application system, or parts of it, is requested. An example of such an online simulation approach is presented in Cardin and Castagna (2009).

On the basis of such simulations, the behavior of a system can be tested extensively, which may lead to an improved control behavior, as well as to the establishment of substantial trust in its functioning, reliability, and flexibility. Additionally, envisioned extensions and adaptations of the system can be tested beforehand. In the meantime, a number of agent-based simulation tools are on the market. Table 1.6 lists some of them (freeware is in italics; for the homepage, see the Reference column).

The Multi-agent-based Simulation Workshops and Book Series (2014) are the most relevant events and publications on this topic and provide excellent insights. Good overview papers about agent-based simulation tools are Zhou et al. (2007), Michel et al. (2009), Theodoropoulos et al. (2009), Troitzsch (2009), and Allan (2010).

On the one hand, intelligent agents are goal-directed problem solvers that solve autonomously and often proactively tasks and problems for their clients. On the other hand, in order to be able to act like that, they need to interoperate with other agents, and maybe human beings. As discussed in Section 1.2.2, in order to do so, intelligent agents need to rely on an abstract model of their environment that allows them to reason about relevant changes in their environment and define their reaction to them. Ontologies, respectively ontology languages, are an appropriate means to develop the foundation for such a model. They became very popular within the Semantic Web and with service-oriented architecture (SOA). In the meantime, they also play a profound role in agent-based systems (cf. Runde and Fay, 2011).

One of the biggest challenges in problem solving by MASs is the autonomous (recursive) decomposition of an assigned complex task into appropriate subtasks. This, especially, implies that the involved agents can communicate with each other on a semantically meaningful level, and as a group have enough common knowledge and reasoning capabilities to understand what they are doing on the macro-level. However, because cooperating agents are usually specialists in different fields of expertise, their vocabulary and knowledge may overlap only partially (or not at all) and may not be consistent (due to homonyms or synonyms in their vocabulary or different interpretations of real-world conditions). In such cases, ontologies can help. An ontology is a formal, machine-processable taxonomy of an underlying domain (cf. Gruber, 1993; Sycara and Paolucci, 2004). As such, it contains all relevant entities or concepts within that domain, the underlying axioms and properties, the relationships between them and the constraints on their logically consistent application (i.e., it defines a domain of knowledge or discourse by a (shared) vocabulary and, by that, permits reasoning about its properties). Ontologies are typically specified in languages that allow abstractions and expressions of semantics (e.g., first-order logic languages). If agents are steered in their behavior by their underlying ontologies, these ontologies need to be merged, or at least synchronized, if two agents are supposed to cooperate in order to provide a common fundament for a meaningful conversation and cooperation between these agents (cf. Stumme and Mädche, 2001). Unfortunately, in reality an overwhelming, steadily growing number of ontologies for the same or overlapping areas exist. Despite its partial or complete overlap, their underlying terminology may vary substantially (e.g., because they model the same domain on a different level of abstraction or have a different overall view of it). Under these circumstances, ontology merging can become quite difficult. Problems such as homonyms, synonyms, different levels of abstractions, and possible contradictions in class and instance descriptions, axioms, and policies of the underlying ontologies need to be resolved, which, in the general case, is not yet possible. Thus, a sufficient merger of ontologies may often not be possible. In general, a merger or an interoperation can only be achieved if the underlying ontologies do overlap in a sufficient way. This implies that agents can only cooperate if their expertise overlaps sufficiently.

One of the most prominent and powerful examples for an ontology language is the Web Ontology Language (cf. OWL, 2014). OWL provides an RDF/RDFS extension based on description logics which permits describing concepts and instances in the real world. More specifically, it is a subset of first-order logics and permits to describe three different types of objects in a domain, namely classes that describe the characteristics of relevant entities (called concepts) in the domain, individuals, which are concepts/objects/instances in the domain, and properties, which define relationships between objects/concepts. Properties are integrity constraints and axioms on the class, as well as the object instance level. They define, among others, transitivity, symmetry, or inverse functions, as well as cardinality or type restrictions. Moreover, due to the underlying logic, OWL provides automatic reasoning capabilities that permit it to infer information that is not explicitly represented in the underlying ontology. By this, the check for consistency of concept/object definitions, subsumption testing, the completion of concept definitions, the classification of new instances and concepts, and the extraction of implicit knowledge are realized. Although OWL is comparatively powerful, the underlying description logic also exposes it to some weaknesses: Description logic only permits the expression of static snapshots of the real world, not the expression of state transitions. Thus, processes, especially, but also workflows and the interaction among agents within a task execution, cannot be modeled appropriately.

OWL-POLAR was invented in order to support OWL-based knowledge representation and reasoning on policies (cf. Sensoy et al., 2010). Policies, respectively norms, are machine-understandable declarations of constraints and rules on the overall global behavior within a distributed system, respectively MAS. In OWL-POLAR, a policy comes with activation and expiration conditions, possible obligations, the policy addressee, and possible actions. It is activated when its activation conditions hold, even if its expiration conditions do not. OWL-POLAR provides a reasonable foundation for the merger of ontologies.

An extension of OWL toward the definition of complex macro-services and their interoperability through loose coupling is OWL-S (cf. Martin et al., 2005; Sycara, 2006; Martin et al., 2014). It provides the foundation for the construction of complex web service profiles, process models, and service grounding. A service profile consists of preconditions which are a set of conditions that need to be fulfilled prior to a service invocation, input parameters, which are a set of necessary inputs that the requester is supposed to provide to invoke the service, output parameters, which are the definition of the results that the requester expects to be delivered after the service was executed, and effects, which are the set of consequences that need to hold true if the service was invoked successfully. Additionally, a service description is created that provides the nonfunctional properties of the service, such as provenance, quality of service, security issues, policy issues, or domain-specific characteristics. A process model specifies the workflow that coordinates the execution of the basic processes involved in a complex task execution. Vaculin and Sycara (2007) propose the necessary extensions for monitoring and error handling during the execution of a complex task. Finally, the task of service grounding is to map the complex task at hand on an adequate WSDL file.

1.3.2 Self-Organization and Emergence

In order to function, the agents of a MAS need to have a common basis and have to follow common rules. Self-organizing respectively emergent systems define such rules that partially overlap with the general characteristics of MAS. Thus, it is no surprise that MAS may be organized according to the rules of self-organizing respectively emergent systems.

1.3.3 Swarm Intelligence and Stigmergy

Swarm intelligence (cf., e.g., IEEE, 2014; Dorigo et al., 2004, 2006; Panigrahi et al., 2011) as an innovative distributed intelligence approach for the optimization problems, as well as specific kinds of general problem solving relies on the ideas of emergence. It uses social swarming behaviors observed in nature as a blueprint to design complex emergent systems. Depending on the underlying concept in nature, such as bird flocks, bee swarms, or ant colonies, different categories of swarm intelligence systems can be identified. Ants and ant colonies, as the most popular technique, will be discussed here in some detail.

The so-called foraging process stands in the center of the behavior of ant colonies. Foraging relies on a few very simple behavioral and collaborative patterns. Communication is indirect by using chemical substances known as pheromones. When successfully returning from a food source, ants drop pheromones, thus slowly creating a well-identifiable pheromone trail to the food source. Because this scent decays over time through the process of evaporation, the pheromone trail will slowly disappear if not renewed by successfully returning ants. In order to solve their need for food as efficiently as possible, ants actually have to deal with the shortest path problem. Altogether, the foraging process of ants exhibits the following characteristics (cf. Valckenaers et al., 2001, 2003):

Simple structure: Structure can be discussed on the level of the ant as well as the ant colony. An ant represents a comparatively simple creature without representation of a real-world model or sophisticated behavior. It does not need to be more sophisticated because the complexity of the solution process is kept away from the single ant. Instead, it is dealt with implicitly on the level of the overall system. For example, an ant does not need a mental map of the environment because it either walks randomly in search of a new food source or is guided by pheromones. Moreover, the evaporation and refreshing of the pheromone trails allows the ants to cope with the dynamics of the environment. Thus, they do not need to be explicitly informed about the current state of reality (e.g., by learning about the exhaustion of an old food source or the spotting of a new one). Ants exhibit only a few simple patterns of behavior. For example, evaporation and refreshment are simple mechanisms to limit the inertia of information that gets accumulated over time and is dealt with as part of their patterns.

Reinforcement learning: Once an ant finds a food source, it drops pheromones on its food-packed way back to its nest. Through this, a scent trail between the ant colony and the food source is created. Whether this trail will be populated or not depends on the behavior/interest of the other ants. Each ant that uses the trail deposits fresh pheromones and, by this, reinforces the information related to this food source. If the trail is not used, or less frequently used for a while (e.g., because there are closer food sources that require less time and will to reach, or the food source is exhausted), it will eventually evaporate. Because many ants go out from their colony in search of food, the ants that return first are presumably those that have visited the food source with the shortest path to the colony. This allows an ant colony to identify the shortest, and respectively the fastest, path to a food source (cf. Bonabeau et al., 1999; Fleischer, 2003).

Stigmergy: The indirect and asynchronous interaction and information exchange between agents by pheromones, mediated by an active environment that changes constantly due to new and diminishing trails, is called stigmergy. Stigmergic agent architectures are reactive and environment-driven. Distinguishing features of this approach are:

• Agents are simple and reactive; they are unaware of other agents or of the emerging complex activities of the agent society.

• The environment is an important mechanism to guide activities of these agents and to serve as an up-to-date information storage and retrieval system for the agent society.

Emergent behavior: The foraging process of ants illustrates the emergence of coordinated behavior at the level of the ant society, although for the individuals themselves it is neither relevant nor visible whether they are taking part in a concerted activity. Ants follow their own simple agenda. Their behavior is only influenced by the permanently changing environment.

Probabilistic behavior: The behavior of an ant is always probabilistic: Whenever it finds a pheromone trail, it will most likely follow it. However, there is no certainty of that because the final reaction mainly depends on the perceived strength of the pheromones and some randomness in the behavior of ants.

There are a number of overlaps between swarm intelligence concepts and those of MASs. A swarm intelligence entity can be regarded as a reactive agent with very limited or even no intelligence. Even in the category of reactive agents this entity can be regarded as being very simple without any direct communication skills. On the other hand, a swarm intelligence approach usually requires a huge number of underlying entities, many more than with normal MASs. Thus, although most agent development tools and frameworks only permit the modeling of simple agents, their capabilities exceed the requirements of swarm intelligence systems on functionality and capabilities by far. However, they may not be able to fulfill the requirements on the number of entities that are to be created and supported. This is why swarm intelligence applications usually do not rely on agent development platforms or tools, but instead use their own. Good application examples for swarm intelligence can be found in Bogue (2008), Chen et al. (2008), Leitão (2008, 2009), Barbosa and Leitão (2010), Leitão et al. (2012), Fornarelli and Mescia (2013), and Springer (2014).