Practical Artificial Intelligence Machine Learning, Bots, and Agent Solutions Using C# — Arnaldo Pérez Castaño

www.allitebooks.com

Practical Artificial Intelligence Machine Learning, Bots, and Agent Solutions Using C#

ArnaldoPérezCastaño

www.allitebooks.com

Practical Artificial Intelligence ArnaldoPérezCastaño Havana, Cuba ISBN-13 (pbk): 978-1-4842-3356-6 https://doi.org/10.1007/978-1-4842-3357-3

ISBN-13 (electronic): 978-1-4842-3357-3

Library of Congress Control Number: 2018943123

Copyright © 2018 by Arnaldo Pérez Castaño This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. Trademarked names, logos, and images may appear in this book. Rather than use a trademark symbol with every occurrence of a trademarked name, logo, or image we use the names, logos, and images only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark. The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights. While the advice and information in this book are believed to be true and accurate at the date of publication, neither the authors nor the editors nor the publisher can accept any legal responsibility for any errors or omissions that may be made. The publisher makes no warranty, express or implied, with respect to the material contained herein. Managing Director, Apress Media LLC: Welmoed Spahr Acquisitions Editor: Natalie Pao Development Editor: James Markham Coordinating Editor: Jessica Vakili Cover designed by eStudioCalamar Cover image designed by Freepik (www.freepik.com) Distributed to the book trade worldwide by Springer Science+Business Media NewYork, 233 Spring Street, 6th Floor, NewYork, NY 10013. Phone 1-800-SPRINGER, fax (201) 348-4505, email [emailprotected], or visit www.springeronline.com. Apress Media, LLC is a California LLC and the sole member (owner) is Springer Science + Business Media Finance Inc (SSBM Finance Inc). SSBM Finance Inc is a Delaware corporation. For information on translations, please email [emailprotected], or visit http://www.apress.com/ rights-permissions. Apress titles may be purchased in bulk for academic, corporate, or promotional use. eBook versions and licenses are also available for most titles. For more information, reference our Print and eBook Bulk Sales web page at http://www.apress.com/bulk-sales. Any source code or other supplementary material referenced by the author in this book is available to readers on GitHub via the book’s product page, located at www.apress.com/9781484233566. For more detailed information, please visit http://www.apress.com/source-code. Printed on acid-free paper

www.allitebooks.com

To ML, thanks for the theater and the lovely moments To my mother, my father, my brother, my grandma, and my entire family, thanks for your immense support

www.allitebooks.com

Table of Contents About the Author�������������������������������������������������������������������������������xiii About the Technical Reviewer������������������������������������������������������������xv Acknowledgments����������������������������������������������������������������������������xvii Introduction���������������������������������������������������������������������������������������xix Chapter 1: Logic & AI����������������������������������������������������������������������������1 What Is Logic?�������������������������������������������������������������������������������������������������������2 Propositional Logic������������������������������������������������������������������������������������������������3 Logical Connectives����������������������������������������������������������������������������������������������6 Negation����������������������������������������������������������������������������������������������������������7 Conjunction������������������������������������������������������������������������������������������������������8 Disjunction�������������������������������������������������������������������������������������������������������9 Implication�����������������������������������������������������������������������������������������������������10 Equivalence���������������������������������������������������������������������������������������������������11 Laws ofPropositional Logic��������������������������������������������������������������������������������12 Normal Forms�����������������������������������������������������������������������������������������������������16 Logic Circuits������������������������������������������������������������������������������������������������������17 Practical Problem: Using Inheritance andC# Operators toEvaluate Logic Formulas���������������������������������������������������������������������������������������������������21 Practical Problem: Representing Logic Formulas asBinary Decision Trees�����������������������������������������������������������������������������������������������������26 Practical Problem: Transforming aFormula intoNegation Normal Form (NNF)���������������������������������������������������������������������������������������������31 v

www.allitebooks.com

Table of Contents

Practical Problem: Transforming aFormula intoConjunctive Normal Form (CNF)���������������������������������������������������������������������������������������������36 Summary������������������������������������������������������������������������������������������������������������40

Chapter 2: Automated Theorem Proving & First-Order Logic�������������41 Automated Theorem Proving�������������������������������������������������������������������������������42 Practical Problem: Clauses andCNFs Classes inC#�������������������������������������������45 DPLL Algorithm���������������������������������������������������������������������������������������������������55 Practical Problem: Modeling thePigeonhole Principle in Propositional Logic����������������������������������������������������������������������������������������������67 Practical Problem: Finding Whether aPropositional Logic Formula is SAT�������������68 First-Order Logic�������������������������������������������������������������������������������������������������75 Predicates inC#��������������������������������������������������������������������������������������������80 Practical Problem: Cleaning Robot����������������������������������������������������������������������82 Summary������������������������������������������������������������������������������������������������������������89

Chapter 3: Agents�������������������������������������������������������������������������������91 What’s anAgent?������������������������������������������������������������������������������������������������92 Agent Properties�������������������������������������������������������������������������������������������������95 Types ofEnvironments����������������������������������������������������������������������������������������99 Agents withState����������������������������������������������������������������������������������������������102 Practical Problem: Modeling theCleaning Robot asanAgent andAdding State toIt����������������������������������������������������������������������������������������103 Agent Architectures�������������������������������������������������������������������������������������������113 Reactive Architectures: Subsumption Architecture�������������������������������������114 Deliberative Architectures: BDI Architecture������������������������������������������������119 Hybrid Architectures������������������������������������������������������������������������������������127 Touring Machines����������������������������������������������������������������������������������������131 InteRRaP������������������������������������������������������������������������������������������������������133 Summary����������������������������������������������������������������������������������������������������������135 vi

Table of Contents

Chapter 4: Mars Rover����������������������������������������������������������������������137 What’s aMars Rover?���������������������������������������������������������������������������������������138 Mars Rover Architecture�����������������������������������������������������������������������������������140 Mars Rover Code�����������������������������������������������������������������������������������������������143 Mars Rover Visual Application���������������������������������������������������������������������������176 Summary����������������������������������������������������������������������������������������������������������192

Chapter 5: Multi-Agent Systems�������������������������������������������������������193 What’s aMulti-Agent System?��������������������������������������������������������������������������194 Multi-Agent Organization����������������������������������������������������������������������������������197 Communication�������������������������������������������������������������������������������������������������199 Speech Act Theory���������������������������������������������������������������������������������������201 Agent Communication Languages (ACL)�����������������������������������������������������204 Coordination & Cooperation������������������������������������������������������������������������������211 Negotiation Using Contract Net�������������������������������������������������������������������215 Social Norms & Societies����������������������������������������������������������������������������218 Summary����������������������������������������������������������������������������������������������������������220

Chapter 6: Communication inaMulti-Agent System Using WCF������221 Services������������������������������������������������������������������������������������������������������������222 Contracts�����������������������������������������������������������������������������������������������������������224 Bindings������������������������������������������������������������������������������������������������������������227 Endpoints����������������������������������������������������������������������������������������������������������229 Publisher/Subscriber Pattern����������������������������������������������������������������������������230 Practical Problem: Communicating AmongMultiple Agents Using WCF�����������231 Summary����������������������������������������������������������������������������������������������������������248

vii

Table of Contents

Chapter 7: Cleaning Agents: AMulti-Agent System Problem�����������249 Program Structure��������������������������������������������������������������������������������������������250 Cleaning Task����������������������������������������������������������������������������������������������������251 Cleaning Agent Platform�����������������������������������������������������������������������������������254 Contract Net������������������������������������������������������������������������������������������������������256 FIPA-ACL�����������������������������������������������������������������������������������������������������������262 MAS Cleaning Agent������������������������������������������������������������������������������������������267 GUI���������������������������������������������������������������������������������������������������������������������280 Running theApplication������������������������������������������������������������������������������������283 Summary����������������������������������������������������������������������������������������������������������288

Chapter 8: Simulation�����������������������������������������������������������������������289 What Is Simulation?������������������������������������������������������������������������������������������290 Discrete-Event Simulation��������������������������������������������������������������������������������292 Probabilistic Distributions���������������������������������������������������������������������������������294 Practical Problem: Airport Simulation���������������������������������������������������������������297 Summary����������������������������������������������������������������������������������������������������������313

Chapter 9: Support Vector Machines������������������������������������������������315 What Is aSupport Vector Machine (SVM)?��������������������������������������������������������318 Practical Problem: Linear SVM inC#�����������������������������������������������������������������328 Imperfect Separation����������������������������������������������������������������������������������������343 Non-linearly Separable Case: Kernel Trick��������������������������������������������������������345 Sequential Minimal Optimization Algorithm (SMO)�������������������������������������������348 Practical Problem: SMO Implementation����������������������������������������������������������356 Summary����������������������������������������������������������������������������������������������������������365

viii

Table of Contents

Chapter 10: Decision Trees���������������������������������������������������������������367 What Is aDecision Tree?�����������������������������������������������������������������������������������368 Generating aDecision Tree: ID3 Algorithm��������������������������������������������������������372 Entropy andInformation Gain����������������������������������������������������������������������375 Practical Problem: Implementing theID3 Algorithm������������������������������������377 C4.5 Algorithm���������������������������������������������������������������������������������������������393 Practical Problem: Implementing theC4.5 Algorithm����������������������������������399 Summary����������������������������������������������������������������������������������������������������������410

Chapter 11: Neural Networks�����������������������������������������������������������411 What Is aNeural Network?�������������������������������������������������������������������������������412 Perceptron: Singular NN������������������������������������������������������������������������������������415 Practical Problem: Implementing thePerceptron NN����������������������������������420 Adaline & Gradient Descent Search�������������������������������������������������������������427 Stochastic Approximation����������������������������������������������������������������������������431 Practical Problem: Implementing Adaline NN����������������������������������������������432 Multi-layer Networks�����������������������������������������������������������������������������������435 Backpropagation Algorithm�������������������������������������������������������������������������440 Practical Problem: Implementing Backpropagation & Solving theXOR Problem�����������������������������������������������������������������������������446 Summary����������������������������������������������������������������������������������������������������������459

Chapter 12: Handwritten Digit Recognition��������������������������������������461 What Is Handwritten Digit Recognition?�����������������������������������������������������������462 Training Data Set�����������������������������������������������������������������������������������������������464 Multi-layer NN forHDR��������������������������������������������������������������������������������������464

ix

Table of Contents

Implementation�������������������������������������������������������������������������������������������������467 Testing��������������������������������������������������������������������������������������������������������������476 Summary����������������������������������������������������������������������������������������������������������478

Chapter 13: Clustering & Multi-objective Clustering������������������������479 What Is Clustering?�������������������������������������������������������������������������������������������480 Hierarchical Clustering��������������������������������������������������������������������������������������484 Partitional Clustering����������������������������������������������������������������������������������������486 Practical Problem: K-Means Algorithm�������������������������������������������������������������490 Multi-objective Clustering���������������������������������������������������������������������������������499 Pareto Frontier Builder��������������������������������������������������������������������������������������501 Summary����������������������������������������������������������������������������������������������������������507

Chapter 14: Heuristics & Metaheuristics������������������������������������������509 What Is aHeuristic?������������������������������������������������������������������������������������������510 Hill Climbing������������������������������������������������������������������������������������������������������512 Practical Problem: Implementing Hill Climbing�������������������������������������������������515 P-Metaheuristics: Genetic Algorithms���������������������������������������������������������������522 Practical Problem: Implementing aGenetic Algorithm fortheTraveling Salesman Problem�����������������������������������������������������������������526 S-Metaheuristics: Tabu Search�������������������������������������������������������������������������538 Summary����������������������������������������������������������������������������������������������������������548

Chapter 15: Game Programming������������������������������������������������������549 What Is aVideo Game?�������������������������������������������������������������������������������������551 Searching inGames������������������������������������������������������������������������������������������553 Uninformed Search�������������������������������������������������������������������������������������������556 Practical Problem: Implementing BFS, DFS, DLS, andIDS��������������������������������560

x

Table of Contents

Practical Problem: Implementing Bidirectional Search ontheSliding Tiles Puzzle��������������������������������������������������������������������������������568 Informed Search�����������������������������������������������������������������������������������������������580 A* fortheSliding Tiles Puzzle���������������������������������������������������������������������������583 Summary����������������������������������������������������������������������������������������������������������588

Chapter 16: Game Theory: Adversarial Search & Othello Game�������589 What Is Game Theory?��������������������������������������������������������������������������������������590 Adversarial Search��������������������������������������������������������������������������������������������593 Minimax Search Algorithm��������������������������������������������������������������������������������596 Alpha-Beta Pruning�������������������������������������������������������������������������������������������599 Othello Game�����������������������������������������������������������������������������������������������������602 Practical Problem: Implementing theOthello Game inWindows Forms�����������607 Practical Problem: Implementing theOthello Game AI Using Minimax�������������628 Summary����������������������������������������������������������������������������������������������������������631

Chapter 17: Reinforcement Learning������������������������������������������������633 What Is Reinforcement Learning?���������������������������������������������������������������������634 Markov Decision Process����������������������������������������������������������������������������������636 Value/Action–Value Functions & Policies����������������������������������������������������������640 Value Iteration Algorithm�����������������������������������������������������������������������������������644 Policy Iteration Algorithm����������������������������������������������������������������������������������646 Q-Learning & Temporal Difference��������������������������������������������������������������������647 Practical Problem: Solving aMaze Using Q-Learning���������������������������������������650 Summary����������������������������������������������������������������������������������������������������������668

Index�������������������������������������������������������������������������������������������������669

xi

About the Author ArnaldoPérezCastañois a computer scientist based in Havana, Cuba. He’s the author of PrestaShop Recipes (Apress, 2017) and a series of programming books—JavaScript Fácil, HTML y CSS Fácil, and Python Fácil (Marcombo S.A.)—and writes AI-related articles for MSDN Magazine, VisualStudio Magazine.com, and Smashing Magazine. He is one of the co-founders of Cuba Mania Tour (http://www.cubamaniatour.com). His expertise includes Java, VB, Python, algorithms, optimization, Matlab, C#, .NET Framework, and artificial intelligence. Arnaldo offers his services through freelancer.com and served as reviewer for the Journal of Mathematical Modelling and Algorithms in Operations Research. Cinema and music are some of his passions. Many of his colleagues around the world call him “Scientist of the Caribbean.” He can be reached at [emailprotected].

xiii

About the Technical Reviewer JamesMcCaffreyworks in the Machine Learning Group at Microsoft Research in Redmond, WA.James has a Ph.D. in cognitive psychology and computational statistics from the University of Southern California, a BA in psychology, a BA in applied mathematics, and an MS in computer science. James is a frequent speaker at developer conferences. James learned to speak to the public while working at Disneyland as a college student, and he can still recite the entire Jungle Cruise ride narration from memory.

xv

Acknowledgments First of all, a big thank you to Dr. James McCaffrey from Microsoft Research in Redmond, WA, who kindly accepted the role of technical reviewer of this book. I e-met James when writing articles for MSDN Magazine. His comments at that time were always very useful, and they continued to be extremely useful throughout the review process of this book. I must also thank James for his patience because what it was supposed to be a nine-chapter book eventually became a seventeen-chapter book, and he stood up with us along the way. Another thank you must go to my editors, Pao Natalie and Jessica Vakili, who were also very patient and understanding during the writing process. Finally, I would like to acknowledge all researchers on AI/machine learning out there who day after day try to push this very important field of science forward with new advancements, techniques, and ideas. Thank you, all!

xvii

Introduction Practical Artificial Intelligence (PAI) is a book that proposes a new model for learning. Most AI books deeply focus on theory and abandon practical problems that demonstrate the theory introduced throughout the book. In PAI we propose a model that follows Benjamin Franklin’s (Founding Father of the United States of America) ideas: “Tell me and I forget. Teach me and I remember. Involve me and I learn.” Therefore, PAI includes theoretical knowledge but guarantees that at least one fully coded (C#) practical problem is included in every chapter as a way to allow readers to better understand and as a way to get them involved with the theoretical concepts and ideas introduced during the chapter. These practical problems can be executed by readers using the code associated with this book and should give them a better insight into the concepts herein described. Explanations and definitions included in PAI are intended to be as simple as they can be (not putting aside the fact that they belong to a mathematical, scientific environment) so readers from different backgrounds can engage with the content and understand it using minimal mathematical or programming knowledge. Chapters 1 and 2 explore logic as a fundamental founding block of many sciences, like mathematics or computer science. In these chapters, we will describe propositional logic, first-order logic, and automated theorem proving; related practical problems coded in C# will be presented. Throughout chapters 3–7, we will focus on agents and multi-agent systems. We’ll dive into the different types of agents and their architectures, then we’ll present a big practical problem where we’ll code a Mars Rover whose task is to find water on Mars. We’ll include another

xix

Introduction

practical problem where we set up a group of agents to communicate using Windows Communication Foundation (WCF), and finally, we’ll end this part of the book by presenting another practical problem (Chapter 7) where a group of agents forming a multi-agent system will collaborate and communicate to clean a room of its dirt. Chapter 8 will describe a sub-field of AI known as simulation, where by using statistical, probabilistic tools we simulate a scenario of real life. In this case, we’ll simulate the functioning of an airport, with airplanes arriving at and departing from the airport during a certain period of time. Chapters 9–12 will be dedicated to supervised learning, a very important paradigm of machine learning where we basically teach a machine (program) to do something (usually classify data) by presenting it with many samples of pairs , where data could be anything; it could be animals, houses, people, and so on. For instance, a sample set could be , , and so forth. Clearly, for the machine to be able to understand and process any data we must input numerical values instead of text. Throughout these chapters we will explore support vector machines, decision trees, neural networks, and handwritten digit recognition. Chapter 13 will explain another very important paradigm of machine learning, namely unsupervised learning. In unsupervised learning we learn the structure of the data received as input, and there are no labels (classifications) as occurred in supervised learning; in other words, samples are simply , and no classification is included. Thus, an unsupervised learning program learns without any external help and by looking only at the information provided by the data itself. In this chapter, we will describe clustering, a classic unsupervised learning technique. We will also describe multi-objetive clustering and multi-objective optimization. A method for constructing the Pareto Frontier, namely Pareto Frontier Builder, proposed by the author, will be included in this chapter.

xx

Introduction

Chapter 14 will focus on heuristics and metaheuristics, a topic we will be mentioning in previous chapters and will finally be studied here. We will describe mainly two metaheuristics: genetic algorithms and tabu search, which are two of the main representatives of the broadest classes of metaheuristics, which are population-based metaheuristics and single solution–based metaheuristics. Chapter 15 will explore the world of game programming, specifically games where executing a search is necessary. Many of the popular search algorithms will be detailed and implemented. A practical problem where we design and code a sliding tiles puzzle agent will also be included. Chapter 16 will dive into game theory, in particular a sub-field of it known as adversarial search. In this field, we will study the Minimax algorithm and implement an Othello agent that plays using this strategy (Minimax). Chapter 17 will describe a machine-learning paradigm that nowadays is considered the future of artificial intelligence; this paradigm is reinforcement learning. In reinforcement learning, agents learn through rewards and punishment; they learn over time like humans do, and when the learning process is long enough they can achieve highly competitive levels in a game, up to the point of beating a human world champion (as occurred with backgammon and Go).

xxi

CHAPTER 1

Logic & AI In this chapter, we’ll introduce a topic that is vital not only to the world of artificial intelligence (AI) but also to many other areas of knowledge, such as mathematics, physics, medicine, philosophy, and so on. It has been deeply studied and formalized since ancient times by great philosophers like Aristotle, Euclid, and Plato and by some of the greatest mathematicians of all time. Born in the early ages of mankind, it represents a basic tool that allowed science to flourish up to the point where it is today. It clarifies and straightens our complicated human minds and brings order to our sometimes disordered thoughts. Logic, this matter to which we have been referring thus far, will be the main focus of this chapter. We’ll be explaining some of its fundamental notions, concepts, and branches, as well as its relation to computer science and AI.This subject is fundamental to understanding many of the concepts that will be addressed throughout this book. Furthermore, how can we create a decent artificial intelligence without logic? Logic directs rationality in our mind; therefore, how can we create an artificial version of our mind if we bypass that extremely important element (logic) that is present in our “natural” intelligence and dictates decisions in many cases—or, to be precise, rational decisions. Propositional logic; first-order logic; practical problems where we’ll learn how to create a logic framework, how to solve the SAT (satisfiability) problem using an outstanding algorithm called DPLL, and how to code a first, simple, naive cleaning robot using first-order logic components— these topics will get us started in this book. © Arnaldo Pérez Castaño 2018 A. Pérez Castaño, Practical Artificial Intelligence, https://doi.org/10.1007/978-1-4842-3357-3_1

1

Chapter 1

Logic & AI

Note Logic can be branched into mathematical logic, philosophical logic, computational logic, Boolean logic, fuzzy logic, quantum logic, and so forth. In this book, we will be dealing with computational logic, the field related to those areas of computer science and logic that necessarily overlap.

What Is Logic? Intuitively we all have a notion of what logic is and how useful it can be in our daily lives. Despite this common sense or cultural concept of logic, surprisingly there is, in the scientific community, no formal or global definition (as of today) of what logic is. In seeking a definition from its founding fathers, we could go back in time to its roots and discover that the word logic actually derives from the Ancient Greek logike, which translates as “concept, idea, or thought.” Some theorists have defined logic as “the science of thought.” Even though this definition appears to be a decent approximation of what we typically associate with logic, it’s not a very accurate definition because logic is not the only science related to the study of thoughts and reasoning. The reality is that this subject is so deeply ingrained at the foundation of all other sciences that it’s hard to provide a formal definition for it. In this book, we’ll think of logic as a way to formalize human reasoning. Since computational logic is the branch of logic that relates to computer science, we’ll be describing some important notions on this subject. Ultimately, the concepts described here will be useful throughout this book and in every practical problem to be presented.

2

Chapter 1

Logic & AI

Note Logic is used extensively in computer science: at the processor level by means of logical gates, in hardware and software verification such as floating-point arithmetic, in high-level programming like constraint programming, and in artificial intelligence for problems such as planning, scheduling, agents control, and so forth.

Propositional Logic In daily life and during our human communication process, we constantly listen to expressions of the language that possess a certain meaning; among these we can find the propositions. Propositions are statements that can be classified according to their veracity (True or 1, False or 0, etc.) or according to their modality (probable, impossible, necessary, etc.). Every proposition expresses a certain thought that represents its meaning and content. Because of the wide variety of expressions in our language, they can be classified as narratives, exclamatory, questioning, and so forth. In this book, we’ll focus on the first type of proposition, narratives, which are expressions of judgment, and we’ll simply call them propositions from this point on. The following list presents a few examples of propositions: 1. “Smoking damages your health.” 2. “Michael Jordan is the greatest basketball player of all time.” 3. “Jazz is the coolest musical genre in the world.” 4. “100 is greater than 1.” 5. “There are wonderful beaches in Havana.”

3

Chapter 1

Logic & AI

6. “World War II ended in 1945.” 7. “I listen to Sting’s music.” 8. “I will read poems from Spanish poet Rafael Alberti.” These are simple or atomic propositions that we can use in any ordinary day during any ordinary conversation. In order to add complexity and transform them into something a bit more meaningful we can rely on compound propositions, which are obtained by means of logical connectors linking simple propositions like the ones previously listed. Hence, from the propositions just listed we could obtain the following (not necessarily correct or meaningful) compound propositions. 1. “There are NOT wonderful beaches in Havana.” 2. “Smoking damages your health AND 100 is greater than 1.” 3. “Michael Jordan is the greatest basketball player of all time OR World War II ended in 1945.” 4. “IF Jazz is the coolest musical genre in the world THEN I listen to Sting’s music.” 5. “I will read poems from Spanish poet Rafael Alberti IF AND ONLY IF 100 is greater than 1.” Logical connectives in these cases are shown in capital letters and are represented by the words or phrases “NOT”, “AND”, “OR”, “IF …THEN” and “IF AND ONLY IF”. Simple or atomic propositions are denoted using letters (p, q, r, etc.) known as propositional variables. We could name some of the preceding propositions as follows: 1. p = “Smoking damages your health.” 2. q = “Michael Jordan is the greatest basketball player of all time.” 4

Chapter 1

Logic & AI

3. r = “Jazz is the coolest musical genre in the world.” 4. s = “100 is greater than 1.” A proposition that can be either True (1) or False (0) depending on the truth value of the propositions that compose it is known as a formula. Note that a formula can be simple; in other words, it can be composed of a single proposition. Consequently, every proposition is considered a formula. The syntax of propositional logic is governed by the following rules: 1. All variables and propositional constants (True, False) are formulas. 2. If F is a formula then NOT F is also a formula. 3. If F, G are formulas then F AND G, F OR G, F => G, F G also represent formulas. An interpretation of a formula F is an assignation of truth values for every propositional variable that occurs in F and determines a truth value for F.Since every variable always has two possible values (True, False or 1, 0) then the total number of interpretations for F is 2n where n is the total number of variables occurring in F. A proposition that is True for every interpretation is said to be a tautology or logic law. A proposition that is False for every interpretation is said to be a contradiction or unsatisfiable. We’ll be interested in studying the truth values of combined propositions and how to compute them. In the Satisfiability problem, we receive as input a formula, usually in a special, standardized form known as Conjunctive Normal Form (soon to be detailed), and we’ll try to assign truth values for its atomic propositions so the formula becomes True (1); if such assignment exists, we say that the formula is Satisfiable. This is a classic problem in computer science and will be addressed throughout this chapter.

5

Chapter 1

Logic & AI

In the next section, we’ll take a closer look at logical connectives, as they are determinant in establishing the final truth value of a formula.

Logical Connectives Commonly, logical connectives are represented using the following symbols: •

¬ denotes negation (“NOT”)

•

∧ denotes conjunction (“AND”)

•

∨ denotes disjunction (“OR”)

•

=> denotes implication (“IF … THEN”)

•

denotes double implication or equivalence (“IF AND ONLY IF”)

Logical connectives act as unary or binary (receive one or two arguments) functions that provide an output that can be either 1 (True) or 0 (False). In order to better understand what the output would be for every connective and every possible input, we rely on truth tables.

Note The tilde symbol (~) is also used to indicate negation. In a truth table, columns correspond to variables and outputs and rows correspond to every possible combination of values for each propositional variable. We’ll see detailed truth tables for every connective in the following subsections.

6

Chapter 1

Logic & AI

N egation If we have a proposition p then its negation is denoted ¬p (read Not p). This is a unary logical connective because it requires a single proposition as input. Let’s try to negate some of the propositions previously presented: 1. “Smoking DOES NOT damage your health.” 2. “Michael Jordan is NOT the greatest basketball player of all time.” 3. “Jazz is NOT the coolest musical genre in the world.” 4. “100 is NOT greater than 1.” 5. “There are NOT wonderful beaches in Havana.” 6. “World War II DID NOT end in 1945.” The truth table for the negation connective is the following (Table1-1).

Table 1-1. Truth Table for Negation Logical Connective p

¬p

1

1

From Table1-1 we can see that if a proposition p is True (1) then its negation (¬p) is False (0), and vice versa if the proposition is False.

7

Chapter 1

Logic & AI

C onjunction If we have propositions p, q then their conjunction is denoted p ∧ q (read p AND q). This is a binary logical connective; it requires two propositions as input. The conjunction of the previous propositions can be obtained by simply using the AND word, as follows: 1. “Smoking damages your health AND I will read poems from Spanish poet Rafael Alberti.” 2. “Michael Jordan is the greatest basketball player of all time AND jazz is the coolest musical genre in the world.” 3. “100 is greater than 1 AND there are wonderful beaches in Havana.” The truth table for the conjunction connective is shown in Table1-2.

Table 1-2. Truth Table for the Conjunction Logical Connective p

q

p∧q

1

1

1

1

1

Table1-2 permits us to see that p ∧ q is True only when both p and q are True simultaneously.

8

Chapter 1

Logic & AI

D isjunction If we have propositions p, q then their disjunction is denoted p ∨ q (read p OR q). This is a binary logical connective; it requires two propositions as input. The disjunction of the previous propositions can be obtained by simply using the OR word, as follows: 1. “I will read poems from Spanish poet Rafael Alberti OR I listen to Sting’s music.” 2. “Michael Jordan is the greatest basketball player of all time OR jazz is the coolest musical genre in the world.” 3. “World War II ended in 1945 OR there are wonderful beaches in Havana.” The truth table for the conjunction connective is as follows (Table1-3).

Table 1-3. Truth Table for the Disjunction Logical Connective p

q

p∨q

1

1

1

1

1

1

1

From Table1-3 we can see that p ∨ q is True when either p or q are True.

9

Chapter 1

Logic & AI

Implication Countless expressions in mathematics are stated as an implication; i.e., in the manner “if . . . then.” If we have propositions p, q then their implication is denoted p => q (read p IMPLIES q). This is a binary logical connective; it requires two propositions as input and indicates that from p veracity we deduce q veracity. We say that q is a necessary condition for p to be True and p is a sufficient condition for q to be True. The implication connector is similar to the conditional statement (if) that we find in many imperative programming languages like C#, Java, or Python. To understand the outputs produced by the connective let us consider the following propositions: •

p = John is intelligent.

•

q = John goes to the theater.

An implication p => q would be written as “If John is intelligent then he goes to the theater.” Let’s analyze each possible combination of values for p, q and the result obtained from the connective. Case 1, where p = 1, q = 1. In this case, John is intelligent and he goes to the theater; therefore, p => q is True. Case 2, where p = 1, q = 0. In this case, John is intelligent but does not go to the theater; therefore, p => q is False. Case 3, where p = 0, q = 1. In this case, John is not intelligent even though he goes to the theater. Since p is False and p => q only indicates what happens when p = John is intelligent, then proposition p => q is not negated; hence, it’s True. Case 4, where p = 0, q = 0. In this case, John is not intelligent and does not go to the theater. Since p is False and p => q only indicates what happens when p is True, then p => q is True.

10

Chapter 1

Logic & AI

In general, proposition p => q is True whenever p = 0 because if condition p does not hold (John’s being intelligent) then the consequence (John goes to the theater) could be anything. It could be interpreted as “If John is intelligent then he goes to the theater”; otherwise, “If John is not intelligent then anything could happen,” which is True. The truth table for the implication connective is shown in Table1-4.

Table 1-4. Truth Table for the Implication Logical Connective p

q

p => q

1

1

1

1

1

1

1

Proposition p => q is True when p is False or both p and q are True.

E quivalence Propositions p, q are said to be equivalent, denoted p q (read p Is Equivalent to q or p If and Only If q), if it occurs that p => q and q => p both have the same value. The double implication or equivalence connective will output True only when propositions p, q have the same value.

11

Chapter 1

Logic & AI

The truth table for the equivalence connective can be seen in Table1-5.

Table 1-5. Truth Table for the Equivalence Logical Connective p

q

p q

1

1

1

1

1

1

Considering propositions p, q, r, the equivalence connective satisfies the following properties: •

Reflexivity: p p

•

Transitivity: if p r and r q then

p q

•

Symmetry: if p q then q p

Both the implication and equivalence connectives have great importance in mathematical, computational logic, and they represent fundamental logical structures for presenting mathematical theorems. The relationship between artificial intelligence, logical connectives, and logic in general will seem more evident as we move forward in this book.

Laws ofPropositional Logic Now that we have gotten acquainted with all logical connectors, let’s introduce a list of logic equivalences and implications that, because of their significance, are considered Laws of Propositional Logic. In this case, 12

Chapter 1

Logic & AI

p, q, and r are all formulas, and we will use the ≡ symbol to denote that p q is a tautology; i.e., it’s True under any set of values for p, q (any interpretation). In such cases we say that p and q are logically equivalent. This symbol resembles the equal sign used in arithmetic because its meaning is similar but at a logical level. Having p ≡ q basically means that p and q will always have the same output when receiving the same input (truth values for each variable). Logical equivalences: 1. p ∨ p ≡ p (idempotent law) 2. p ∧ p ≡ p (idempotent law) 3. [p ∨ q] ∨ r ≡ p ∨ [q ∨ r] (associative law) 4. [p ∧ q] ∧ r ≡ p ∧ [q ∧ r] (associative law) 5. p ∨ q ≡ q ∨ p(commutative law) 6. p ∧ q ≡ q ∧ p(commutative law) 7. p ∧ [q∨ r] ≡ [p ∧ q] ∨ [p ∧r] (distributive law over ˄) 8. p ∨ [q ∧ r] ≡ [p ∨ q] ∧ [p ∨ r] (distributive law over ˅) 9. p ∨ [p ∧ q] ≡ p 10. p ∧ [p ∨ q] ≡ p 11. p ∨ 0 ≡ p 12. p ∧ 1 ≡ p 13. p ∨ 1 ≡ 1 14. p ∧ 0 ≡ 0 15. p ∨ ¬p ≡ 1 16. p ∧ ¬p ≡ 0(contradiction) 17. ¬[¬p] ≡ p(double negation) 13

Chapter 1

Logic & AI

18. ¬1 ≡ 0 19. ¬0 ≡ 1 20. ¬[p ∨ q] ≡ ¬p ∧ ¬q(De Morgan’s law) 21. ¬[p ∧ q] ≡ ¬p ∨ ¬q(De Morgan’s law) 22. p => q ≡ ¬p ∨ q(definition =>) 23. [p q] ≡ [p => q] ∧ [q => p] (definition ) Note the use of brackets in some of the previous formulas. As occurs in math, brackets can be used to group variables and their connectives all together to denote order relevance, association with logical connectives, and so forth. For instance, having a formula like p ∨ [q ∧ r] indicates the result of subformula q ∧ r is to be connected with the disjunction logical connective and variable p. In the same way as we introduced the ≡ symbol for stating that p, q were logically equivalent we now introduce the ≈ symbol for denoting that p, q are logically implied, written p ≈ q. If they are logically implied then p => q must be a tautology. Logical implications: 1. p ≈ q => [p ∧ q] 2. [p => q] ∧ [q => r] ≈ p => q 3. ¬q => ¬p ≈ p => q 4. [p => q] ∧ [¬p => q] ≈ q 5. [p => r] ∧ [q => r] ≈ [p ∨ q] => r 6. ¬p => [q ∧ ¬q] ≈ p 7. p => [q ∧ ¬q] ≈ ¬p 8. ¬p => p ≈ p 9. p => ¬p ≈ ¬p 14

Chapter 1

Logic & AI

10. p => [¬q => [r ∧ ¬r]] ≈ p => q 11. [p ∧ ¬q] => q ≈ p => q 12. [p ∧ ¬q] => ¬p ≈ p => q 13. [p => q] ∧ [¬p => r ] ≈ q ∨ r 14. ¬p => q ≈ p ∨ q 15. p => q ≈ q ∨ ¬p 16. p ≈ p ∨ q 17. p ∧ q ≈ p 18. p ≈ q => p Many of these laws are very intuitive and can be easily proven by finding all possible values of the variables involved and the final outcome of every formula. For instance, equivalence ¬[p ∨ q] ≡ ¬p ∧ ¬q, which is known as De Morgan’s law, can be proven by considering every possible value for p, q in a Truth table, as shown in Table1-6.

Table 1-6. Truth Table Verifying ¬[p ∨ q] ≡ ¬p ∧ ¬q p

q

¬[p ∨ q]

¬p ∧ ¬q

1

1

1

1

1

1

So far we have presented some of the basic topics of computational logic. At this point, the reader might wonder what the relationship between propositional logic and artificial intelligence may be. First of all, propositional logic and logic in general are the founding fields of many

15

Chapter 1

Logic & AI

areas related to AI.Our brain is crowded with logical decisions, On (1) / Off (0) definitions that we make every step of the way, and that on multiple occasions are justified by our “built-in” logic. Thus, because AI tries to emulate our human brain at some level, we must understand logic and how to operate with it in order to create solid, logical AIs in the future. In the following sections we’ll continue our studies of propositional logic, and we’ll finally get a glimpse of a practical problem.

Normal Forms When checking satisfiability, certain types of formulas are easier to work with than others. Among these formulas we can find the normal forms. •

Negation Normal Form (NNF)

•

Conjunctive Normal Form (CNF)

•

Disjunctive Normal Form (DNF)

We will assume that all formulas are implication free; i.e., every implication p => q is transformed into the equivalent ¬p ∨ q. A formula is said to be in negation normal form if its variables are the only subformulas negated. Every formula can be transformed into an equivalent NNF using logical equivalences 17, 20, and 21 presented in the previous section.

Note Normal forms are useful in automated theorem proving (also known as automated deduction or ATP), a subfield of automated reasoning, which at the same time is a subfield of AI.ATP is dedicated to proving mathematical theorems by means of computer programs.

16

Chapter 1

Logic & AI

A formula is said to be in conjunctive normal form if it’s of the form (p1 ∧ p2 … ∨ pn) ∧ (q1 ∨ q2 … ∨ qm) where each pi, qj is either a propositional variable or the negation of a propositional variable. A CNF is a conjunction of disjunctions of variables, and every NNF can be transformed into a CNF using the Laws of Propositional logic. A formula is said to be in disjunctive normal form if it’s of the form (p1 ∧ p2 … ∧ pn) ∨ (q1 ∧ q2 … ∨ qm) where each pi, qj is either a propositional variable or the negation of a propositional variable. A DNF is a disjunction of conjunctions of variables, and every NNF can also be transformed into a CNF using the Laws of Propositional Logic. At the end of this chapter, we’ll examine several practical problems where we’ll describe algorithms for computing NNF and CNF; we’ll also look at the relationship between normal forms and ATP.

Note A canonical or normal form of a mathematical object is a standard manner of representing it. A canonical form indicates that there’s a unique way of representing every object; a normal form does not involve a uniqueness feature.

Logic Circuits The topics presented thus far regarding propositional logic find applications in design problems and, more importantly, in digital logic circuits. These circuits, which execute logical bivalent functions, are used in the processing of digital information. Furthermore, the most important logical machine ever created by mankind (the computer) operates at a basic level using logical circuits.

17

Chapter 1

Logic & AI

The computer, the most basic, classical example of an AI container, receives input data (as binary streams of ones and zeroes). It processes that information using logic and arithmetic (as our brain does), and finally it provides an output or action. The core of the computer is the CPU (central processing unit), which is composed of the ALU (arithmetic-logic unit) and the CU (control unit). The ALU—and therefore the entire computer— processes information in digital form using a binary language with the symbols 1 and 0. These symbols are known as bits, the elemental unit of information in a computer. Logical circuits represent one of the major technological components of our current computers, and every logical connective described so far in this chapter is known in the electronics world as a logical gate. A logical gate is a structure of switches used to calculate in digital circuits. It’s capable of producing predictable output based on the input. Generally, the input is one of two selected voltages represented as zeroes and ones. The 0 has low voltage and the 1 has higher voltage. The range is between 0.7 volts in emitter-coupled logic and approximately 28 volts in relay logic.

Note Nerve cells known as neurons function in a more complex yet similar way to logical gates. Neurons have a structure of dendrites and axons for transmitting signals. A neuron receives a set of inputs from its dendrites, relates them in a weighted sum, and produces an output in the axon depending on the frequency type of the input signal. Unlike logical gates, neurons are adaptables. Every piece of information that we input into the computer (characters from the keyboard, images, and so on) are eventually transformed into zeroes and ones. This information is then carried on and transported via logic circuits in a discontinuous or discreet manner. Information flows as successive signals commonly made by electronic impulses constituted by high (1) and low (0) voltage levels, as illustrated in Figure1-1. 18

Chapter 1

Logic & AI

Figure 1-1. Digital information flow Logic circuits in the ALU transform the information received by executing the proper logical gates (AND, OR, and so on). As a result, any transformation endured by the incoming information is describable using propositional logic. Circuits are built that connect various elementary electronic components. We will abstract each electronic component and the operation it represents into one of the diagrams shown in Figures1-2, 1-3, and 1-4.

Figure 1-2. Representation of negation component (NOT) In Figure1-5 we can see, as a first example of a logic circuit, a binary comparer. This circuit receives two inputs p, q (bits) and outputs 0 if p and q are equal; otherwise, it outputs 1. To verify that the output of the diagram illustrated in Figure1-5 is correct and actually represents a binary comparer, we could go over all possible values of input bits p, q and check the corresponding results. A simple analysis of the circuit will show us that whenever inputs p, q have different values then each will follow a path in which it is negated, with the other bit left intact. This will activate one of the conjunction gates, outputting 1 for it; thus, the final disjunction gate will output 1 as well, and the bits will not be considered equals. In short, when the two inputs are equal, the output will be 1, and if the inputs are not equal the output will be 0. 19

Chapter 1

Logic & AI

Figure 1-3. Representation of disjunction component (OR)

Figure 1-4. Representation of conjunction component (AND)

20

Chapter 1

Logic & AI

Figure 1-5. Binary comparer circuit Now that we have studied various topics related to propositional logic, it’s time to introduce a first practical problem. In the following section we’ll present a way to represent logic formulas in C# using the facilities provided by this powerful language. We’ll also see how to find all possible outputs of a formula using binary decision trees.

ractical Problem: Using Inheritance and P C# Operators toEvaluate Logic Formulas Thus far, we have studied the basics of propositional logic, and in this section we’ll present a first practical problem. We’ll create a set of classes, all related by inheritance, that will allow us to obtain the output of any formula from inputs defined a priori. These classes will use structural recursion. In structural recursion the structure exhibited by the class—and therefore the object—is recursive itself. In this case, recursion will be present in methods from the Formula class as well as its descendants. Using recursion, we’ll be calling methods all the way through the hierarchy tree. Inheritance in C# will aid recursion by calling the proper version of the method (the one that corresponds to the logical gate that the class represents). 21

Chapter 1

Logic & AI

In Listing 1-1 the parent of every other class in our formula design is presented.

Listing 1-1. Abstract Class Formula public abstract class Formula { public abstract bool Evaluate(); public abstract IEnumerable Variables(); } The abstract Formula class states that all its descendants must implement a Boolean method Evaluate() and an IEnumerable method Variables(). The first will return the evaluation of the formula and the latter the variables contained within it. The Variable class will be presented shortly. Because binary logic gates share some features we’ll create an abstract class to group these features and create a more concise, logical inheritance design. The BinaryGate class, which can be seen in Listing 1-2, will contain the similarities that every binary gate shares.

Listing 1-2. Abstract Class BinaryGate public abstract class BinaryGate : Formula { public Formula P { get; set; } public Formula Q { get; set; } public BinaryGate(Formula p, Formula q) { P = p; Q = q; }

22

Chapter 1

Logic & AI

public override IEnumerable Variables() { return P.Variables().Concat(Q.Variables()); } } In Listing 1-3 the first logic gate, the AND gate, is illustrated.

Listing 1-3. And Class public class And: BinaryGate { public And(Formula p, Formula q): base(p, q) { } public override bool Evaluate() { return P.Evaluate() &&Q.Evaluate(); } } The implementation of the And class is pretty simple. It receives two arguments that it passes to its parent constructor, and the Evaluate method merely returns the logic AND that is built in to C#. Very similar are the Or, Not, and Variable classes, which are shown in Listing 1-4.

Listing 1-4. Or, Not, Variable Classes public class Or : BinaryGate { public Or(Formula p, Formula q): base(p, q) { } public override bool Evaluate()

23

Chapter 1

Logic & AI

{ return P.Evaluate() || Q.Evaluate(); } } public class Not : Formula { public Formula P { get; set; } public Not(Formula p) { P = p; } public override bool Evaluate() { return !P.Evaluate(); } public override IEnumerable Variables() { return new List(P.Variables()); } } public class Variable : Formula { public bool Value { get; set; } public Variable(bool value) { Value = value; }

24

Chapter 1

Logic & AI

public override bool Evaluate() { return Value; } public override IEnumerable Variables() { return new List() { this }; } } Notice the Variable class is the one we use for representing variables in formulas. It includes a Value field, which is the value given to the variable (true, false), and when the Variables() method is called it returns a List whose single element is itself. The recursive inheritance design that we have come up with then moves this value upward in the inheritance to output the IEnumerable with the correct objects of type Variable when requested. Now, let’s try to create a formula and find its output from some defined inputs, as illustrated in Listing 1-5.

Listing 1-5. Creating and Evaluating Formula ¬p ∨ q var p = new Variable(false); var q = new Variable(false); var formula = new Or(new Not(p), q); Console.WriteLine(formula.Evaluate()); p.Value = true; Console.WriteLine(formula.Evaluate()); Console.Read();

25

Chapter 1

Logic & AI

The result obtained after executing the previous code is illustrated in Figure1-6.

Figure 1-6. Result after executing code in Listing 1-5 Since every implication can be transformed into a free implication formula using the OR and NOT expressions (according to the laws of propositional logic) and every double implication can be set free of implications’ transforming it into a conjunction of implications, then having the preceding logic gates is enough to represent any formula.

ractical Problem: Representing Logic P Formulas asBinary Decision Trees A binary decision tree (BDT) is a labelled binary tree satisfying the following conditions:

26

•

The leaves are labelled with either 0 (False) or 1 (True).

•

Non-leaf nodes are labelled with positive integers.

•

Every non-leaf node labelled i has two child nodes, both labelled i + 1.

•

Every branch leading to a left child has a low value (0), and every branch leading to a right child has a high value (1).

Chapter 1

Logic & AI

Note A binary decision tree is just another way of representing or writing the truth table of a formula. In Figure1-7 we can see a binary decision tree with leaf nodes represented as squares and non-leaf nodes represented as circles.

Figure 1-7. Binary decision tree for p ∨ ¬q In a BDT, every level of the tree matches a variable, and its two branches correspond to its possible values (1, 0). A path from the root to a leaf node represents an assignment for all variables of the formula. The value found at a leaf node represents an interpretation of the formula; i.e., the result of an assignation from the root. Now that we have studied some topics related to propositional logic, it’s time to create our first AI data structure. As we’ll see, by using the Formula class introduced in the last practical problem we will be able to create our binary decision tree in just a few lines of code. Three constructors, for different uses, will be included in the class, as shown in Listing 1-6.

27

Chapter 1

Logic & AI

Listing 1-6. Constructors and Properties of BinaryDecisionTree Class public class BinaryDecisionTree { private BinaryDecisionTreeLeftChild { get; set; } private BinaryDecisionTreeRightChild { get; set; } private int Value { get; set; } public BinaryDecisionTree() { } public BinaryDecisionTree(int value) { Value = value; } public BinaryDecisionTree(int value, BinaryDecisionTreelft, BinaryDecisionTreergt) { Value = value; LeftChild = lft; RightChild = rgt; } ... } A binary decision tree is a recursive structure; as a result, its template or class will include two properties, LeftChild and RightChild, that are of type BinaryDecisionTree. The Value property is an integer that identifies the variable as provided in the order given by the Variables() method in the Formula class; this order is equivalent to the height of the tree; i.e., in the first level the root node will have value 0, then at level (height) 1 every node (all representing the same variable) will have value 1 and so on. 28

Chapter 1

Logic & AI

Note In a binary decision tree every level represents a variable in the formula. The left branch leaving a node (variable) corresponds to the decision where that variable will have value 0 (false), and the right branch indicates that the variable will have value 1 (true). The static methods shown in Listing 1-7 will take care of building the binary decision tree.

Listing 1-7. Methods to Build Binary Decision Tree from Formula public static BinaryDecisionTreeFromFormula(Formula f) { return TreeBuilder(f, f.Variables(), 0, ""); } private static BinaryDecisionTreeTreeBuilder(Formula f, IEnumerable variables, intvarIndex, string path) { if (!string.IsNullOrEmpty(path)) variables.ElementAt(varIndex- 1).Value = path[path.Length- 1] != '0'; if (varIndex == variables.Count()) return new BinaryDecisionTree(f.Evaluate() ? 1 : 0); return new BinaryDecisionTree(varIndex, TreeBuilder(f, variables, varIndex + 1, path + "0"), TreeBuilder(f, variables, varIndex + 1, path + "1")); } The public method FromFormula uses an auxiliary private method that relies on recursion to create the tree.

29

Chapter 1

Logic & AI

The varIndex variable defines the height of the tree or, equivalently, the index of the variable representing that tree level. Path stores the evaluation of every variable as a binary string; e.g., “010” denotes the path where the root variable r is evaluated false, then its left child lft is evaluated true, and finally lft’s right child is evaluated false. Once we have reached a depth that equals the number of variables of the formula, we evaluate the formula with the assignment matching the path built so far and leave the final result in a leaf node. By traversing the decision tree we can obtain the output of the formula under a predefined set of values (path from root to leaf node) for its variables. This feature can be very useful during decision-making processes because the tree structure is very intuitive and easy to interpret and understand. Decision trees will be covered deeply in Chapter 4; for now we should know that they provide several advantages or benefits. Among these, it’s worth mentioning that they create a visual representation of all possible outputs and follow-up decisions in one view. Each subsequent decision resulting from the original choice is also depicted on the tree so we can see the overall effect of any one decision. As we go through the tree and make choices, we’ll see a specific path from one node to another and the impact a decision made now could have down the road. As mentioned before, we will describe in the next section various practical problems related to normal forms. We’ll learn how to transform a formula in its regular state to negation normal form (NNF) and from there to conjunctive normal form (CNF). This transformation will come in handy when manipulating formulas and especially for developing logic-related algorithms like DPLL.

30

Chapter 1

Logic & AI

ractical Problem: Transforming aFormula P intoNegation Normal Form (NNF) In this problem, we’ll finally study an algorithm that transforms any formula into negation normal form. Remember, normal forms are useful because •

they reduce logic operators (implication, etc.);

•

they reduce syntactical structure (nesting of subformulas); and

•

they can be taken advantage of to seek efficient data structures.

The NNF transformation algorithm is determined by the following recursive ideas; assuming F is the input formula, this is a pseudocode. Function NNF(F): If F is a variable or negated variable Then return F If F is ¬(¬p) Then return NNF(p) If F is p ∧ q Then return NNF(p) ∧ NNF(q) If F is p ∨ q Then return NNF(p) ∨ NNF(q) If F is ¬(p ∨ q) Then return NNF(¬p) ∧ NNF(¬q) If F is ¬(p ∧ q) Then return NNF(¬p) ∨ NNF(¬q) We will assume that all formulas are implication free and take advantage of the Formula hierarchy to implement the pseudocode described.

31

Chapter 1

Logic & AI

Note The formulas ¬p ∧ q, p ∨ q, (p ∧ (¬q ∨ r)) are all in negation normal form. The formulas ¬(q ∨¬r), ¬(p ∧ q) on the other hand are not in negation normal form as some of these formulas include Or, And formulas that are being negated. To be in NNF only variables can be negated. We’ll start by modifying the Formula abstract class as shown in Listing 1-8.

Listing 1-8. Abstract Method ToNnf() Added to Abstract Class Formula public abstract class Formula { public abstract bool Evaluate(); public abstract IEnumerable Variables(); public abstract Formula ToNnf(); } The And, Or classes require a little modification, including an override to the newly created ToNnf() abstract method (Listing 1-9).

Listing 1-9. And, Or Classes with ToNnf() Method Override public class And: BinaryGate { public And(Formula p, Formula q): base(p, q) { } public override bool Evaluate() { return P.Evaluate() &&Q.Evaluate(); }

32

Chapter 1

Logic & AI

public override Formula ToNnf() { return new And(P.ToNnf(), Q.ToNnf()); } } public class Or : BinaryGate { public Or(Formula p, Formula q): base(p, q) { } public override bool Evaluate() { return P.Evaluate() || Q.Evaluate(); } public override Formula ToNnf() { return new Or(P.ToNnf(), Q.ToNnf()); } } The Not class incorporates most of the steps (if statements) from the NNF pseudocode; its final implementation can be seen in Listing 1-10.

Listing 1-10. Not Class with Nnf() Override public class Not : Formula { public Formula P { get; set; } public Not(Formula p) { P = p; } 33

Chapter 1

Logic & AI

public override bool Evaluate() { return !P.Evaluate(); } public override IEnumerable Variables() { return new List(P.Variables()); } Public override Formula ToNnf() { if (P is And) return new Or(new Not((P as And).P), new Not((P as And).Q)); if (P is Or) return new And(new Not((P as Or).P), new Not((P as Or).Q)); if (P is Not) return new Not((P as Not).P); return this; } } Finally, the Variable class includes a simple override of the Nnf() abstract method inherited from its parent; the entire class is shown in Listing 1-11.

34

Chapter 1

Logic & AI

Listing 1-11. Variable Class with Nnf() Override public class Variable : Formula { public bool Value { get; set; } public Variable(bool value) { Value = value; } public override bool Evaluate() { return Value; } public override IEnumerable Variables() { return new List() { this }; } public override Formula ToNnf() { return this; } } To obtain an NNF out of a formula we can simply call the Nnf() method in some instance of the Formula class.

35

Chapter 1

Logic & AI

ractical Problem: Transforming aFormula P intoConjunctive Normal Form (CNF) A conjunctive normal form (CNF) is basically an AND of ORs; i.e., groups of variables or negated variables all connected using disjunction connectives where all groups are related among themselves by conjunctive connectives; e.g., (p ∨ q) ∧ (r ∨ ¬q). Because of the multiple reasons detailed earlier, we are interested in taking a formula to CNF.A pseudocode of the CNF transformation algorithm is presented in the next lines. Function CNF(F): If F is a variable or negated variable Then return F If F is p ∧ q Then return CNF(p) ∧ CNF(q) If F is p ∨ q Then return DISTRIBUTE-CNF (CNF(p),CNF(q)) Function DISTRIBUTE-CNF(P, Q): If P is R ∧ S Then return DISTRIBUTE-CNF (R, Q) ∧ DISTRIBUTE-CNF (R, Q) If Q is T ∨ U Then return DISTRIBUTE-CNF (P, T) ∧ DISTRIBUTE-CNF (P, U) return P ∨ Q The CNF algorithm relies on an auxiliary method called DISTRIBUTE- CNF that uses the distributive laws of propositional logic to decompose a formula in order to get it closer to the excepted form of a CNF.

Note The CNF algorithm assumes the input formula is already in NNF.Every NNF formula can be transformed into an equivalent CNF formula using the distributive laws of propositional logic.

36

Chapter 1

Logic & AI

As we did with the NNF algorithm, we’ll insert the CNF algorithm into the Formula hierarchy that we have been enhancing in the previous practical problems. Necessary edits to the Formula abstract class are shown in Listing 1-12.

Listing 1-12. Adding ToCnf() and DistributeCnf() Methods to the Formula Class public abstract { public public public public

class Formula abstract abstract abstract abstract

bool Evaluate(); IEnumerable Variables(); Formula ToNnf(); Formula ToCnf();

public Formula DistributeCnf(Formula p, Formula q) { if (p is And) return new And(DistributeCnf((p as And).P, q), DistributeCnf ((p as And).Q, q)); if(q is And) return new And(DistributeCnf(p, (q as And).P), DistributeCnf(p, (q as And).Q)); return new Or(p, q); } } Now that we have added the abstract method to the parent class we can include the corresponding overrides in the child classes And, Or as shown in Listings 1-13 and 1-14.

37

Chapter 1

Logic & AI

Listing 1-13. And Class with ToCnf() Method Override public class And: BinaryGate { public And(Formula p, Formula q): base(p, q) { } public override bool Evaluate() { return P.Evaluate() &&Q.Evaluate(); } public override Formula ToNnf() { return new And(P.ToNnf(), Q.ToNnf()); } public override Formula ToCnf() { return new And(P.ToNnf(), Q.ToNnf()); } } The override implementation of the ToCnf() methods in the Or and And classes represents a direct result drawn from the pseudocode of the CNF function (Listing 1-14).

Listing 1-14. Or Class with ToCnf() Method Override public class Or : BinaryGate { public Or(Formula p, Formula q): base(p, q) { } public override bool Evaluate() 38

Chapter 1

Logic & AI

{ return P.Evaluate() || Q.Evaluate(); } public override Formula ToNnf() { return new Or(P.ToNnf(), Q.ToNnf()); } public override Formula ToCnf() { return DistributeCnf(P.ToCnf(), Q.ToCnf()); } } The Not and Variable classes will simply return a reference to themselves on their ToCnf() override as shown in Listing 1-15.

Listing 1-15. ToCnf() Method Override in Not, Variable Classes public override Formula ToCnf() { return this; } Remember: The CNF algorithm expects as input a formula in NNF; therefore, before executing this algorithm we need to call the ToNnf() method and then the ToCnf() on the Formula object created. In the following chapter, we’ll start diving into an application of AI and logic that’s directly related to all the practical problems we have seen thus far: automated theorem proving.

39

Chapter 1

Logic & AI

Summary In this chapter, we analyzed the relationship between AI and logic. We introduced a basic logic—propositional logic. We described various codes that included a hierarchy for representing formulas (variables, logical connectives, and so on), and we complemented this hierarchy with different methods. Among these methods were the negation normal form transformation algorithm and the conjunctive normal form transformation algorithm (relies on the distributive laws previously introduced). We also described a binary decision tree for representing formulas and their possible evaluations. In the next chapter, we’ll begin studying a very important logic that extends propositional logic: first-order logic. At the same time, we’ll dive into the world of automated theorem proving (ATP) and present a very important method for determining satisfiability of a formula, the DPLL algorithm: (x)IsFriend(x, Arnaldo)(x)IsFriend(x, Arnaldo) (y) IsWorkingWith(y, Arnaldo)

40

CHAPTER 2

Automated Theorem Proving & First-Order Logic Following the line of thought begun in Chapter 1, we’ll start this chapter by introducing a topic related to AI and logic: automated theorem proving. This is a field of AI that serves mathematicians in their research and assists them in proving theorems, corollaries, and so forth. In this chapter, we’ll also devote some pages to first-order logic, a logic that extends propositional logic by allowing or including quantifiers (universal and existential) and providing a more complete framework for easily representing different types of logical scenarios that could arise in our regular life. At the same time, we’ll keep extending the Formula hierarchy introduced in Chapter 1 by inserting clauses and CNF C# classes and describing a very important method for solving the SAT (satisfiability) problem: the DPLL algorithm. Practical problems will help us to better understand every concept hereafter described. We will end the chapter by presenting a simple cleaning robot that will use some of the terms of first-order logic and show how they can be applied in a real-life problem.

© Arnaldo Pérez Castaño 2018 A. Pérez Castaño, Practical Artificial Intelligence, https://doi.org/10.1007/978-1-4842-3357-3_2

41

Chapter 2

Automated Theorem Proving & First-Order Logic

Automated Theorem Proving An automated theorem Prover (ATP) is a computer program that can generate and check mathematical theorems and search for a proof of the theorem’s veracity; i.e., its statement is always true. Theorems are expressed using some mathematical logic, such as propositional logic, first-order logic, and so on. In this case, we’ll only consider an ATP that uses propositional logic as its language. We can think of an ATP’s workflow as illustrated in the diagram in Figure2-1.

Figure 2-1. ATP workflow diagram ATPs were originally created for mathematical computation but recently have gained notice in the scientific community as a wide range of potential applications have been associated with them. One of the several applications of ATPs is adding intelligence to databases of mathematical theorems; in other words, using automated theorem provers to astutely query for equivalent theorems within a database of mathematical theorems. An ATP would be used to verify whether a theorem within the database was mathematically equivalent to another entered by the user. String-matching algorithms or similar techniques wouldn’t be good enough for such an application since the user may have phrased the theorem in a different way than how it was stored in the database, or the searched-for theorem could be a logical consequence rather than a direct clone of existing theorems.

42

Chapter 2

Automated Theorem Proving & First-Order Logic

Another application of theorem provers and formal methods can be found in the verification of hardware and software designs. Hardware verification happens to be an extremely important task. The commercial cost of an error in the design of a modern microprocessor, for instance, is potentially so large that verification of designs is essential. Software verification is similarly crucial as mistakes can be very costly in this area. Examples of the catastrophic consequences of such mistakes are the destruction of the Ariane 5 rocket (caused by a simple integer overow problem that could have been detected by a formal verification procedure) or the error in the floating-point unit of the Pentium II processor. The classical application of ATPs of course is that for which it was created—as a tool to aid mathematicians in their research. One could say ATPs are mathematicians’ favorite robots.

Note Some logics are more powerful and can express and prove more theorems than others. Propositional logic is usually the weakest and simplest of them all. Theorem provers vary depending on the amount of human guidance that is required in the proof search and the sophistication of the logical language that may be used to express the theorem that is to be proven. A tradeoff between the automation degree and the sophistication of the logical language must be taken into account. A high degree of automation is only possible if the language is constrained. Proofs for flexible, high-order languages generally require human guidance, and the associated theorem prover is referred to as a proof assistant. This human assistance can be provided by the programmer’s giving hints a priori or interacting with the ATP during the proof process through a prompt.

43

Chapter 2

Automated Theorem Proving & First-Order Logic

The simplest type of ATP is the SAT (SATisfiability) solver, which relies on propositional logic as theorem language. SAT solvers are very useful, but the expressive power of propositional logic is limited, and Boolean expressions can become quite large. Additionally, the SAT problem was the first to be proved NP(Non-Polynomial)-complete in complexity (S.A.Cook, “The Complexity of Theorem-proving Procedures”). There is a large amount of research done in finding heuristics for efficient SAT solving. In pure mathematics, proofs are somewhat informal; they are “validated” by peer review and are intended to convince and convey an intuitive, clear idea of how the proof works, and the theorem statement should be always true. ATPs provide formal proofs where the output could be, as shown in Figure1-8, the Boolean values Yes, No (True, False), or maybe a counterexample if the statement is found to be False.

Note Software and hardware verification using the approach of model checking works well with propositional logic. Expressions are obtained after considering a state machine description of the problem and are manipulated in the form of binary decision trees. An Automated Theorem Proving (ATP) can usually handle two types of tasks: they can check theorems in their logic or they can automatically generate proofs. When proof checking, the ATP receives as input a formal proof, which consists of a list (steps) of formulas, each justified either by an axiom or inference rule applied to previous formulas: FormulasJustification F1Axiom F2Rule X and F1 ...... Theorem

44

Chapter 2

Automated Theorem Proving & First-Order Logic

These types of proofs are very easy to check mechanically; we just need to make sure that every justification is valid or is applied correctly. However, proof generation is much harder. We need to generate a list of formulas, each with a valid justification and guaranteeing that the last formula is the theorem to be proven. For simple problems, proof generation is very useful; for example, type inference (C#, Java), safety of web applications, and so forth. So far we have described a SAT solver—the binary decision tree, which is suitable for small problems. However, its size is exponential, and to check satisfiability we would need to explore the entire tree in the worst- case scenario. Hence, in future sections we’ll detail more on this topic and on how to obtain better results using other methods.

Note In 1976 Kenneth Appel and Wolfgang Haken proved the four-color theorem using a program that performed a gigantic case analysis of billions of cases. The four-color theorem states that it’s possible to paint a world map using only four colors and guaranteeing that there will not be two neighboring countries that share the same color.

ractical Problem: Clauses andCNFs P Classes inC# In this section, we’ll enhance the logic framework we have been developing throughout this chapter with the addition of the Clause and Cnf classes. We’ll make use of these classes when coding the DPLL algorithm, probably the most ingenious algorithm for determining the satisfiability of a logic formula and a basic tool for automated theorem proving.

45

Chapter 2

Automated Theorem Proving & First-Order Logic

Before we start developing this new enhancement, let’s take a brief look at some definitions that will come in handy for understanding the classes that we’ll be developing soon. A literal is either a variable or the negation of a variable (e.g., p, ¬p, q, ¬q). A clause is a disjunction of literals p1 ∨ p2 ∨ ... ∨ pm, and every CNF is a set of clauses. From now on we’ll denote a clause as {p1, p2, ... pm} where every pi(i = 1, 2, ... ,m) is a literal. In Listing 2-1 we illustrate the proposed Clause class.

Listing 2-1. Clause Class public class Clause { public List Literals { get; set; } public Clause() { Literals = new List(); } public bool Contains(Formula literal) { if (!IsLiteral(literal)) throw new ArgumentException("Specified formula is not a literal"); foreach (var formula in Literals) { if (LiteralEquals(formula, literal)) return true; } return false; }

46

Chapter 2

Automated Theorem Proving & First-Order Logic

public Clause RemoveLiteral(Formula literal) { if (!IsLiteral(literal)) throw new ArgumentException("Specified formula is not a literal"); var result = new Clause(); for (vari = 0; i | formula ::= predicate (term , ..., term) | (term = term) | ¬formula | ((formula) binary connective (formula)) | ∀(variable) formula | ∃(variable) formula

77

Chapter 2

Automated Theorem Proving & First-Order Logic

In propositional logic we interpreted a formula as the assignment of truth values to its propositional variables. In FOL the introduction of predicates and quantifiers gives us formulas whose evaluation depends on the interpretation given in some domain (integers, real numbers, cars, pencils . . . anything we can think of ) or universe of objects; the concept of interpretation in this case is a bit more complicated.

Note An interpretation of a formula is a pair (D, A) where D is the domain and A an assignment for each constant, function, predicate, and so on. In order to define the interpretation I of a formula in a domain or set of objects D we must consider the following rules of interpretation: 1. If c is a constant then c has domain D.This mapping indicates how names (constants are basically names) are connected to objects of the universe. We may have a constant Johnny, and the interpretation of Johnny in the world of dogs could be a particular dog. 2. If P is a predicate then P has D x D x ... D domain; i.e., there’s a mapping from predicates to relations in D. 3. If f is a function then f has domain D, an image also in D; i.e., there’s a mapping from functions to functions in D.

78

Chapter 2

Automated Theorem Proving & First-Order Logic

Given an interpretation I of a formula F under domain D, I follows the following rules of evaluation: 1. If P(v1, v2, ... ,vn) is a predicate then P is True if (v1, v2, ... , vn) is a relation in D; i.e., (v1, v2, ... , vn)∈ D x D x ... x D.Recall that an n-ary relation is a set of n-tuples. 2. If F, F' are formulas of FOL then F ∧ F', F ∨ F', F => F', F F', ¬F have the same veritative value in domain D as they would have using the same operators in propositional logic; i.e., these operators have the same truth tables in both logics. 3. The formula ∀(v)F(v) is True if F(v) is True for all values of v in D. 4. The formula ∃(v)F(v) is True if F(v) is True for at least one value of v in D. Let’s examine an example that will clarify how interpretation and evaluation works in FOL; consider the following interpretation I of a formula under domain D: ∃(x)IsFriend(x, Arnaldo)∧∃(y)IsWorkingWith(y, Arnaldo) D = {John, Arnaldo, Mark, Louis, Duke, Sting, Jordan, Miles, Lucas, Thomas, Chuck, Floyd, Hemingway} Constants = {Arnaldo} Predicates = {IsFriend, IsWorkingWith} I(Arnaldo) = Arnaldo I(IsFriend) = {(John, Arnaldo), (Mark, Louis), (Duke, Sting), (Jordan, Miles)} I(IsWorkingWith) = {(Lucas, Arnaldo), (Thomas, Chuck), (Floyd, Hemingway)}

79

Chapter 2

Automated Theorem Proving & First-Order Logic

For determining the truth value of the previous interpretation we have that ∃(x)IsFriend(x, Arnaldo) for x = John is True because tuple or relation (John, Arnaldo) belongs to IsFriend; therefore, ∃(x)IsFriend(x, Arnaldo) is also True. ∃(y)IsWorkingWith(y, Arnaldo) for y = Lucas is True because tuple or relation (Lucas, Arnaldo) belongs to IsWorkingWith; therefore, ∃(y)IsWorkingWith(y, Arnaldo) is also True. Since both ∃(x)IsFriend(x, Arnaldo) and ∃(y)IsWorkingWith(y, Arnaldo) are True, their conjunction is True, and the interpretation is also True.

P redicates inC# Since we are exploring the world of FOL and its most notable components (predicates, quantifiers, and so forth) it would be worth mentioning that in C# we can make use of the Predicate delegate, a construct that allows us to test whether an object of type T fulfills a given condition. For example, we could have the Dog class as follows (Listing 2-14).

Listing 2-14. Dog Class public class Dog { public string Name { get; set; } public double Weight { get; set; } public Gender Sex { get; set; } public Dog(string name, double weight, Gender sex) { Name = name; 80

Chapter 2

Automated Theorem Proving & First-Order Logic

Weight = weight; Sex = sex; } } public enum Gender { Male, Female } Then, we can use a predicate to filter and get objects that satisfy certain properties, as illustrated in Listing 2-15, where we create a list of dogs and then use the Find() method, which expects a predicate as argument, to “find” all objects (dogs) satisfying the given predicates.

Listing 2-15. Using a Predicate in C# to Filter and Get Objects (Dogs in This Case) That Are Males and Dogs Whose Weight Exceeds 22 Pounds varjohnny = new Dog("Johnny", 17.5, Gender.Male); var jack = new Dog("Jack", 23.5, Gender.Male); varjordan = new Dog("Jack", 21.2, Gender.Male); varmelissa = new Dog("Melissa", 19.7, Gender.Female); var dogs = new List { johnny, jack, jordan, melissa }; PredicatemaleFinder = (Dog d) => { return d.Sex == Gender. Male; }; PredicateheavyDogsFinder = (Dog d) => { return d.Weight>= 22; }; varmaleDogs = dogs.Find(maleFinder); varheavyDogs = dogs.Find(heavyDogsFinder); At this point, we have gotten ourselves into the world of propositional logic and FOL. In the next section we will present a practical problem where we’ll see some FOL in action. 81

Chapter 2

Automated Theorem Proving & First-Order Logic

Practical Problem: Cleaning Robot In this section we’ll see many of the concepts described earlier (functions, predicates, and so forth) being applied in the creation of a cleaning robot, whose world is illustrated in Figure2-5.

Figure 2-5. Cleaning robot in the grid. Dirt is marked as orange balls and logically represented on the grid as integers. Following this idea, the cell on the upper-left corner (first one) has value 5. This cleaning robot tries to get rid of the dirt in a grid of n x m (n rows, m columns). Each cell in the grid is an integer d, where d indicates the count of dirt in that cell. When d = 0 that cell is considered clean. The robot will have the following features:

82

•

It moves one step at a time in four possible directions (left, up, right, down).

•

It does not abandon a cell until is completely clean, and it picks dirt up one step at a time; i.e., if on a dirty cell it will clean a unit of dirt at a time (leaving -1 dirt) and then continue to its next decision stage.

•

It stops when everything is clean or its task has exceeded a given time in milliseconds.

Chapter 2

Automated Theorem Proving & First-Order Logic

Our cleaning robot will rely on the following predicates and functions: •

IsDirty() is a predicate that determines if the cell where the robot is happens to be dirty.

•

IsTerrainClean() is a predicate that determines if every cell on the terrain is clean.

•

MoveAvailable(int x, int y) is a predicate that determines whether a move to (x, y) in the terrain is legal.

•

SelectMove() is a function that randomly selects a move.

•

Clean() is a function that simply cleans (-1) a dirt from current cell; i.e., the cell where the robot is at that moment.

•

Move(Direction m) is a function that moves the robot in direction m.

•

Print() is a function that prints the terrain.

•

Start(intmilliseconds) is a function that commands the robot to start cleaning up. The code of this method matches the robot behavior explained earlier. The integer argument milliseconds represents the maximum time the robot will be cleaning, in milliseconds.

The robot is encoded in a CleaningRobot C# class that goes as shown in Listing 2-16.

83

Chapter 2

Automated Theorem Proving & First-Order Logic

Listing 2-16. CleaningRobot Class public class CleaningRobot { private readonlyint[,] _terrain; private static Stopwatch _stopwatch; public int X { get; set; } public int Y { get; set; } private static Random _random; public CleaningRobot(int [,] terrain, int x, int y) { X = x; Y = y; _terrain = new int[terrain.GetLength(0), terrain.GetLength(1)]; Array.Copy(terrain, _terrain, terrain.GetLength(0) * terrain. GetLength(1)); _stopwatch = new Stopwatch(); _random = new Random(); } public void Start(intmilliseconds) { _stopwatch.Start(); do { if (IsDirty()) Clean(); else Move(SelectMove()); } while (!IsTerrainClean() && !(_stopwatch.Elapsed Milliseconds>milliseconds)); } 84

Chapter 2

Automated Theorem Proving & First-Order Logic

// Function private Direction SelectMove() { var list = new List { Direction.Down, Direction.Up, Direction.Right, Direction.Left }; return list[_random.Next(0, list.Count)]; } // Function public void Clean() { _terrain[X, Y] -= 1; } // Predicate public bool IsDirty() { return _terrain[X, Y] > 0; } // Function private void Move(Direction m) { switch (m) { case Direction.Up: if (MoveAvailable(X- 1, Y)) X -= 1; break; case Direction.Down: if (MoveAvailable(X + 1, Y)) X += 1; break; 85

Chapter 2

Automated Theorem Proving & First-Order Logic

case Direction.Left: if (MoveAvailable(X, Y- 1)) Y -= 1; break; case Direction.Right: if (MoveAvailable(X, Y + 1)) Y += 1; break; } } // Predicate public bool MoveAvailable(int x, int y) { return x >= 0 && y >= 0 && x < _terrain. GetLength(0) && y < _terrain.GetLength(1); } // Predicate public bool IsTerrainClean() { // For all cells in terrain; cell equals 0 foreach (var c in _terrain) if (c > 0) return false; return true; } public void Print() { var col = _terrain.GetLength(1); vari = 0; var line = ""; 86

Chapter 2

Automated Theorem Proving & First-Order Logic

Console.WriteLine("--------------"); foreach (var c in _terrain) { line += string.Format("{0}", c); i++; if (col == i) { Console.WriteLine(line); line = ""; i = 0; } } } } public enumDirection { Up, Down, Left, Right } The constructor of the class receives as arguments the terrain and two integers x, y that represent the initial position of the robot on the terrain. The print() method was included for testing purposes. Let’s suppose we have the terrain as shown in the following code and then we execute the robot, i.e., call the Start() method on it, as seen in Listing 2-17.

Listing 2-17. Starting the Cleaning Robot var terrain = new [,] { {0, 0, 0}, {1, 1, 1}, {2, 2, 2} }; 87

Chapter 2

Automated Theorem Proving & First-Order Logic

varcleaningRobot = new CleaningRobot(terrain, 0, 0); cleaningRobot.Print(); cleaningRobot.Start(50000); cleaningRobot.Print(); The terrain contains dirt on the second (1 on each column) and third rows (2 on each column), and after the robot has finished his task, according to one of the termination conditions (everything’s clean or time’s up) stated before, we obtain the result seen in Figure2-6.

Figure 2-6. Terrain before and after the cleaning of the robot As occurred before when developing the DPLL algorithm, we need a heuristic for selecting the next move of the agent. We’ll get into the field of heuristics and metaheuristics in Chapter 7. This cleaning robot is a very naïve, simple agent; the topic of agents in AI will be addressed in the next chapter. For the moment, we have created the necessary basis to start diving into more complicated and interesting subjects and branches of AI.In any case, future topics to be studied will be related to logic as it’s the basis of many sciences and areas of knowledge.

88

Chapter 2

Automated Theorem Proving & First-Order Logic

Summary In the last two chapters we analyzed the relationship between AI and logic. We introduced two fundamental types of logic: propositional logic and first-order logic. We examined various codes that included a hierarchy for representing formulas (variables, logical connectives, and so on), and we complemented this hierarchy with different methods. Among these methods we presented the negation normal form transformation algorithm, the conjunctive normal form transformation algorithm (relies on the distributive laws previously introduced), and the DPLL algorithm, which is a classic algorithm for determining the satisfiability of a formula. Additionally, we described a binary decision tree for representing formulas and their possible evaluations and a practical problem where a simple, naïve cleaning robot uses first-order logic concepts to formulate its simple intelligence. In the next chapter, we’ll begin explaining agents and many of the concepts around these (proactive, reactive) that we may have heard of before from video-game fans, AI fans, friends, or colleagues.

89

CHAPTER 3

Agents In this chapter, we’ll begin describing a very important field of study in the world of AI: agents. Nowadays, agents represent an area of strong interest for many subfields of computer science and AI.They are being used in a great number of applications, ranging from comparatively small systems such as email filters to complex, colossal systems such as air traffic control. In the next pages we’ll address agents as fundamental AI entities; we will start by getting acquainted with a possible agent definition (as there’s no global agreement regarding this concept). We’ll examine different agents’ properties and architectures and analyze a practical problem that will help us understand how to develop agents in C#. Practical problems examined in this and the following chapter will set the concepts presented throughout this chapter on firm ground, and many of them will be connected to classical problems of AI. We’ll give meaning and definition to many of the words that we typically hear today from videogamers, AI hobbyists, or programmers associated with AI—words such as reactive, proactive, perceptions, actions, intentions, or deliberation. Typical examples of agents that we might know are a robot (like the cleaning robot from last chapter), a web-based shopping program, a traffic-control system, software daemons, and so on.

© Arnaldo Pérez Castaño 2018 A. Pérez Castaño, Practical Artificial Intelligence, https://doi.org/10.1007/978-1-4842-3357-3_3

91

Chapter 3

Agents

Note Agents are colloquially known as bots, which derives from the word robot. They could use metallic bodies similar to the ones we see in science fiction films or just consist of computer software installed on our phone, like Siri. They may possess human abilities like speech and speech recognition and be able to act on their own.

W hat’s anAgent? As mentioned earlier, there’s no agreement on a global concept of the term agent. Let’s remember that the same thing occurred with the concept of logic (recall that we analyzed it in Chapter 2). To provide a definition of the term agent we will consider different definitions from various authors and take the most generic features from all of them, attaching some self-logic to it. Since agent is a term drawn from AI, we must bear in mind that, as happens with everything in the field of AI, it relates to creating an artificial entity, something that emulates and enhances, if possible, the making of a set of human tasks in a certain way and environment. Hence, an agent is an entity (human, computer program) that, using a set of sensors (to sense maybe heat, pressure, and so on, kind of like humans do), is capable of obtaining a set of percepts or inputs (warm, high pressure, and so forth) and has the ability to act (turn on AC, move to different location) upon that environment through actuators. Actuators for the human case can be their legs, arms, or mouth, and in the robot case it can be their robotic arms, wheels, or similar. Percepts or inputs are every piece of data that the agent receives through its sensors. In the human case sensors can be eyes, nose, ears, or anything that we actually have for pulling information out of the world, our daily

92

Chapter 3

Agents

environment. In the robot case, sensors can be their cameras, microphone, or anything that they can use to obtain inputs from the environment. In both cases the input received is transformed into percepts, which represent pieces of information with some logic attached. For instance, using our ears we could notice that, when entering a room, the music in it is too loud. How does the process of noticing and receiving this perception work? Our ears sense the loud sounds in the room, and that information is passed on to our brain, which processes it and creates a percept labelled “loud music,” and then we know. Optionally, we could act upon that percept and use our arms and hands (actuators) to lower the volume on the music. The same occurs with nonhuman agents, but at a software level and maybe using some robotic parts (arms, wheels, and so on). From a mathematical point of view, the definition of agent can be viewed as a function that uses a set of tuples or relations from a set of percepts as the domain and has a set of actions (Figure3-1); i.e., assuming F is the agent’s function, P the set of percepts, and A the set of actions, F: P* → A. Now that we have provided a definition for the very important term of agent, it’s time to define what we will refer to as an intelligent agent.

93

Chapter 3

Agents

Figure 3-1. An agent in its environment. The agent uses its sensory components to receive inputs from the environment. It processes these inputs and eventually outputs an action that affects the environment. This will be a continuous interaction as long as the agent remains active. An intelligent agent is an autonomous agent capable of executing its actions while considering several agent properties, such as reactivity, proactiveness, and social ability. The main difference between an agent and an intelligent agent are the words intelligent and autonomous, the latter of which is associated with the independence that is expected in its behavior, while the first relates to the properties just mentioned. These properties and others will be the main topic of the following sections.

Note An agent does not necessarily need to be an intelligent agent since that feature involves a set of more human or advanced attributes (reactivity, proactiveness, social ability, and so on) that a simple agent such as a movement detector may not need. Thus, to be as general as possible, we begin with the more generic agent definition and then discuss the intelligent agent definition.

94

Chapter 3

Agents

Agent Properties Now that we have gotten acquainted with the agent and intelligent agent concepts, it’s time to describe those properties mentioned that make an agent intelligent. Autonomy refers to the ability of agents to act without the direct intervention of humans or other agents and have control over their own actions and internal state. Reactivity refers to the ability of agents to perceive their environment and respond in a timely fashion (response must be useful) to the percepts received in it so as to meet the agent’s designated goals. Proactiveness refers to the ability of agents to exhibit goal-directed behavior and take the initiative by creating plans or similar strategies that would lead them to satisfy their designated goals. Social ability refers to the capability of an agent to interact with other agents (possible humans) in a multi-agent system to achieve its designated goals. Since this property relates to multi-agents’ environments, we’ll address it further in the next chapter. Another very important property is that of rationality. We say that an agent is rational if it acts in order to achieve its goals and will never act in such a way as to prevent its goals from being achieved. Purely reactive agents decide what to do without looking at their percepts history. Their decision-making process is based solely on the current percept without looking at their past; hence, they have no memory or do not consider it. Mathematically speaking the agent function of a purely reactive agent is F: P → A. As we can see, an agent that only exhibits the reactive property will only need the current percept in order to provide an action.

95

Chapter 3

Agents

Note The agent’s function for a generic agent is F: P * → A. The asterisk on top of the P denotes a relation of zero or more percepts; i.e., a set of tuples of length n where n >= 0; this is the number that replaces the asterisk. In the purely reactive agent case, n = 1. The decision-making process in a reactive agent is implemented as a direct mapping from state to action. Agents incorporating this property react to the environment without reasoning about it. The cleaning robot described in the last chapter is an example of a reactive agent; remember we had rules like the ones shown in Listing 3-1.

Listing 3-1. Simple Rule of the Cleaning Robot from Last Chapter, a Reactive Agent if (IsDirty()) Clean(); else Move(SelectMove()); These were simply rules that made our robot react to the environment without any reasoning whatsoever. The SelectMove() method returned a random move to be executed by the agent, so no heuristic (to be seen in Chapter 14) or any other type of goal-directed analysis or behavior was incorporated into this agent. As happens with the cleaning robot, every reactive agent is basically hardwired as a set of if … then rules. What advantages do we get from developing reactive agents? 1. It is really easy to code them, and they allow us to obtain an elegant, legible code. 2. They are easy to track and understand. 3. They provide robustness against failures.

96

Chapter 3

Agents

What would be the disadvantages or limitations of a purely reactive agent? 1. Since they make decisions based on local information—in other words, information about the agent’s current state—it's difficult to see how such decision making could take into account non-local information; hence, they have a “short horizon” view. 2. It is difficult to make them learn from experience and improve their performance over time. 3. It’s hard to code reactive agents that must incorporate a large number of behaviors (too many situations -> action rules). 4. They don’t have any proactive behavior; therefore, they do not make plans or care about the future, just about the present or immediate action to execute. Reacting to an environment is quite easy, but we regularly need more from our agents; we need them to act on our behalf and do things for us. In order to accomplish these tasks, they must have goal-directed behavior—they must be proactive. Proactive agents will be looking to create and achieve secondary goals that will eventually lead them to fulfill their primary goals. As part of their operation, such agents should be able to anticipate needs, opportunities, and problems, and act on their own initiative to address them. They should also be able to recognize opportunities on the fly; for example, available resources, pattern anomalies, chances of cooperation, and so forth. A common example of a proactive agent is a personal assistant agent, like those likely installed on one of our devices. This agent can be running constantly on our phone, keeping track of our location and preferences and proactively suggesting places to visit according to those preferences (cultural activities in the area, restaurant offering our type of food, and so on).

97

Chapter 3

Agents

In general we’ll want our agents to be reactive; that is, respond to the changing conditions of the environment in a timely fashion or equivalently respond to short-term goals. We also want them to be proactive and systematically work toward meeting long-term goals. Having an agent that balances these two properties is an open research problem. In this chapter, we’ll analyze a practical problem in which we’ll add proactive features to the cleaning robot presented in Chapter 1. Other properties of agents that, although not considered basic properties like the ones previously mentioned, still are relevant are shown in Table3-1.

Table 3-1. Other Agent Properties Property

Description

Coordination

It means the agent is capable of executing some activity in a shared environment with other agents. It answers the question, How do you divide a task between a group of agents? Coordination occurs through plans, workflows, or any other management tool.

Cooperation

It means the agent is able to cooperate with other agents so as to fulfill their common goal (share resources, results, distributed problem solving). They either succeed or fail all together, as a team.

Adaptability

Also referred to as learning, it means the agent is reactive, proactive, and capable of learning from its own experiences, the environment, and its interactions with others.

Mobility

It means the agent is able to transport itself from one shell to another and use different platforms.

Temporal continuity

It means the agent is continuously running.

(continued)

98

Chapter 3

Agents

Table 3-1. (continued) Property

Description

Personality

It means the agent has a well-defined personality and a sense of emotional state.

Reusability

It means successive agent instances can require keeping instances of the agent class for information reuse or to check and analyze previously generated information.

Resource limitation

It means the agent can act only as long as it has resources at its disposal. These resources are modified by its actions and also by delegating.

Veracity

It means the agent will not knowingly communicate false information.

Benevolence

It means the agent will run under the assumption it does not have conflicting goals, and it will always try to do what is asked of it.

Knowledge-level It means the agent will have the ability to communicate with communication human agents and maybe other nonhuman agents using a humanlike language (English, Spanish, etc.). Now that we have detailed some significant agent properties, let’s examine some of the different types of environment in which our agent can be interacting; eventually, we’ll also introduce various agent architectures that we could implement for our agent.

Types ofEnvironments Depending on the type of environment, an agent may or may not need a set of properties. Hence, the decision-making process of the agent is affected by the features exposed by the environment in which it runs. These features make up the types of environment that will be described in this section. 99

Chapter 3

Agents

In a deterministic environment every action taken by the agent will have a single possible outcome; i.e., there is no uncertainty about the resulting state or percept after executing an action (Figure3-2).

Figure 3-2. Deterministic environment; an agent is in state S and can only move to state or percept S1 after executing an action A.Every state is linked to just one state; i.e., there’s a single possible outcome for every action executed by the agent. On the other hand, a non-deterministic environment is one in which actions executed by agents do not have a well-determined state and rather than just being a single state it could be a set of states; for instance, executing action A could lead to states S1, S2, or S3. This is non- deterministic, as illustrated in Figure3-3. Non-deterministic environments are the most complicated environments for agent design. Board games using dice are usually non-deterministic, as the roll of the dice could bring the agent to any state, and it depends on the values displayed on the dice.

100

Chapter 3

Agents

Figure 3-3. Non-deterministic environment; an agent is in state S and after executing action A it could move to states S1, S2, or S3. Every state is linked to a set of states; i.e., there are multiple possible outcomes for every action executed by the agent. In a static environment only actions executed by the agent will affect the environment and cause it to alter. In dynamic environments there are multiple processes operating, many of which are not related in any way to the agent, yet they still affect the environment and change it. The physical world is a highly dynamic environment. A discrete environment is one in which there are a fixed, finite number of actions and percepts. Alternatively, a continuous environment is one in which both actions and percepts are not determined by a finite number. Board games like Chess, Sliding Tiles Puzzle, Othello, or Backgammon represent discrete environments. However, an environment consisting of an actual city represents a continuous environment as there’s no way to limit to a fixed, finite number the percepts that the agent may perceive in such an environment. An accessible environment is one in which the agent can obtain accurate, complete, and updated information about the environment’s state. An inaccessible environment is the opposite—it’s one in which the agent cannot obtain accurate, complete, updated information. The more accessible an environment is the easier it will be to design an agent for it.

101

Chapter 3

Agents

Finally, an episodic environment is one in which the agent’s performance depends on a discrete number of episodes and there’s no relation between the performance of the agent in different episodes. In this type of environment the agent can decide what action to execute based only on the current episode.

Note The most complex class of environment is composed of those that are inaccessible, non-deterministic, non-episodic, dynamic, and continuous.

Agents withState Thus far we have considered agents that map a percept or sequence of percepts to an action. Because agents (not reactive ones) are capable of mapping from a sequence of percepts, they are aware of their history. In this section, we’ll go further and examine agents that also maintain state. The state of an agent will be maintained by means of an internal data structure, which will be used to store information about the environment while the agent is being executed. As a result, the decision-making process could be based on the information stored in this data structure. The agent function then slightly changes to incorporate this new feature. F: I x P* → A where I is the set of internal environmental states stored by the agent, P the set of percepts, and A the actions set. Hence, with stateless agents we just had F: P* → A; now in this case we added the necessary consideration of the internal data structure by making the agent function receive as arguments an internal state and a percept or sequence of percepts; i.e., F(I, P1, P2 ... PN) = A. 102

Chapter 3

Agents

It is worth noting that state-based agents like the ones defined in this section are actually vastly more powerful than an agent without state. In the next practical problem, we’ll enhance the cleaning robot described in Chapter 1 by adding state to it.

ractical Problem: Modeling theCleaning P Robot asanAgent andAdding State toIt In this practical problem, we’ll modify the CleaningRobot class that we described in the last chapter to adapt it to the agent paradigm (percepts, actions, and so on), specifically to the agent’s function. We’ll also add state to this agent in the form of a List that will store cells already visited and cleaned. We’ll see the benefits of having such state and compare it with the CleaningRobot class that is stateless. We shall name this class CleaningAgent, and its constructor will be very much like the constructor of the CleaningRobot, as seen in Listing 3-2. For this new class, we’ll add the Boolean TaskFinished field, which will indicate when the task of the agent is finished, and the List __cellsVisited, which will determine the set of cells that have been already visited.

Listing 3-2. Constructor and Fields of the Cleaning Agent public class CleaningAgent { private readonly int[,] _terrain; private static Stopwatch _stopwatch; public int X { get; set; } public int Y { get; set; } public bool TaskFinished { get; set; } // Internal data structure for keeping state

103

Chapter 3

Agents

private readonly List __cellsVisited; private static Random _random; public CleaningAgent(int [,] terrain, int x, int y) { X = x; Y = y; _terrain = new int[terrain.GetLength(0), terrain. GetLength(1)]; Array.Copy(terrain, _terrain, terrain.GetLength(0) * terrain.GetLength(1)); _stopwatch = new Stopwatch(); _cellsVisited= new List(); _random = new Random(); } } The working loop of the agent is now related to the agent function; i.e., it executes an action based on the set of perceptions it gets from the environment. The loop ends when the task is finished or the maximum execution time (in milliseconds) is reached, as shown in Listing 3-3.

Listing 3-3. Loop of the Agent Matching the Agent’s Function Definition public void Start(int miliseconds) { _stopwatch.Start(); do { AgentAction(Perceived()); }

104

Chapter 3

Agents

while (!TaskFinished && !(_stopwatch. ElapsedMilliseconds > miliseconds)); } The methods Clean(), IsDirty(), MoveAvailable(int x, int y), and Print() will remain as they were in the CleaningRobot class; these are illustrated in Listing 3-4.

Listing 3-4. Methods Clean(), IsDirty(), MoveAvailable(int x, int y), and Print() as They Were in the CleaningRobot Class public void Clean() { _terrain[X, Y] -= 1; } public bool IsDirty() { return _terrain[X, Y] > 0; } public bool MoveAvailable(int x, int y) { return x >= 0 && y >= 0 && x < _terrain. GetLength(0) && y < _terrain.GetLength(1); } public void Print() { var col = _terrain.GetLength(1); var i = 0; var line = ""; Console.WriteLine("--------------"); foreach (var c in _terrain)

105

Chapter 3

Agents

{ line += string.Format("{0}", c); i++; if (col == i) { Console.WriteLine(line); line = ""; i = 0; } } } The set of perceptions will be obtained by a method shown in Listing 3-5, which returns a list of percepts that will be represented by an enum (declared outside of the CleaningAgent class) that defines every possible perception in the CleaningAgent environment; this enum can also be seen in Listing 3-5.

Listing 3-5. Percepts enum and the Perceived() Method That Returns a List Containing Every Perception the Agent Has Obtained from the Environment public enum Percepts { Dirty, Clean, Finished, MoveUp, MoveDown, MoveLeft, MoveRight } private List Perceived() { var result = new List(); if (IsDirty()) result.Add(Percepts.Dirty); 106

Chapter 3

Agents

else result.Add(Percepts.Clean); if (_cellsVisited.Count == _terrain.GetLength(0) * _terrain.GetLength(1)) result.Add(Percepts.Finished); if (MoveAvailable(X- 1, Y)) result.Add(Percepts.MoveUp); if (MoveAvailable(X + 1, Y)) result.Add(Percepts.MoveDown); if (MoveAvailable(X, Y- 1)) result.Add(Percepts.MoveLeft); if (MoveAvailable(X, Y + 1)) result.Add(Percepts.MoveRight); return result; } As mentioned before, this agent will maintain a state corresponding to the history of cells visited. For that purpose we implement the UpdateState() method seen in Listing 3-6.

Listing 3-6. Method for Updating the State of the Agent; i.e., Cells Visited private void UpdateState() { if (!_cellsVisited.Contains(new Tuple(X, Y))) _cellsVisited.Add(new Tuple(X, Y)); }

107

Chapter 3

Agents

The method that puts it all together is AgentAction(List percepts) shown in Listing 3-7. In this method, we go through every percept obtained from the environment and act accordingly. For instance, if the current cell is clean we update the state (internal data structure) of the agent by adding that cell to the _cellsVisited list; if we perceive that the current cell is dirty we clean it and so on for each situation or percept and its consequence or action. Additionally, Listing 3-7 also illustrates the methods RandomAction(List percepts) and Move(Percepts p). The first selects a random movement percept (MoveUp, MoveDown, etc.) to be executed, and the latter executes the movement percept supplied as argument. Note that this agent will always check its state and percept (recall I x P is the domain of agents with state) before moving, and it will always try to move to an adjacent cell not previously visited.

Listing 3-7. Method for Updating the State of the Agent; i.e., Cells Visited public void AgentAction(List percepts) { if (percepts.Contains(Percepts.Clean)) UpdateState(); if (percepts.Contains(Percepts.Dirty)) Clean(); else if (percepts.Contains(Percepts.Finished)) TaskFinished = true; else if (percepts.Contains(Percepts.MoveUp) && !_ cellsVisited.Contains(new Tuple(X- 1, Y))) Move(Percepts.MoveUp);

108

Chapter 3

Agents

else if (percepts.Contains(Percepts.MoveDown) && !_cellsVisited.Contains(new Tuple(X + 1, Y))) Move(Percepts.MoveDown); else if (percepts.Contains(Percepts.MoveLeft) && !_cellsVisited.Contains(new Tuple(X, Y- 1))) Move(Percepts.MoveLeft); else if (percepts.Contains(Percepts.MoveRight) && !_cellsVisited.Contains(new Tuple(X, Y + 1))) Move(Percepts.MoveRight); else RandomAction(percepts); } private void RandomAction(List percepts) { var p = percepts[_random.Next(1, percepts.Count)]; Move(p); } private void Move(Percepts p) { switch (p) { case Percepts.MoveUp: X -= 1; break; case Percepts.MoveDown: X += 1; break; case Percepts.MoveLeft: Y -= 1; break;

109

Chapter 3

Agents

case Percepts.MoveRight: Y += 1; break; } } What advantages does the cleaning agent provide us over the stateless cleaning robot? In order to answer this question, let’s first note that the strategy (recording of its environment history by saving visited cell coordinates) we are using with the cleaning agent is very intuitive. Imagine you need to find some product X in a big city where there exist over 100 stores; how would you accomplish such a task? Intuitively, you would visit a store once and then record in your mind that you already visited that store and the product was not there, thus saving the time of having to revisit it. You would then move from one store to the next until you found the product, always keeping in mind that stores already visited are a waste of time. That’s basically what our cleaning agent tries to do this with the exception that there might be times when already-visited cells will have to be revisited because the agent can only move to adjacent cells and they all may have been visited at some point. In Figure3-4 we can see a basic comparison between the cleaning agent and the cleaning robot.

110

Chapter 3

Agents

Figure 3-4. The cleaning agent (in blue) searches the environment, saving coordinates of visited cells, while the cleaning robot (in red) does not save the state of the environment or its history; therefore, it simply makes random moves that could take it up or down and even going in circles, thus consuming more time to clean the dirt on the last cell. In Listing 3-8 we have an environment of 1000 x 1 cells, i.e., 1000 rows and one column, and dirt is located just in the last row.

Listing 3-8. Method for Updating the State of the Agent; i.e., Cells Visited var terrain = new int[1000, 1]; for (int i = 0; i < terrain.GetLength(0); i++) { for (int j = 0; j < terrain.GetLength(1); j++) { if (i == terrain.GetLength(0)- 1) terrain[i, j] = 1; } } 111

Chapter 3

Agents

var cleaningEntity = new CleaningRobot(terrain, 0, 0); cleaningEntity.Print(); cleaningEntity.Start(200); cleaningEntity.Print(); var cleaningEntity = new CleaningAgent(terrain, 0, 0); cleaningEntity.Print(); cleaningEntity.Start(200); cleaningEntity.Print(); The cleaning agent marks every visited cell and thus moves faster to the last cell and to the point where its task is complete. The cleaning robot, on the other hand, does not save the state of the environment, so it doesn’t have any internal structure that may help it decide what move should be the correct one and can basically move up and down randomly several times and even in circles. The cleaning agent has a data structure with information on the environment to aid it in applying some logic and making rational decisions, and the cleaning robot does not. As a result of the code shown in Listing 3-8, the random robot is incapable of cleaning the dirt on the last cell, whereas the agent is able to do it in the time given (Figure3-5).

112

Chapter 3

Agents

Figure 3-5. On the left, the result obtained after executing CleaningRobot; on the right, the result after executing CleaningAgent. The first leaves dirt on the last row, while the latter is able to clean it. Thus far in this chapter we have examined agents’ properties and environments and described a practical problem where we could see an agent with state overrunning the cleaning robot presented in the last chapter. In future sections, we’ll study some of the most popular agent architectures.

Agent Architectures Agent architectures represent predefined designs that consider different agent properties, like the ones studied earlier, to provide a scheme or blueprint for building agents. One can think of the different concepts presented so far in an analogy where agents are buildings; their properties are similar to building properties (color, height, material used, etc.); their architecture is what it would be in a building, i.e., the infrastructure supporting it and defining its functionality; and agent types (soon to be detailed) would be as the types of buildings that we have (commercial, governmental, military, etc.). 113

Chapter 3

Agents

Agent architecture as the basis of the agent’s functionality indicates how the agent will function. Up to this moment we have seen the agent’s function as an abstract one; architecture’s being a functionality-defining component will give us a model to implement such a function.

eactive Architectures: Subsumption R Architecture In the same way we could have an illuminated property and luminous architecture—in other words, one that is focused on offering the greatest lightness—we could also have a reactive agent and reactive-based architecture, one that is focused on reactivity above all. This is the case with agent-reactive architectures. In a reactive architecture as it occurs in a reactive agent, each behavior is a mapping from percepts or environment states to actions. In Figure3-6 we can see a diagram showing a reactive architecture.

Figure 3-6. Reactive architecture diagram

114

Chapter 3

Agents

The cleaning agent developed in previous sections is a clear example of reactive architecture. We already know from the agent’s properties section that being purely reactive involves some setbacks: there’s no learning in this type of architecture; it’s usually handcrafted, which makes it very difficult to create large systems; it can be used only for its original purpose, and so on. One of the most popular—and arguably the best known—reactive architectures is the Subsumption architecture, developed by Rodney Brooks in the mid-1980s. His architecture is said to be a behavior-based architecture; it rejected the idea of logic-based agents—i.e., those that rely fully on logic to represent the world, its interactions, and its relations—in an attempt to set a new approach apart from the traditional AI of his time.

Note Behavior-based agents use biological systems as building blocks and rely on adaptability. They tend to show more biological features than their AI counterparts and can repeat actions, make mistakes, demonstrate tenacity, and so forth, sort of like ants do. The main ideas behind Brooks’ architecture are the following: 1. Intelligent behavior can be generated without explicit representations like the ones proposed by symbolic AI. 2. Intelligent behavior can be generated without explicit abstract reasoning of the kind that symbolic AI proposes. 3. Intelligence is an emergent property of certain complex systems.

115

Chapter 3

Agents

The Subsumption architecture possesses two fundamental characteristics: 1. An agent’s decision-making process is executed through a set of task-accomplishing behaviors where each behavior module can be seen as an individual agent function. Because this is a reactive architecture every agent function is a mapping from a percept or state to an action. 2. Behavior modules are intended to achieve a particular task, and each behavior “competes” with others to exercise control over the agent. 3. Many behaviors can fire simultaneously, and the multiple actions proposed by these behaviors are executed according to a subsumption hierarchy, with the behaviors arranged into layers. 4. Lower layers in the hierarchy are able to inhibit higher layers: the lower a layer is the higher is its priority. The principle of the subsumption hierarchy is that higher layers will indicate more abstract behaviors. For instance, considering our cleaning agent, one would like to give a high priority to the “clean” behavior; thus, it’d be encoded in the lower layers where it has a higher priority.

Note Symbolic AI is sometimes called Old Fashioned AI or Good Old Fashioned AI.It was popular in the 1950s and 1960s and was based on the idea of representing knowledge through symbols (logic formulas, graphs, rules, etc.). Hence, methods of Symbolic AI are developed on the basis of logic, theory of formal languages, various areas of discrete mathematics, and so forth. 116

Chapter 3

Agents

Looking again at the cleaning agent, we can see that it follows the Subsumption architecture (Listing 3-9).

Listing 3-9. Cleaning Agent Action Function Follows the Subsumption Architecture public void AgentAction(List percepts) { if (percepts.Contains(Percepts.Clean)) UpdateState(); if (percepts.Contains(Percepts.Dirty)) Clean(); else if (percepts.Contains(Percepts.Finished)) TaskFinished = true; else if (percepts.Contains(Percepts.MoveUp) && !_ cellsVisited.Contains(new Tuple(X- 1, Y))) Move(Percepts.MoveUp); else if (percepts.Contains(Percepts.MoveDown) && !_ cellsVisited.Contains(new Tuple(X + 1, Y))) Move(Percepts.MoveDown); else if (percepts.Contains(Percepts.MoveLeft) && !_ cellsVisited.Contains(new Tuple(X, Y- 1))) Move(Percepts.MoveLeft); else if (percepts.Contains(Percepts.MoveRight) && !_ cellsVisited.Contains(new Tuple(X, Y + 1))) Move(Percepts.MoveRight); else RandomAction(percepts); } The cleaning agent establishes an order for the behaviors exhibited; this order corresponds to the subsumption hierarchy illustrated in Figure3-7. 117

Chapter 3

Agents

Figure 3-7. Subsumption hierarchy for cleaning agent The order of priority established by the subsumption hierarchy in the cleaning agent is 1, 2, 3, 4, 5, 6, and 7, with 7 being the behavior with the highest priority. This architecture inherits the problems of reactive architectures (no learning, hardwired rules, and so on). Beyond that, modeling complex systems requires many behaviors to be included in the hierarchy, making it too extensive and unfeasible. Up to this point we have described agent properties and the reactive architecture, providing an example of one of these (probably the best-known example), the Subsumption architecture. In the next sections, we’ll look at other agent architectures, like the BDI (Belief Desire Intention) and Hybrid architectures.

118

Chapter 3

Agents

Deliberative Architectures: BDI Architecture In a purely deliberative architecture agents follow a goal-based behavior where they are able to reason and plan ahead. Deliberative architectures usually incorporate some sort of symbolic representation of the world via logic, graphs, discreet math, and so forth, and decisions (for example, about what actions to perform) are typically made via logical reasoning using pattern matching and symbolic manipulation. Readers familiar with logical or functional programming languages like Prolog, Haskell, or FSharp may be able to understand the meaning of symbolic a lot easier. Deliberative architectures usually face two problems that need to be solved: 1. Translating the real world into an appropriate, accurate symbolic version of it that is efficient and useful for the purpose of the agent. This problem is usually time-consuming, especially if the environment is too dynamic and changing from time to time. 2. Symbolically representing information about realworld entities, relations, processes, and so forth and how to reason and make decisions with this information. Problem number 1 guided work on face recognition, speech recognition, learning, and so on, and Problem number 2 inspired the work on knowledge representation, automated scheduling, automated reasoning, automatic planning, and so forth. Regardless of the immense volume of scientific material that these problems generated, most researchers accepted the fact that they weren’t even near solved. Even apparently trivial problems, such as essential reasoning, turned out to be exceptionally difficult. The underlying problem seems to be the difficulty of theorem proving in even very simple logics, and the complexity of 119

Chapter 3

Agents

symbol manipulation in general; recall that first-order logic (FOL) is not even decidable, and modal extensions attached to it (including representations of belief, desire, time, and so on) tend to be highly undecidable.

Note The term decidable or decidability relates to the decision problem; i.e., the problem that can be defined as outputting Yes (1) or No (0) to a question on the input values. The satisfiability problem (SAT) is a particular case of decision problem. Thus, we say that a theory (set of formulas) is decidable if there is a method or algorithm for deciding whether a given randomly chosen formula belongs to that theory. The generic deliberative architecture is illustrated in Figure3-8.

Figure 3-8. Deliberative architecture

120

Chapter 3

Agents

Multiple deliberative architectures like BDI (soon to be detailed) find their roots in the philosophical tradition of understanding practical reasoning, the process of deciding moment by moment which action to execute when seeking to fulfill our goals. Human practical reasoning consists of two activities: 1. Deciding what state of affairs we want to achieve (deliberation). 2. Deciding how to achieve these states of affairs (means-end reasoning or planning). From the preceding activities we can conclude that deliberations output intentions and means-end reasoning outputs plans.

Note There is a difference between practical reasoning and theoretical reasoning. The former is directed toward actions, while the latter is directed toward beliefs. Means-end reasoning is the process of deciding how to achieve an end using means available; in the AI world this is known as planning. For the agent to generate a plan it typically requires a representation or goal intention to achieve, a representation of actions it can perform, and a representation of its environment (Figure3-9).

121

Chapter 3

Agents

Figure 3-9. Inputs and output flow of the planning component of an agent How does deliberation occur? In the deliberation process there’s a first step called alternatives generation in which the agent generates a set of alternatives (goals, desires) for consideration. In a second step called filtering the agent chooses between available options and commits to some of them. These chosen options or alternatives are its intentions. The key question in deliberative architectures is “How can the agent deliberate on its (probably conflicting) goals to decide which ones it will pursue?” The answer to this question is provided by the goal-deliberation strategy that is particular to every deliberative architecture; the most popular of these is the BDI architecture created by Michael E.Bratman in his book Intentions, Plans and Practical Reason (1987).

Note Considering their interaction with time, a reactive architecture exists in the present (with short duration), while a deliberative architecture reasons about the past and projects (plans, etc.) into the future.

122

Chapter 3

Agents

The Beliefs, Desires, and Intentions (BDI) architecture contains explicit representations of an agent’s beliefs, desires, and intentions. Beliefs (what it thinks) are generally regarded as the information an agent has about its environment; we could say knowledge instead of belief, but we would rather use the more general term belief because what the agent believes may be false sometimes. Desires (what it wants) are those things the agent would like to see achieved, we don’t expect an agent to act on all its desires. Intentions (what it is doing) are those things the agent is committed to doing, and they are basically the result of filtering desires; the BDI architecture is illustrated in Figure3-10.

Figure 3-10. BDI architecture

123

Chapter 3

Agents

Beliefs are usually described by predicates outputting True or False values (for example, IsDirty(x,y)) and represent the internal knowledge the agent has of the world. Desires are fulfilled when they are present in the belief base (or manually removed by the agent). Like the belief base, the desire base is updated during the execution of the agent. Desires can be related by hierarchical links (sub/super desires) when a desire is created as an intermediary goal (for example, to clean dirt on a terrain one could have two subdesires or subgoals: move to every dirty cell and clean it). Desires have a priority value that can change dynamically and is used to select a new intention from among the set of desires when necessary. Once the agent considers all its options it must commit to some of them, in this case and as an example it will commit to just one, to its only available option, which later becomes its intention. Intentions eventually lead to actions, and the agent is supposed to act by trying to achieve its intentions. The agent is supposed to make reasonable attempts to achieve its intentions, and it may follow a sequence of actions (plan) for this purpose. The intention chosen by the agent will constrain its practical reasoning from that point on; once a commitment to an intention exists the agent will not contemplate other intentions that are conflicting with the ones already set in motion. Intentions can be put on hold (for example, when they require a subdesire to be achieved). For this reason, there is a stack of intentions; the last one is the current intention and the only one that is not on hold. Intentions should be persistent; in other words, we must devote every available resource to fulfilling them and not drop them immediately if they aren’t achieved in the short run, because then we will be achieving none all the time. On the other hand, intentions can’t persist for too long, because there might be a logical reason to drop them. For example, there may come a time when the cleaning agent has nothing else to do (clean), maybe because it inhabits a multi-agent environment and other agents have finished the cleaning task. 124

Chapter 3

Agents

Intentions make up a set of important roles associated with practical reasoning: •

Intentions motivate planning: Once an agent has decided to achieve an intention it must plan a course of action to accomplish that intention.

•

Intentions constrain future deliberation: Once an agent commits to an intention it will not contemplate other intentions that are conflicting with the chosen intention.

•

Intentions persist: The agent will not renounce its intentions without any rational cause; it will persist typically until either the agent believes it has successfully achieved them or it believes it cannot achieve them, or because the purpose for the intention is no longer present.

•

Intentions influence beliefs upon the future: Once the agent adopts certain intentions, some planning for the future under the assumption that those intentions chosen will be achieved is necessary and logical.

From time to time it is important for the agent to stop and reconsider its intentions, as some could have become irrational or impossible. This reconsideration stage implies a cost at both spatial and temporal lines, and it also presents us with a problem: •

A bold agent that doesn’t stop enough to reconsider its intentions might be trying to achieve an intention that is no longer possible.

•

A cautious agent that stops too frequently to reconsider its intentions might be spending too many resources on the reconsideration stage and not enough on achieving its intentions. 125

Chapter 3

Agents

A balance or tradeoff between the event-driven and goal-directed behaviors of the agent is the solution for this dilemma.

Note Experiments have demonstrated that bold agents do better than cautious agents in environments that don’t change too often. In the other scenario (environment changes frequently), cautious agents outperform bold agents. The process of practical reasoning in a BDI agent relies on the following components. In the next points B is assumed to be the set of beliefs, D the set of desires, and I the set of intentions: •

A set of current beliefs representing information the agent has about its environment

•

A belief revision function (brf ) that receives percepts and the agent’s beliefs as inputs and determines a new set of beliefs: brf: P x B -> B

•

An option-generation function (options) that receives beliefs about its environment and intentions (if any) as inputs and determines the options (desires) of the agent: options: B x I -> D

•

A set of current options representing probable courses of action for the agent to follow

•

A filter function (filter) representing the deliberation process of the agent and using beliefs, desires, and intentions as inputs to determine the agent’s intentions: filter: B x D x I -> I

126

Chapter 3

•

A set of current intentions representing the agent’s commitments

•

An action-selection function that uses current intentions as inputs to determine an action to perform

Agents

It comes as no surprise that the state of a BDI agent at any moment is a triple (B, D, I). The BDI agent’s action function seems pretty simple when we don’t get into details; it’s shown in the next pseudocode here: function AgentAction(P): B = brf(P, B) D = options(D, I) I = filter(B, D, I) end In the next chapter we’ll present a practical problem where we’ll develop an AI for a Mars Rover whose architecture will be BDI; this problem will help us set firm ground for many of the concepts introduced during this section.

Hybrid Architectures Multiple researchers have argued that neither a purely deliberative agent nor a purely reactive agent is a good strategy when we design an agent. Hybrid architectures in which the agent possesses both a goal-based component where they are able to reason and plan ahead and a reactive component that allows them to react immediately to situations of the environment are usually preferred over the alternative of a purely deliberative or purely reactive agent.

127

Chapter 3

Agents

In general, hybrid architecture agents are composed of the following subsystems or components: •

Deliberative component: contains a representation of the world that can be at some level symbolic; it builds plans and makes decisions as in the deliberative architecture

•

Reactive component: capable of reacting to certain situations without complex reasoning (situation -> consequence rules)

Thus, hybrid agents have reactive and proactive properties, and the reactive component is usually given some precedence over the deliberative one. The divided and somewhat hierarchical structure where reactive and deliberative components coexist has lead to the natural idea of layering architectures, which represents the hybrid agents’ design. In this type of architecture, an agent’s control components are arranged into a hierarchy, with higher layers dealing with information at higher levels of abstraction. Typically, we will have at least two layers in a layered architecture: one to deal with the reactive behavior and one to deal with the proactive behavior. In practice, there is no reason why there couldn’t be more layers. Generally speaking, we can count two types of layered architectures: •

128

Horizontal layering: In horizontally layered architectures, the agent’s layers are each directly connected to the sensory input and action output. As a result, each layer acts like an agent, producing suggestions as to what action to perform.

Chapter 3

•

Agents

Vertical layering: In vertically layered architectures, sensory input and action output are each processed through every layer in one or possibly various directions.

Both horizontal and vertical layering are illustrated in Figure3-11.

Figure 3-11. Horizontally layered architecture (on the left) and vertically layered architecture (on the right). Note that in vertically layered architectures there could be more than just one pass through every layer. Horizontally layered architectures are very simple in their conceptual design; agents exhibiting n behaviors will require n layers, one for each behavior. Despite this positive point, the fact that each layer is actually competing with others to suggest an action could cause the agent to show incoherent behavior. In order to provide consistency, a mediator function is usually required to act as “middle man” and decide which layer controls the agent at any given moment.

129

Chapter 3

Agents

The mediator function involves high complexity, as all possible interactions between all layers must be considered to finally output an action. Creating such a control mechanism is extremely difficult from a designer’s point of view. In vertically layered architectures these problems are diminished because there’s an order between layers, and the last layer is the one outputting the action to be executed. Vertically layered architectures are usually divided into two types: one-pass architectures and two-pass architectures. In the former type, the agent’s decision-making process flows sequentially through each layer until the last layer generates an action. In two-pass architectures, information flows up the architecture (the first pass) and then back down. There exist some remarkable similarities between the principle of two-pass vertically layered architectures and the way organizations and enterprises work in the sense that information flows up to the highest levels and orders then flow down. In both one-pass and two-pass vertically layered architectures the complexity of interactions between layers is reduced. Since there are n- 1 edges between n layers, if each layer is capable of suggesting m actions, there are at most m2(n − 1) interactions to be considered between layers. Clearly, this is a much simpler level of interaction than the one a horizontally layered architecture forces us to have. This simplicity comes at a cost, and that cost is flexibility. In order for a vertically layered architecture to make a decision, control must pass between each different layer. Vertically layered architectures are not flawless, and failures in any layer can have serious consequences for an agent’s performance. In the next section we’ll study a particular case of horizontally layered architecture: touring machines.

130

Chapter 3

Agents

Touring Machines Touring machines represent horizontally layered architectures composed of three layers (modeling layer, planning layer, and the reactive layer). Figure3-12 illustrates a touring machine.

Figure 3-12. Touring machine The reactive layer provides immediate responses to changes detected in the environment as a set of situation action rules resembling those of the Subsumption architecture. In the next pseudocode we illustrate a reactive rule of an autonomous vehicle agent. This example shows the obstacle- avoidance rule of the vehicle: rule-1: obstacle-avoidance if (in_front(vehicle, observer) andspeed(observer) > 0 andseparation(vehicle, observer) next.Item1) return TypesAction.MoveUp; if (_rover.X < next.Item1) return TypesAction.MoveDown; if (_rover.Y < next.Item2) return TypesAction.MoveRight; if(_rover.Y > next.Item2) return TypesAction.MoveLeft; return TypesAction.None; } public void BuildPlan(Tuple source, Tuple dest) { switch (Name) { case TypesPlan.PathFinding: Path = PathFinding(source.Item1, source.Item2, dest.Item1, dest.Item2). Item2; break; } }

154

Chapter 4

Mars Rover

private Tuple PathFinding(int x1, int y1, int x2, int y2) { var queue = new Queue(); queue.Enqueue(new Tuple(new Tuple(x1, y1), new List())); var hashSetVisitedCells = new HashSet(); while(queue.Count > 0) { var currentCell = queue.Dequeue(); var currentPath = currentCell.Item2; hashSetVisitedCells.Add(currentCell.Item1); var x = currentCell.Item1.Item1; var y = currentCell.Item1.Item2; if (x == x2 && y == y2) return currentCell; // Up if (_rover.MoveAvailable(x- 1, y) && !hashSetVisitedCells.Contains(new Tuple(x- 1, y))) { var pathUp = new List(currentPath); pathUp.Add(new Tuple(x- 1, y)); queue.Enqueue(new Tuple(new Tuple(x- 1, y), pathUp)); } 155

Chapter 4

Mars Rover

// Down if (_rover.MoveAvailable(x + 1, y) && !hashSetVisitedCells.Contains(new Tuple(x + 1, y))) { var pathDown = new List(currentPath); pathDown.Add(new Tuple(x + 1, y)); queue.Enqueue(new Tuple(new Tuple(x + 1, y), pathDown)); } // Left if (_rover.MoveAvailable(x, y- 1) && !hashSetVisitedCells.Contains(new Tuple(x, y- 1))) { var pathLeft = new List(currentPath); pathLeft.Add(new Tuple(x, y- 1)); queue.Enqueue(new Tuple(new Tuple(x, y- 1), pathLeft)); } // Right if (_rover.MoveAvailable(x, y + 1) && !hashSetVisitedCells.Contains(new Tuple(x, y + 1))) { var pathRight = new List(currentPath); pathRight.Add(new Tuple(x, y + 1)); 156

Chapter 4

Mars Rover

queue.Enqueue(new Tuple(new Tuple(x, y + 1), pathRight)); } } return null; } public bool FulFill() { return Path.Count == 0; } } The Percept class is very simple; we are merely using it to make it easier for us to know where a percept has occurred. By using this class we can save the percept location. The Plan class, on the other hand, is a bit more complicated. The Plan class contains a property List Path, which defines the Path the agent created as a result of executing a plan; in this case, a path-finding plan. The BuildPlan() method will allow us to build different types of plans. It’s supposed to act as a plan-selection mechanism. The NextAction() method updates the Path property by returning and deleting the next action to execute in the present plan. Finally, the PathFinding() method implements the Breadth First Search (BFS) algorithm for finding the optimal route from a given source to a given destination or location in the terrain. We’ll see more of this algorithm in a future chapter; for now let us consider it an essential algorithm for different graph-related tasks and remember that it starts at the source, discovering new steps of the path from source to destination and escalating by levels (Figure4-3). For this purpose it uses a queue for enqueuing all non-visited neighbors of the cell being examined at the time. 157

Chapter 4

Mars Rover

The FulFill() method determines when a plan has been completely executed.

Figure 4-3. BFS is capable of discovering paths by levels; S is the source and D the destination. Each numbered cell determines a level in the search; i.e., level 1, 2, etc. Now that we have gotten acquainted with all the classes that our Mars Rover will be using, let’s dive into the Mars Rover AI code. Resembling the method implemented for the agent from Chapter 3, our Mars Rover includes a GetPercepts() method (Listing 4-6) that provides a list of percepts perceived by the agent at the current time and in its radius of sight.

Listing 4-6. GetPercepts() Method public List GetPercepts() { var result = new List(); if (MoveAvailable(X- 1, Y)) result.Add(new Percept(new Tuple (X- 1, Y), TypePercept.MoveUp));

158

Chapter 4

Mars Rover

if (MoveAvailable(X + 1, Y)) result.Add(new Percept(new Tuple (X + 1, Y), TypePercept.MoveDown)); if (MoveAvailable(X, Y- 1)) result.Add(new Percept(new Tuple (X, Y- 1), TypePercept.MoveLeft)); if (MoveAvailable(X, Y + 1)) result.Add(new Percept(new Tuple(X, Y + 1), TypePercept.MoveRight)); result.AddRange(LookAround()); return result; } The GetPercepts() method makes use of the MoveAvailable() and LookAround() methods, both illustrated in Listing 4-7.

Listing 4-7. MoveAvailable() and LookAround() Methods public bool MoveAvailable(int x, int y) { return x >= 0 && y >= 0 && x < _terrain. GetLength(0) && y < _terrain.GetLength(1) && _terrain[x, y] < RunningOverThreshold; } private IEnumerable LookAround() { return GetCurrentTerrain(); }

159

Chapter 4

Mars Rover

Since we want to code our Mars Rover to be as generic as possible in the way it “looks around” (one may have a different definition of what it is to look around), the final implementation of this functionality is given by the GetCurrentTerrain() method shown in Listing 4-8.

Listing 4-8. GetCurrentTerrain() Method public IEnumerable GetCurrentTerrain() { var R = SenseRadius; CurrentTerrain.Clear(); var result = new List(); for (var i = X- R > 0 ? X- R : 0; i = _terrain.GetLength(0)) break; // In the circle result.AddRange(CheckTerrain(Mars. TerrainAt(i, j), new Tuple(i, j))); CurrentTerrain.Add(new Tuple(i, j)); UpdatePerceivedCellsDicc(new Tuple(i, j)); } } return result; } The method from Listing 4-8 includes several loops that depend on the circle circumference formula:

( x - h) + ( y - k ) 2

2

= r2

where (h, k) represent the center of the circle, in this case the agent’s location; r represents the radius of the circle, or in this case the SenseRadius. These loops allow the rover to track every cell at distance SenseRadius of its current location. Within these loops we make calls to the UpdatePerceivedCellsDicc() and CheckTerrain() methods (Listing 4-9). The first simply updates the visited cells dictionary that we use in the Statistics and Probability component to inject new beliefs to the rover. The latter checks a given cell from the terrain to see if it’s an obstacle or a water location. It also updates the internal _terrain data structure the rover has initially and maintains later by updating the value that corresponds to the perceived coordinate. 161

Chapter 4

Mars Rover

Listing 4-9. UpdatePerceivedCellsDicc() and CheckTerrain() Methods private void UpdatePerceivedCellsDicc(Tuple position) { if (!_perceivedCells.ContainsKey(position)) _perceivedCells.Add(position, 0); _perceivedCells[position]++; } private IEnumerable CheckTerrain (double cell, Tuple position) { var result = new List(); if (cell > RunningOverThreshold) result.Add(new Percept(position, TypePercept.Obstacle)); else if (cell < 0) result.Add(new Percept(position, TypePercept.WaterSpot)); // Update the rover's internal terrain _terrain[position.Item1, position.Item2] = cell; return result; } The method responsible for generating the next action to be executed by the rover is the Action() method shown in Listing 4-10.

162

Chapter 4

Mars Rover

Listing 4-10. Action() Method public TypesAction Action(List percepts) { // Reactive Layer if (Mars.WaterAt(X, Y) && !WaterFound.Contains (new Tuple(X, Y))) return TypesAction.Dig; var waterPercepts = percepts.FindAll(p => p.Type == TypePercept.WaterSpot); if (waterPercepts.Count > 0) { foreach (var waterPercept in waterPercepts) { var belief = Beliefs.FirstOrDefault(b => b.Name == TypesBelief.PotentialWaterSpots); List pred; if (belief != null) pred = belief.Predicate as List; else { pred = new List {waterPercept.Position}; Beliefs.Add(new Belief(TypesBelief. PotentialWaterSpots, pred)); } if (!WaterFound.Contains (waterPercept.Position)) pred.Add(waterPercept.Position);

163

Chapter 4

Mars Rover

else { pred.RemoveAll( t => t.Item1 == waterPercept. Position.Item1 && t.Item2 == waterPercept.Position.Item2); if (pred.Count == 0) Beliefs.RemoveAll(b => (b.Predicate as List).Count == 0); } } if (waterPercepts.Any(p => !WaterFound. Contains(p.Position))) CurrentPlan = null; } if (Beliefs.Count == 0) { if (_wanderTimes == WanderThreshold) { _wanderTimes = 0; InjectBelief(); } _wanderTimes++; return RandomMove(percepts); } if (CurrentPlan == null || CurrentPlan.FullFill()) { // Deliberative Layer Brf(percepts);

164

Chapter 4

Mars Rover

Options(); Filter(); } return CurrentPlan.NextAction(); } In this method we incorporate the reactive and deliberative layers of the agent. The first lines correspond to the reactive layer, and different scenarios are considered that demand an Fimmediate response: 1. There’s water at the current location of the rover, and that spot has not been discovered before. 2. There’s a percept of a possible water location in the surrounding areas (defined by the circle with radius SenseRadius) of the rover. In this case, and always checking that the possible water location has not been already found, we add a water belief to the rover. 3. If the water location perceived at step 2 has not been previously found then the current plan is deleted. A new one considering the new belief will be built. 4. If the rover has no beliefs it will execute a random action (Listing 4-11); i.e., wanders around. Once this “wandering around” reaches a certain number of actions (ten, in this case) then a belief is injected. The four previous steps make up the reactive layer of our agent; the last part of the method composed of the Brf(), Options(), and Filter() methods represent the deliberative layer (BDI architecture). The InjectBelief() method is also part of this deliberative layer as it involves a “deliberative” process where the agent decides its next course of action.

165

Chapter 4

Mars Rover

Listing 4-11. RandomMove() Method private TypesAction RandomMove(List percepts) { var moves = percepts.FindAll(p => p.Type. ToString().Contains("Move")); var selectedMove = moves[_random.Next(0, moves.Count)]; switch (selectedMove.Type) { case TypePercept.MoveUp: return TypesAction.MoveUp; case TypePercept.MoveDown: return TypesAction.MoveDown; case TypePercept.MoveRight: return TypesAction.MoveRight; case TypePercept.MoveLeft: return TypesAction.MoveLeft; } return TypesAction.None; } The Statistics and Probability component of the rover, and the one that allows it to inject beliefs based on its past history, is represented by the InjectBelief() method, which can be seen in Listing 4-12 along with its helper methods.

166

Chapter 4

Mars Rover

Listing 4-12. InjectBelief(), SetRelativeFreq(), and RelativeFreq() Methods private void InjectBelief() { var halfC = _terrain.GetLength(1) / 2; var halfR = _terrain.GetLength(0) / 2; var firstSector = _perceivedCells.Where(k => k.Key. Item1 < halfR && k.Key.Item2 < halfC).ToList(); var secondSector = _perceivedCells.Where(k => k.Key. Item1 < halfR && k.Key.Item2 >= halfC).ToList(); var thirdSector = _perceivedCells.Where(k => k.Key. Item1 >= halfR && k.Key.Item2 < halfC).ToList(); var fourthSector = _perceivedCells.Where(k => k.Key. Item1 >= halfR && k.Key.Item2 >= halfC).ToList(); var var var var

freq1stSector freq2ndSector freq3rdSector freq4thSector

= = = =

SetRelativeFreq(firstSector); SetRelativeFreq(secondSector); SetRelativeFreq(thirdSector); SetRelativeFreq(fourthSector);

var min = Math.Min(freq1stSector, Math. Min(freq2ndSector, Math.Min(freq3rdSector, freq4thSector))); if (min == freq1stSector) Beliefs.Add(new Belief(TypesBelief. PotentialWaterSpots, new List { new Tuple(0, 0) }));

167

Chapter 4

Mars Rover

else if (min == freq2ndSector) Beliefs.Add(new Belief(TypesBelief.Potential WaterSpots, new List { new Tuple(0, _terrain.GetLength(1)- 1) })); else if (min == freq3rdSector) Beliefs.Add(new Belief(TypesBelief.Potential WaterSpots, new List { new Tuple(_terrain.GetLength(0)- 1, 0) })); else Beliefs.Add(new Belief(TypesBelief.Potential WaterSpots, new List { new Tuple(_terrain.GetLength(0)- 1, _terrain.GetLength(1)- 1) })); } private double SetRelativeFreq(List cells) { var result = 0.0; foreach (var cell in cells) result += RelativeFrequency(cell.Value, cells.Count); return result; } private double RelativeFrequency(int absFreq, int n) { return (double) absFreq/n; }

168

Chapter 4

Mars Rover

As it was detailed in the last section, the relative frequency is calculated for every cell of a given sector and then summed up in the SetRelativeFreq() method to obtain the total frequency of the group of cells. Note that in this case we decided to divide the terrain into four equal sectors, but you may decide to do it in as many sectors as you deem necessary or to the level of detail you believe necessary, like you would do in a QuadTree. One could even decide to divide the terrain into a certain number of sectors considering the SenseRadius of the rover and the time it wanders around. These values are all related, and most of them are considered in the heuristics attached to the rover. In this case—and seeking simplicity in the example proposed—we choose to attach truly naïve heuristics for the rover; for instance, always injecting a water belief at a corner of the selected sector could be a bad idea in different scenarios, as it’s not going to work well every time. Thus, the sector selection and cell-within-sector selection mechanisms need to be more generic for the rover to perform well in multiple environments. Let’s keep in mind that the heuristics presented here can be greatly improved, and as a result the rover will improve its performance.

Note A QuadTree is a tree data structure where each internal node has exactly four children. They are often used to partition a two- dimensional space or region by recursively subdividing it into four quadrants or regions. Lastly, let’s examine the deliberative layer and all its methods, starting with the Beliefs Revision Function (Listing 4-13).

169

Chapter 4

Mars Rover

Listing 4-13. Brf() Method public void Brf(List percepts) { var newBeliefs = new List(); foreach (var b in Beliefs) { switch (b.Name) { case TypesBelief.PotentialWaterSpots: var waterSpots = new List(b.Predicate); waterSpots = UpdateBelief(TypesBelief. PotentialWaterSpots, waterSpots); if (waterSpots.Count > 0) newBeliefs.Add(new Belief(TypesBelief. PotentialWaterSpots, waterSpots)); break; case TypesBelief.ObstaclesOnTerrain: var obstacleSpots = new List(b.Predicate); obstacleSpots = UpdateBelief (TypesBelief.ObstaclesOnTerrain, obstacleSpots); if (obstacleSpots.Count > 0) newBeliefs.Add(new Belief (TypesBelief.ObstaclesOnTerrain, obstacleSpots)); break; } } Beliefs = new List(newBeliefs); } 170

Chapter 4

Mars Rover

In the Brf() method we examine every belief (possible water locations, possible obstacle locations) and update them, creating a new set of beliefs. The UpdateBelief() method is illustrated in Listing 4-14.

Listing 4-14. UpdateBelief() Method private List UpdateBelief(TypesBelief belief, IEnumerable beliefPos) { var result = new List(); foreach (var spot in beliefPos) { if (CurrentTerrain.Contains(new Tuple(spot.Item1, spot.Item2))) { switch (belief) { case TypesBelief.PotentialWaterSpots: if (_terrain[spot.Item1, spot. Item2] >= 0) continue; break; case TypesBelief.ObstaclesOnTerrain: if (_terrain[spot.Item1, spot. Item2] < RunningOverThreshold) continue; break; } } result.Add(spot); } return result; } 171

Chapter 4

Mars Rover

In the UpdateBelief() method we check every belief against the currently perceived terrain. If there’s a wrong belief—like, for instance, we thought or believed we would find water at location (x, y) and it happens that we were just there and there’s nothing—then that belief must be deleted. The Options() method, which is responsible for generating desires, is shown in Listing 4-15.

Listing 4-15. Options() Method public void Options() { Desires.Clear(); foreach (var b in Beliefs) { if (b.Name == TypesBelief.PotentialWaterSpots) { var waterPos = b.Predicate as List; waterPos.Sort(delegate(Tuple tupleA, Tuple tupleB) { var distA = Manhattan Distance(tupleA, new Tuple(X, Y)); var distB = Manhattan Distance(tupleB, new Tuple(X, Y)); if (distA < distB) return 1; 172

Chapter 4

Mars Rover

if (distA > distB) return -1; return 0; }); foreach (var wPos in waterPos) Desires.Enqueue(new Desire (TypesDesire.FindWater, new Desire (TypesDesire.GotoLocation, new Desire (TypesDesire.Dig, wPos)))); } } } We will consider only one type of desire—the desire to find water at specific locations. Thus, using the set of beliefs as a base, we generate desires and sort them by proximity using the distance (Listing 4-16) as the proximity measure.

Listing 4-16. Manhattan Distance public int ManhattanDistance(Tuple x, Tuple y) { return Math.Abs(x.Item1- y.Item1) + Math.Abs(x.Item2- y.Item2); } Using the set of desires, we push new intentions into our Intentions set in the Filter() method; if there’s no plan in motion for the current intention then we choose one using the ChoosePlan() method (Listing 4-17).

173

Chapter 4

Mars Rover

Listing 4-17. Filter() and ChoosePlan() Methods private void Filter() { Intentions.Clear(); foreach (var desire in Desires) { if (desire.SubDesires.Count > 0) { var primaryDesires = desire. GetSubDesires(); primaryDesires.Reverse(); foreach (var d in primaryDesires) Intentions.Push(Intention. FromDesire(d)); } else Intentions.Push(Intention. FromDesire(desire)); } if (Intentions.Any() && !ExistsPlan()) ChoosePlan(); } private void ChoosePlan() { var primaryIntention = Intentions.Pop(); var location = primaryIntention.Predicate as Tuple;

174

Chapter 4

Mars Rover

switch (primaryIntention.Name) { case TypesDesire.Dig: CurrentPlan = PlanLibrary.First(p => p.Name == TypesPlan.PathFinding); CurrentPlan.BuildPlan(new Tuple(X, Y), location); break; } } To conclude, the ExistsPlan() method determines if there’s a plan in motion, and the ExecuteAction() method executes the action selected by the agent (Listing 4-18). The latter method is also responsible for updating the WaterFound data structure with the locations where water has been found.

Listing 4-18. ExistsPlan() and ExecuteAction() Methods public bool ExistsPlan() { return CurrentPlan != null && CurrentPlan.Path. Count > 0; } public void ExecuteAction(TypesAction action, List percepts) { switch (action) { case TypesAction.MoveUp: X -= 1; break;

175

Chapter 4

Mars Rover

case TypesAction.MoveDown: X += 1; break; case TypesAction.MoveLeft: Y -= 1; break; case TypesAction.MoveRight: Y += 1; break; case TypesAction.Dig: WaterFound.Add(new Tuple(X, Y)); break; } } In the next section, we’ll take a look at our Mars Rover in action as it is executed in a Windows Forms Application that we created for experimenting and seeing how its AI works on a test world.

Mars Rover Visual Application As mentioned at the beginning of this chapter, we created a Windows Forms application with which to test our Mars Rover and see how it would do on a test Mars world with hidden water locations and obstacles along the way. This example will not only help us to understand how to set up the MarsRover and Mars classes, but it will also demonstrate how the AI presented during this chapter will perform its decision-making process under different scenarios. The complete details of the Windows Form application (Listing 4-19) are beyond the scope of this book; we will simply present a fragment of it to illustrate to readers where the graphics are coming from. For further reference, the source code associated with this book can be consulted. 176

Chapter 4

Mars Rover

Listing 4-19. Fragment of Windows Forms Visual Application Code public partial class MarsWorld : Form { private MarsRover _marsRover; private Mars _mars; private int _n; private int _m; public MarsWorld(MarsRover rover, Mars mars, int n, int m) { InitializeComponent(); _marsRover = rover; _mars = mars; _n = n; _m = m; } private { var var var var var

void TerrainPaint(object sender, PaintEventArgs e) pen = new Pen(Color.Wheat); waterColor = new SolidBrush(Color.Aqua); rockColor = new SolidBrush(Color.Chocolate); cellWidth = terrain.Width/_n; cellHeight = terrain.Height/_m;

for (var i = 0; i < _n; i++) e.Graphics.DrawLine(pen, new Point(i * cellWidth, 0), new Point(i * cellWidth, i * cellWidth + terrain.Height)); for (var i = 0; i < _m; i++) e.Graphics.DrawLine(pen, new Point(0, i * cellHeight), new Point(i * cellHeight + terrain.Width, i * cellHeight)); 177

Chapter 4

Mars Rover

if (_marsRover.ExistsPlan()) { foreach (var cell in _marsRover.CurrentPlan.Path) { e.Graphics.FillRectangle(new SolidBrush (Color.Yellow), cell.Item2 * cellWidth, cell.Item1 * cellHeight, cellWidth, cellHeight); } } for (var i = 0; i < _n; i++) { for (var j = 0; j < _m; j++) { if (_mars.TerrainAt(i, j) > _marsRover. RunningOverThreshold) e.Graphics.DrawImage(new Bitmap("obstacle-transparency.png"), j*cellWidth, i*cellHeight, cellWidth, cellHeight); if (_mars.WaterAt(i, j)) e.Graphics.DrawImage(new Bitmap("water- transparency.png"), j * cellWidth, i * cellHeight, cellWidth, cellHeight); // Draw every belief in white foreach (var belief in _marsRover.Beliefs) { var pred = belief.Predicate as List; if (pred != null && !pred.Contains(new Tuple(i, j))) continue; 178

Chapter 4

Mars Rover

if (belief.Name == TypesBelief. ObstaclesOnTerrain) { e.Graphics.DrawImage(new Bitmap("obstacle-transparency. png"), j * cellWidth, i * cellHeight, cellWidth, cellHeight); e.Graphics.DrawRectangle(new Pen(Color.Gold, 6), j * cellWidth, i * cellHeight, cellWidth, cellHeight); } if (belief.Name == TypesBelief. PotentialWaterSpots) { e.Graphics.DrawImage(new Bitmap("water-transparency.png"), j * cellWidth, i * cellHeight, cellWidth, cellHeight); e.Graphics.DrawRectangle(new Pen(Color.Gold, 6), j * cellWidth, i * cellHeight, cellWidth, cellHeight); } } } } e.Graphics.DrawImage(new Bitmap("rovertransparency.png"), _marsRover.Y * cellWidth, _marsRover.X * cellHeight, cellWidth, cellHeight);

179

Chapter 4

Mars Rover

var sightColor = Color.FromArgb(80, Color.Lavender); _marsRover.GetCurrentTerrain(); foreach (var cell in _marsRover.CurrentTerrain) e.Graphics.FillRectangle(new SolidBrush (sightColor), cell.Item2 * cellWidth, cell. Item1 * cellHeight, cellWidth, cellHeight); } private void TimerAgentTick(object sender, EventArgs e) { var percepts = _marsRover.GetPercepts(); agentState.Text = "State: Thinking ..."; agentState.Refresh(); var action = _marsRover.Action(percepts); _marsRover.ExecuteAction(action, percepts); var beliefs = UpdateText(beliefsList, _marsRover. Beliefs); var desires = UpdateText(beliefsList, _marsRover. Desires); var intentions = UpdateText(beliefsList, _marsRover.Intentions); if (beliefs != beliefsList.Text) beliefsList.Text = beliefs; if (desires != desiresList.Text) desiresList.Text = desires; if (intentions != intentionsList.Text) intentionsList.Text = intentions; foreach (var wSpot in _marsRover.WaterFound)

180

Chapter 4

Mars Rover

{ if (!waterFoundList.Items.Contains(wSpot)) waterFoundList.Items.Add(wSpot); } Refresh(); } private string UpdateText(RichTextBox list, IEnumerable elems) { var result = ""; foreach (var elem in elems) result += elem; return result; } private void PauseBtnClick(object sender, EventArgs e) { if (timerAgent.Enabled) { timerAgent.Stop(); pauseBtn.Text = "Play"; } else { timerAgent.Start(); pauseBtn.Text = "Pause"; } } }

181

Chapter 4

Mars Rover

From this code we may notice that the visual application consists of a grid where we have included Play/Pause buttons and used a timer to control rover actions and execute them every second. In order to set up our Mars Rover and world we would need to define a set of initial beliefs, a terrain for the rover, and a real terrain of Mars (Listing 4-20).

Listing 4-20. Setting Up the Mars Rover and World var water = new List { new Tuple (1, 2), new Tuple (3, 5), }; var obstacles = new List { new Tuple (2, 2), new Tuple (4, 5), }; var beliefs = new List { new Belief(TypesBelief.PotentialWaterSpots, water), new Belief(TypesBelief.ObstaclesOnTerrain, obstacles), }; var marsTerrain = new [,] { {0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, {0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, {0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, {0, 0, 0.8, -1, 0, 0, 0, 0, 0, 0}, {0, 0, 0.8, 0, 0, 0, 0, 0, 0, 0}, 182

Chapter 4

{0, {0, {0, {0, {0, }; var roverTerrain = new [,] { {0, {0, {0, {0, {0, {0, {0, {0, {0, {0, };

Mars Rover

0, 0, 0, 0, 0, 0, 0, 0, 0}, 0, 0, 0, 0, 0, 0, 0, 0, 0}, 0, 0, 0, 0, 0, 0, 0, 0, 0}, 0, 0, 0, 0, 0.8, 0, 0, 0, 0}, 0, 0, 0, 0, 0, 0, 0, 0, 0}

0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.8, 0, 0, 0, 0, 0, 0, 0, 0.8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,

0}, 0}, 0}, 0}, 0}, 0}, 0}, 0}, 0}, 0}

var mars = new Mars(marsTerrain); var rover = new MarsRover(mars, roverTerrain, 7, 8, beliefs, 0.75, 2); Application.EnableVisualStyles(); Application.SetCompatibleTextRenderingDefault(false); Application.Run(new MarsWorld(rover, mars, 10, 10)); Once we run the application, a GUI like the one illustrated in Figure4-4 will show up. In this program, one can easily differentiate water locations (water drops images) from obstacle locations (rocks images).

183

Chapter 4

Mars Rover

Figure 4-4. Windows Forms application showing the rover, its SenseRadius, beliefs of water locations and obstacles marked as yellow squares, and actual water and obstacle locations without any yellow square surrounding them Notice the light-color cells surrounding the rover at all times; these are the cells that the rover can “see” or perceive at any given moment and are defined by the SenseRadius parameter (defined as a [Manhattan distance] value of 2in the setup code) and the “discrete” circle whose radius is precisely the SenseRadius and whose center is the rover’s current location. On the right side of the application we have a panel with various information sections, such as Beliefs, Desires, Intentions, WaterFoundAt. All of these are Windows Forms controls and ultimately use the ToString() overrides presented in the last section. The time to see our Mars Rover agent in action has come. Let’s see what happens when we run the application (Figure4-5).

184

Chapter 4

Mars Rover

Figure 4-5. The rover creates a plan to go to location (3, 5), its closest probable water location, and so it creates a plan or sequence of actions (denoted in yellow cells) to get there and dig in. Notice that the plan (sequence of actions) or path returned by our path-finding algorithm is denoted in yellow with the purpose of making it easier for us to comprehend where the rover is going and why. In this case, the rover is going after its closest water-location belief. Once it gets there (Figure4-6), it discovers that its belief was wrong and there was no water in the pursued location as there was no obstacle in a cell adjacent to that water-location belief. The good news is that while exploring that area the rover perceived a water location nearby (in its sensing circle) and so it adventures to go there to find out more.

185

Chapter 4

Mars Rover

Figure 4-6. The rover perceives a water location while exploring a belief and finds the first water location on Mars The previous location sought by the agent is a water location, so the WaterFound data structure is updated, and the rover has found water on Mars! Afterward, it continues pursuing its next belief (Figure4-7): water at (1, 2). Once again when approaching (entering its perception or sense radius), the next water-location belief is discarded by the agent as well as another obstacle-location belief, and so the beliefs set is updated.

186

Chapter 4

Mars Rover

Now that the rover has exhausted its beliefs set it will wander around (during ten actions; it was hardwired like that in the code, see Figure4-8) until our Statistics and Probability deliberative component is activated and causes the rover to inject itself with a new belief that is drawn from logical conclusions. In this case—and imitating what our human mind would do, because we are merely trying to mimic what a human would do in this situation—we would think that it’s more likely, or that our chances of finding water are far greater, in an unexplored area. In the “Heuristics and Metaheuristics” chapter 14 we will see that this concept is known as diversification and is very common in metaheuristics such as Genetic Algorithm, Tabu Search, and so on.

Figure 4-7. The rover discards both a water-location belief and an obstacle-location belief

187

Chapter 4

Mars Rover

Figure 4-8. The rover wanders around after having exhausted its beliefs set In the same way we can have a diversification stage to explore poorly visited or unexplored areas of the terrain we can also have an intensification stage to better explore areas where water has been previously found; that is, promising areas of the terrain. In our case the intensification phase could involve having the rover wander around in some sector of the terrain. As we shall see in future chapters, finding a balance between the intensification and diversification stages (sometimes called the explore– exploit tradeoff ) in search-related problems is essential, and most problems we face in our daily lives are search problems or optimization problems that in the end are search problems, as we search in the space of all possible solutions for one that is the best or optimal. Thus, many problems can be reduced to merely searching, and this is a complicated task that typically requires cleverness. 188

Chapter 4

Mars Rover

Continuing with our Mars Rover example, Figure4-9 shows the rover after it finishes its wandering-around stage and injects itself with a belief of water at the lower-left corner cell of the third sector, and so it sets course to reach that cell. The injection of this belief allows the rover to find an actual water location that was in the vicinity of the injected water-location belief. Thus, by diversifying the search to unexplored areas we found an actual water location (Figure4-10). This process is repeated again; the rover wanders around (random moves), eventually injects a new belief, and moves to that location (Figure4-11).

Figure 4-9. The rover injects itself with a belief of a possible water location on the lower-left corner of the third sector

189

Chapter 4

Mars Rover

Figure 4-10. The rover follows the injected belief and in the process finds an actual water location

190

Chapter 4

Mars Rover

Figure 4-11. The rover repeats the process, wanders around, and then injects a new water-location belief The Mars Rover presented in this chapter has multiple features that can be refined to improve its performance. For instance, the WanderThreshold may be adjusted since the rover spends more and more time on Mars, looking to prolong the time it stays wandering in a certain area; this decision may be dependent on the square area of the sector where it’s wandering. The strategy of always choosing a corner of the less-frequently visited sector to inject the water-location belief can also change and be made dependent on various conditions related to the rover’s history or state. The choice can also be made randomly; i.e., choose a random cell in the selected sector to inject the water-location belief or maybe choose the least-visited cell in that sector. The division of the terrain may also change; we could use a set of division patterns collected

191

Chapter 4

Mars Rover

in a database to divide the terrain in different ways (not always with 2n subdivisions) and give the rover the opportunity to explore different areas of diverse shapes. The possibilities are endless, and it’s up to the reader to use the skeleton provided in this chapter and create their perfect Mars Rover. Now that we have examined a complete practical problem of an agent and an agent’s architecture, we can move forward and explore multi-agent systems in which various agents coexist and maybe collaborate or compete to achieve certain goals that could be common to them all. This will be the main focus of the next chapter.

Summary Throughout this chapter we presented the practical problem of designing a Mars Rover AI using a hybrid architecture composed of a reactive layer and a deliberative layer that implements the BDI (Beliefs, Desires, and Intentions) paradigm. The Mars Rover example included a visual application (Windows Forms) that demonstrated how the rover reacts to different scenarios, how it’s able to plan via a path-finding algorithm, and how it’s able to provide timely responses to immediately perceived situations. We also presented a Statistics and Probability component in the agent that acts as a deliberative component and allows it to explore unexplored or poorly visited areas of the terrain.

192

CHAPTER 5

Multi-Agent Systems Thus far we have studied agents as single entities interacting with the environment; in real life many problems can be solved much more quickly and efficiently when multiple agents collaborate to achieve a common goal. Recall the cleaning agent from Chapters 2 and 3; this agent was dealing with the problem of cleaning an entire terrain on its own. Undoubtedly, this task could be completed much quicker if various cleaning robots were on the terrain communicating and helping each other to complete, in a shorter time, the task that for a single agent would take much longer and at higher resource consumption. Nowadays, multi-agent systems (MAS) are applied in real-world applications such as computer games, military defense systems, air traffic control, transportation, graphic information systems (GIS), logistics, medical diagnosis, and so on. Other uses involve mobile technologies, where they are applied to achieve automatic, dynamic load balancing and high scalability. Throughout this chapter we will examine multi-agent systems in which multiple agents may collaborate, coordinate, communicate, or compete to achieve a certain goal. MAS fall into an area where distributed systems and AI join to form what is known as distributed artificial intelligence. At the end of this part, which will take the next three chapters, we will present a practical problem where various cleaning robots will collaborate to clean a room.

© Arnaldo Pérez Castaño 2018 A. Pérez Castaño, Practical Artificial Intelligence, https://doi.org/10.1007/978-1-4842-3357-3_5

193

Chapter 5

Multi-Agent Systems

Note Multi-agent systems represent distributed computing systems. As with any distributed system, they are composed of a number of interacting computational entities. However, unlike classical distributed systems, their constituent entities are intelligent and have the capacity to have intelligent interactions with one another.

What’s aMulti-Agent System? As occurred with the logic and agent terms previously presented, there’s no global agreement on a definition for multi-agent system. In this book, we’ll provide a personal definition that we regard as logical and that considers other MAS definitions taken from the scientific literature. A multi-agent system (MAS) is a set S of agents that interact with each other in either a competitive manner—looking to achieve the goals defined by the subset S' of agents to which they belong (S' belongs to a partition of S)—or a collaborative manner—seeking to achieve a common goal defined in S. Additionally, it can happen that every agent in S is acting to achieve its own goals; in such cases we say that we are dealing with an independent MAS. In Table5-1 we can see a first and very frequent scenario of an MAS being applied to air traffic control; in this scenario, Agent Controller 1 (A1) deals directly with pilots and collaborates with Agent Controller 2 (A2) in finding them a runway available for landing. Refer to Table5-1 for a complete dialogue between the two collaborative agents.

194

Chapter 5

Multi-Agent Systems

Table 5-1. MAS Example in Air Traffic Control Scenario Pilot

Agent Controller 1 (A1)

Agent Controller 2 (A2)

To A1: Can I land? To A2: Any runway available? To A1: Runway P. To Pilot: Clear for P. To A1: OK To A2: Runway P is busy now. Now that we have introduced a self-definition for the MAS term we’ll continue presenting other relevant, related concepts. A coalition is said to be a subset of the set of agents; for an MAS such as basketball, baseball, or soccer games there are always two coalitions—the two teams competing. A strategy is a function that receives the current state of the environment and outputs the action to be executed by a coalition. The strategy for Team A usually depends on the actions executed by each agent in Team B at the current moment. A platform, also known as a multi-agent infrastructure, is a framework, base, or support that describes the agent architecture, the multi-agent organization, and their relations or dependencies. It allows agents to interact without taking into consideration the properties of such a platform (centralized or not, embedded into the agents or not, and so on), and it usually provides agents with a set of services (agent location and so forth) depending on the system needs, with the aim of enhancing MAS activity and organization; it is considered a tool for agents.

195

Chapter 5

Multi-Agent Systems

Agent architecture describes the layers or modules constituting a single agent as well as the relations and interactions among them. For instance, agents (in the context of MAS) regularly have a communication module to augment communication with users and other agents. As we know (from Chapters 3 and 4), some types of agents also have a planning layer. Normally, incoming messages arriving at the communication module will affect the planning layer by some connection, and the planning layer may create outgoing messages to be handled by the communication module. A multi-agent organization describes the manner in which multiple agents are organized to form an MAS.Relations, interactions between agents, and their specific roles within the organization constitute a multi- agent organization. Agent architecture is not part of the multi-agent organization even though interrelations among them are common. An agent is said to be autonomous in an MAS if it’s autonomous with respect to every other agent in the set of agents making up the MAS; in other words, if it’s beyond the control or power of any other agent. An MAS is discrete if it is independent and the goals of the agents bear no relation to one another. Thus, discrete MAS involve no cooperation as each agent will be going its own way trying to achieve its own goals. Modularity is one of the benefits of MAS; sometimes solving a complex problem is subdivided into easier subproblems of the original problem, and each agent can be specialized in the solution of one of these particular types of problem, therefore leading to reusability. Imagine an MAS dealing with a city disaster like an earthquake. Such an MAS would be composed of different agents (policemen, firemen, and so forth) where each agent would be devoted to a single task and all of them would have the global assignment of establishing order and saving lives. Problem solving through MAS leads to efficiency; the solution to a problem can often be achieved much quicker if various concurrent, parallel agents are working at the same time to solve the problem.

196

Chapter 5

Multi-Agent Systems

An MAS also provides improved reliability because we may have multiple agents taking care of a single task, and if one of them fails then the others can continue its work by distributing among the rest. One last important benefit that MAS provides us is flexibility; we can add or delete agents from an MAS at will, and different agents that have complementary skills may form coalitions to work together and solve problems. In the following sections we’ll be exploring some key concepts in the area of distributed AI and especially on the topic of MAS: communication, cooperation, negotiation, and coordination. We’ll also take a deeper look at some of the concepts previously presented.

Note One of the services a platform can offer is agent location; in other words, the facility by which an agent or a third party is able to locate another agent in an MAS environment.

Multi-Agent Organization Earlier in the chapter we provided a definition for the term multi-agent organization. In this section, we will detail some of the most common multi-agent organizations one can find: •

Hierarchical: organization in which agents can only communicate by following a hierarchical structure. Because of this restriction there’s no need to have an agent-location mechanism. Instead a set of facilitators act as middle men and receive and send all messaging between agents. These facilitators are usually at the upper levels of the hierarchy. Consequently, lower levels usually depend on higher levels. Communication is really reduced in this type of organization. 197

Chapter 5

Multi-Agent Systems

•

Flat or Democracy: organization in which agents can communicate directly with one another. There’s no fixed structure in this type of organization, but agents can form their own structures if they judge it is necessary to solve some specific task. Furthermore, no control of one agent over another is assumed. Agent location must be provided as part of the infrastructure or platform or the system must be closed; in other words, every agent must know about the others at all times. This type of organization can lead to communication overhead.

•

Subsumption: organization in which agents (subsumed) can be components of other agents (container). This type of organization resembles that of the hierarchical model except that in this case subsumed agents surrender all control to their container agents. As occurs with the hierarchical organization, it involves low communication overhead.

•

Modular: organization in which the MAS is composed of various modules, and each of these can be conceived of as a stand-alone MAS.The partition of the system into modules is usually done by considering measures such as geographical vicinity or a necessity for extreme interaction among agents and services within the same module. Modularity increases the efficiency of task execution and reduces communication overhead.

Hybrids of these organization types and dynamic changes from one style to another are possible. From the multi-agent organizations detailed in the previous points we can easily see that communication plays a vital role in defining the architecture and way of functioning of agents. We’ll devote the next section to explaining some key aspects of this very important topic. 198

Chapter 5

Multi-Agent Systems

Note In recent years, a large variety of agent architectures have been proposed. In the case of MAS architectures, this number greatly decreases because for an agent to be incorporated in an MAS it must be equipped with vital components (communication, coordination, and so on) that would allow it to properly interact with other agents.

C ommunication Agents in an MAS must coordinate their actions to solve problems. In this scenario, coordination is achieved by means of communication, which plays a vital role in providing agent interaction and facilitating not only coordination but also information sharing and cooperation. In the last section we discussed MAS organizations and how they can affect agent communication depending on the type of organization they are in. Now, we’ll look at some detailed aspects of this topic. The communication link established between agents can be classified as: •

Point to Point: agents communicate directly with each other

•

Broadcast/Multicast: agents are capable of sending information to a subset of the set of agents. If this subset equals the set of agents then the agent is broadcasting; otherwise, it is multicasting.

•

Mediated: communication between agents is mediated by a third party (facilitators; see Figure5-1).

199

Chapter 5

Multi-Agent Systems

Figure 5-1. Agent 1 and Agent 2 communicate via a facilitator acting as middle man Considering the nature of the medium by which messages travel from one agent to another, communication can be classified as: •

Direct routing: Messages are sent directly to other agents with no loss of signal.

•

Signal-propagation routing: Agents send a signal whose intensity decreases as distance increases.

•

Public-notice routing: using blackboard systems

Blackboard systems and direct message passing are two options for establishing agent communication. A blackboard system (Figure5-2) represents a common, shared space for every agent to place their data, information, and knowledge. Each agent can write and read from the blackboard at any given time, and in this centralized system there’s no direct communication between agents. The blackboard also acts as a dispatcher, handling agent requests, data of the common problem, current state of the solution, current task of each agent, and so on. Since the blackboard system consists of a shared resource, one must be aware of all the concurrent issues that can arise in such a model (various agents trying to access the same info, agents using partial, not updated data written by other agents, and so on).

200

Chapter 5

Multi-Agent Systems

Figure 5-2. The blackboard system is a centralized, common space for all agents to place and share their information In the other variant (message passing), information is passed from one agent (sender) to another (receiver). Communication among agents means more than communication in distributed systems; therefore, it is more appropriate to speak about interaction instead of communication. When we communicate we perform more than an exchange of messages with a specified syntax and a given protocol, as in distributed systems. Therefore, a more elaborate type of communication that tends to be specific to MAS is communication based on the Speech Act Theory (Searle, 1969; Vanderveken, 1994), which is the one that best describes the message-passing alternative for establishing agent communication.

Speech Act Theory The origin of the Speech Act Theory (also called Communicative Act Theory) can be traced back to John Austin’s book How to Do Things with Words (1962); most treatments of communication in MAS are inspired in this theory. The main point behind this theory is that we should consider communication as a form of action. Furthermore, Austin noticed that some utterances are like physical actions and appear to change the state of the world. Examples of this could be a declaration of war or simply “I declare you man and wife.” 201

Chapter 5

Multi-Agent Systems

Austin argued that all communications could be phrased via declarative forms using the appropriate performative verbs. Therefore, a simple informative phrase such as “the jazz concert will take place on October 10th” can be treated as “I inform you that the jazz concert will take place on October 10th.” Directives—as, for example, “give me that bottle of rum”—can be treated as “I request (demand) that you give me that bottle of rum.” A commissive such as “I’ll give you $100 for your furniture” can be treated as “I promise I’ll give you $100 for your furniture.” Everything we utter is said with the intention of satisfying some goal; a theory of how utterances are used to achieve intentions is Speech Act Theory, and by using the different types of speech acts agents can interact effectively.

Note Communicative act theories are theories of language use; they try to explain how language is used by people every day to achieve their goals and intentions. Examples of some speech-act constructs are presented here:

202

•

Inform other agents about some data.

•

Query others about their state or current situation.

•

Answer questions.

•

Request others to act.

•

Promise to do something.

•

Offer deals.

•

Acknowledge offers and requests.

Chapter 5

Multi-Agent Systems

Searle (1969) classified speech acts into the following categories: •

Representatives: when we are informing, asserting, claiming, describing; for example, it’s cloudy

•

Directives: an attempt to make the hearer do something; in other words, requesting, commanding, advising, forbidding; for example, bring me that bottle of rum

•

Commissives: when we commit the speaker to do something, such as when promising, agreeing, offering, threatening, inviting; for example, I promise I'll bring you tea

•

Expressives: when the speaker expresses a mental state; in other words, congratulating, thanking, apologizing; for example, I’m sorry you did not make it to Harvard

•

Declarations: when the speaker brings about a state of affairs; in other words, declaring, marrying, arresting; or example, I declare (pronounce) you man and wife

A speech act has two components: a performative verb (for example, inform, declare, request, and so on) and a propositional content (for example, the bottle is open). Constructing speech acts involves combining a performative verb with a propositional content. See the following examples: Performative = inform Content = the bottle is open Speech act = the bottle is open. Performative = request Content = the bottle is open Speech Act = please open the bottle.

203

Chapter 5

Multi-Agent Systems

Performative = inquiry Content = the bottle is open Speech Act = is the bottle open? Performative = refuse Content = the bottle is open Speech Act = I refuse to open the bottle Performative = agree Content = the bottle is open Speech Act = I agree to open the bottle In the same way we typically create a language for communication among co-workers at work, an MAS containing different agents that might be running in different machines, under different operating systems requires an agent communication language standardized to allow the exchange of messages in a standard format.

Agent Communication Languages (ACL) Agent communication languages began to emerge in the 1980s; at first, they were dependent on the projects for which they were created and also on the internal representation of the agents that used them; there were no standard languages at that time. Around the same time, but more generic than its predecessors, appeared the Knowledge Query and Manipulation Language, commonly known as KQML. It was created by the DARPA Knowledge Sharing Effort and was supposed to be a complement to the studies being made on knowledge-representation technologies, specifically on ontologies. KQML is comprised of two parts: the language itself acts as an “outer” language, and the Knowledge Interchange Format (KIF) acts as an “inner” language; the first describes performatives, while the latter describes propositional content and is largely based on first-order predicate calculus. KQML represents knowledge that relies on the construct of a 204

Chapter 5

Multi-Agent Systems

knowledge base; thus, instead of using a specific internal representation, it assumes that each agent maintains a knowledge base described in terms of knowledge assertions. KQML proposed a number of performatives such as query and tell. The idea was that each performative could be given semantics based on the effect it had on the knowledge bases of the communicating agents. Moreover, an agent would send a tell performative for some content only if it believed in the content sent; in other words, if it thought the content belonged in its knowledge base. An agent that receives a tell performative for some content would insert that content into its knowledge base; in other words, it would begin believing what it was told.

Note An ontology is an explicit description of a domain (concepts, properties, restrictions, individuals, and so on). It defines a vocabulary and is used to share an understanding of the structure of information among computer agents or humans. In the Blocks World, Block represents a concept and OnTop represents a relationship. The elegance of KQML is that all information for understanding the content of the message is included in the communication itself. Its generic syntax is described in Figure5-3; notice it resembles the Lisp programming language:

Figure 5-3. Basic structure of a KQML message 205

Chapter 5

Multi-Agent Systems

In the following lines we show an example of a KQML dialogue between AgentX and AgentY: (stream-about :sender AgentX :receiver AgentY :language KIF :ontology CleaningTerrains :query :reply-for query_from_AgentY :content cell_i cell_j ) (query :sender AgentX :receiver AgentY :content(> (dirt cell_i) (0)) ) (tell :sender AgentX :receiver AgentY :content(= (cell_j) (1)) ) (eos :sender AgentX :receiver AgentY :query :reply-for query_from_AgentY )

206

Chapter 5

Multi-Agent Systems

In this little fragment of a KQML message, AgentX asks AgentY if there’s dirt at cell i; it also replies to a previous query received from AgentY and tells it that cell j has 1 of dirt; eos stands for End of Signal. Note that the value of the content field is written in the language defined by the language tag, in this case KIF.

Note KIF, a particular logic language, has been proposed as a standard to describe things within expert systems, databases, intelligent agents, and so on. One could say that KIF is a mediator used in the translation of other languages. Even though KQML is usually combined with KIF as content language, it can also be used in combination with other languages like Prolog, Lisp, Scheme, and so on. In 1996, the Foundation for Intelligent Physical Agents (FIPA), a stand- alone non-profit organization now part of IEEE Computer Society, started working on several specifications for agent-based applications; one of these specifications was for an ACL of the same name as the organization; i.e., FIPA-ACL. The basic structure of FIPA is quite similar to that of KQML, as illustrated in Figure5-4.

Figure 5-4. Components of a FIPA message 207

Chapter 5

Multi-Agent Systems

The parameters admitted by the FIPA language specification are the following: •

:sender— who sends the message

•

:receiver— who is the recipient of the message

•

:content— content of the message

•

:reply-with— identifier of the message

•

:reply-by— deadline for replying to the message

•

:in-reply-to— identifier of the message being replied to

•

:language— language in which the content is written

•

:ontology— ontology used to represent the domain

•

:protocol— communication protocol to be followed

•

:conversation-id— identifier of the conversation

Table5-2 details some FIPA performatives and the purpose for which they were created.

208

Chapter 5

Multi-Agent Systems

Table 5-2. Some FIPA Performatives Performative

Passing Requesting Negotiation Perform Error Info Info Actions Handling x

accept-proposal

x

agree x

cancel

x x

cfp confirm

x

disconfirm failure

x

inform

x

inform-if

x

inform-ref

x

x

x

not-understood x

propose query-if

x

query-ref

x x

refuse x

reject-proposal request

x

request-when

x

request-whenever

x

subscribe

x

209

Chapter 5

Multi-Agent Systems

Inform and Request represent two basic performatives, while the others are defined in terms of these. Their meaning is composed of two parts: a precondition list that states what must be true for the speech act to succeed and a rational effect—i.e., what the sender of the message hopes to achieve. In the FIPA inform performative, content is a statement, and sender informs the receiver that a given proposition is true; sender states the following: •

Some proposition is true.

•

The receiving agent must also believe that the proposition is true.

•

The receiver has no knowledge whatsoever of the truth of the proposition.

The next lines show an example of a FIPA inform performative: (inform :sender(agent-identifier :x) :receiver(agent-identifier :y) :content dirt( cell_i, 0 ) :language Prolog ) On the other hand, content in the request performative consists of an action; in this case, the sender requests the receiver to perform some action; sender states the following:

210

•

The action content is to be performed.

•

Recipient is capable of performing this action.

•

Does not believe that receiver already intends to perform the action.

Chapter 5

Multi-Agent Systems

In this section, we analyzed a critical topic in MAS design: communication. Even though this is an essential aspect of every MAS, there are other components that are also relevant, one of which is coordination. We need our agents to coordinate and avoid problems like having two of them executing the same action at the same time (both trying to go through the same door at the same time) when it might be impossible. Coordination will be the focus point of the next section.

Coordination & Cooperation An agent that is part of an MAS exists and performs its decision making in an environment where other agents exist as well. To avoid chaos and to ensure rational behavior in this environment we need our agents to coordinate and achieve their goals in a concise, logical manner. There are two main criteria points for assessing MAS: coherence and coordination. Coherence refers to how well the MAS behaves considering some criteria of evaluation (solution quality, efficiency in applying resources, logical decision making, and so forth). A common problem for an MAS is how it can maintain overall coherence while lacking explicit global control. In such cases, agents must be able on their own to determine goals they share with other agents; they must also determine common tasks, avoid unnecessary conflicts, and collect knowledge. Having some form of organization among the agents is useful in this scenario. Coordination refers to the ability of agents to avoid, by means of synchronization, irrational activities in which two or more agents could be involved. It implies the consideration of the actions of other agents when planning and executing one agent’s actions. It is also a means to achieve the coherent behavior of the MAS, and it may imply cooperation. When agents in an MAS cooperate, they work toward achieving common goals. When they are competing, they have opposite goals. Coordination in both cases is essential because the agent must take into account the actions 211

Chapter 5

Multi-Agent Systems

of others when competing or asking for a given resource or offering a service. Examples of coordination include ensuring the actions of agents are synchronized, providing opportune information to other agents, and avoiding redundant problem solving. Cooperation is coordination among non-antagonistic agents. Typically, to cooperate successfully, each agent must maintain a model of the other agents and also develop a model of future interactions; this implies sociability. For agents in an MAS to work together they must be able to share tasks and information. If we had an MAS where agents were designed by different individuals then we could end up having an MAS with various goals, all derived from different agents. Alternatively, if we are responsible for designing the entire system then we can have agents helping each other whenever we deem necessary; our best interest is going to be their best interest. In this cooperative model we say that agents are benevolent because they are working all together to achieve a common goal. A benevolent MAS, or those in which all agents are benevolent, simplifies the design task of the system significantly. When agents represent the interests of individuals, organizations, companies, and so on, we say that they are self-interested. These agents will have their own set of goals, apart from the goals of other agents in the MAS, and will act to achieve them even at the expense of other agents’ welfare; this could potentially lead to conflict between some of them.

Note Self-interested agents complicate the design task of an MAS seriously. For an MAS with self-interested agents, we typically have to incorporate mechanisms for intelligent behavior, such as those based on game theory or rule-based algorithms. Figure5-5 illustrates a tree with some of the possible approaches for achieving coordination. 212

Chapter 5

Multi-Agent Systems

Figure 5-5. Taxonomy for agent coordination possibilities A basic strategy for cooperation in an MAS is to decompose and then distribute tasks among agents. This divide-and-conquer approach can certainly reduce the complexity of the global task because by dividing it into smaller subtasks the global solution can be obtained in a shorter time and using fewer resources. In general, task sharing can be divided into three stages: •

Problem decomposition (Divide)

•

Sub-problem solution

•

Solution synthesis (Conquer)

In the problem decomposition stage the global problem is divided into subproblems, typically by a recursive or hierarchical procedure. Deciding how to do the division is a design choice and is problem dependent. Deciding who makes the problem decomposition and how it’s made can 213

Chapter 5

Multi-Agent Systems

be left to an agent that we appoint as task distributor. This agent may not take care of anything other than distributing tasks among other agents in what would be a centralized design. Alternatively, it could be part of the subproblem solution team and act as any other agent but with the special attribute of being a work organizer. Once the problem decomposition stage has provided us with a division of the global problem, each agent contributes to the subproblem assigned. During this process agents may need to share some information and update others on their current situation. Finally, in the solution synthesis stage all solutions to subproblems are joined (recursively or hierarchically). In this cooperative model we can distinguish two main activities that will most likely be present during MAS execution: task sharing and results sharing. In the first, components of the task are distributed to agents, while in the latter partial or complete results are also distributed. We can use a Subscribe/Notify (Publisher/Subscriber) pattern for results sharing; in such a pattern an object (subscriber) subscribes to another object (informant), requesting a notification for when event evt occurs. Once evt has occurred the informant notifies the subscriber of its occurrence, and they proactively exchange information in this manner. At this point we have some unanswered questions. How is the process of allocating or matching tasks to agents done? How do we assemble a solution from the solved parts? In order to answer the first question we will look at a task-sharing protocol known as Contract Net.

Note Some of the commonly used mechanisms for task sharing include the Market mechanism, where tasks are assigned to agents by generalized agreement or mutual selection; multi-agent planning, where planning agents have the responsibility of task assignment; and Contract Net protocol, one of several task-sharing mechanisms.

214

Chapter 5

Multi-Agent Systems

Negotiation Using Contract Net The Contract Net mechanism is an interaction mechanism for task sharing among agents. It follows the model used by entities (governments, corporations, and so forth) to regulate the exchange of goods and services. Contract Net offers a solution to the problem of finding an appropriate agent to work on a task. The agent who wants a task done is called manager. The candidate agents who can fulfill the task are known as contractors. The Contract Net process can be summarized in the next stages (Figure5-6): 1. Announcement: The manager sends out an announcement of the task, which includes a specification of the task to be achieved. This specification must include a description of the task, any constraints (deadlines, etc.), and meta task info (bids must be submitted prior to deadline, due date, etc.). The announcement is broadcast. 2. Bidding: Agents receive the broadcast corresponding to the manager’s announcement and decide for themselves whether they want to bid for the task. In this process they must take into account various factors like capacity to carry out the task and being able to meet all constraints. If they finally decide to bid then they submit a tender. 3. Awarding: The manager must choose between bids and decide on an agent to award the contract to. The result of this process is communicated to every agent that submitted a bid. 4. Expediting: The winner or successful contractor expedites the task.

215

Chapter 5

Multi-Agent Systems

Figure 5-6. Contract Net process 216

Chapter 5

Multi-Agent Systems

Generally, any agent can act as manager and any agent can act as contractor by replying to task announcements. Because of this flexibility, task decomposition can be taken further to different depth levels. Furthermore, if a contractor is unable to complete or provide a suitable solution for a task then the manager can look for other contractor candidates, and as long as there are agents in the MAS the manager can seek a candidate contractor that at some point in time will be available to execute a task according to the manager’s requirements. From the contractor’s perspective, he receives various offers (announcements) from various managers and decides upon what he thinks is the best offer. This decision is made based on some criteria (proximity, reward, etc.), and he sends a bid to the corresponding manager. From the manager’s perspective, he receives and evaluates bids for each task announcement. Any bid for a given task that is considered satisfactory will be accepted and always before the expiration time of the task announcement is met. Afterward, the manager notifies the winning contractor and possibly all other candidates who sent a bid with an “award notice” announcement that the task has been awarded. Perhaps one could say that a negative point of the Contract Net mechanism is that the awarded agent does not have to be the best or most suitable agent for the task, as the most suitable agent for the task could be busy at award time.

Note There exist several reasons why a manager may not receive bids on an announcement. All agents might be busy at the time of receiving the announcement, a candidate contractor (agent) ranks the task announced below other offered tasks, or no contractor is capable of working on the announced task.

217

Chapter 5

Multi-Agent Systems

The FIPA-ACL specification was projected to support the Contract Net negotiation mechanism. The cfp (call for proposals) performative is used to announce the task; the propose and refuse performatives are used to propose or refuse a proposal; accept and reject are used to accept or decline a proposal, and inform and failure are used to communicate completion of the task with its corresponding result.

Social Norms & Societies Classical AI has been concerned with designing single agents that incorporate reasoning and control logics implemented using a von Neumann architecture. However, agents are not always in isolation; they exist in an environment where they might find other agents and be in need of some type of interaction to complete their task in an optimal manner. Thus, it’s logical to see agents as a society where well-known rules govern their behavior and actions. Sociability is vital in cooperative MAS and aims to aid true peer-to-peer distributed and flexible paradigms that recent applications require and where agents can find their utmost contribution. A social commitment in an MAS is an obligation created between an agent and another agent or group of agents, constraining the behavior of the first to follow a given prearranged commitment or rule. Imagine an MAS where agents must stay together at the same line of work in a 2D space, but AgentX moves faster than the remaining agents and always tends to go ahead and leave the team behind. A social commitment from this agent to the others could be to always stay in the same line and not move ahead and leave someone behind. To establish rules for an MAS, we can design social norms or laws to rule agents’ behavior (Figure5-7). A social law is a set of constraints, and a constraint comes in the form of a pair (S, A) stating that an agent cannot execute an action A while being in state S.

218

Chapter 5

Multi-Agent Systems

Figure 5-7. Social law determining agent movement in a grid of 3 × 3. This law prevents collisions. The set of focal states is the set of states we want our agent to have access to; thus, from any focal state there must exist a path to the remaining focal states. A useful law is one that does not stop agents from getting to one state from another; the law from Figure5-5 is a useful law. Now that we have set the grounds for MAS terminology, concepts, and ideas we will introduce in the following chapter a complete practical application consisting of multi-agent communication software that allows various agents to exchange messages using a WCF Publisher/Subscriber pattern in a two-sided (service, client) program. This communication program will be used later (in Chapter 7) to create a complete example of a multi-agent system where a set of cleaning robots will communicate, coordinate, and cooperate to clean an n x m room of its dirt.

219

Chapter 5

Multi-Agent Systems

Summary In this chapter, we introduced the field of multi-agent systems. We presented various definitions and concepts that set us on the right path to getting acquainted with some MAS terminology necessary for diving into the scientific literature associated with this topic. We examined multi- agent organizations, agent communication, and its subfields (Speech Act Theory and Agent Communication Languages), and we concluded the chapter by detailing the vital topics of coordination and cooperation among agents. We also included in this final part the topics of negotiation and social norms. In the next chapter, we’ll present a very interesting practical problem where we’ll have a set of N agents exchanging messages in a WCF application created under the Publisher/Subscriber pattern.

220

CHAPTER 6

Communication inaMulti-Agent System Using WCF In the previous chapter, we examined the basics of multi-agent systems (MAS) and got acquainted with concepts like MAS platform, coordination, cooperation, and communication. In this chapter, we will describe an application that uses Windows Communication Foundation (WCF) to create a network of agents capable of interacting with and passing messages among each other. This application will use the Publisher/Subscriber design pattern to set up the communication component that every agent in the MAS will incorporate. We will use the application described throughout this chapter again in the next chapter, adapting it as the communication module of every agent in an MAS consisting of cleaning agents whose task is cleaning a room of its dirt. WCF emerged in 2006 as a development kit and eventually became part of the .NET Framework; it’s an application programming interface (API) for developing connected systems where both security and reliability in any communication between internal systems of an organization or systems over the internet is possible and provided. It is designed to offer a manageable approach to distributed computing, broad interoperability,

© Arnaldo Pérez Castaño 2018 A. Pérez Castaño, Practical Artificial Intelligence, https://doi.org/10.1007/978-1-4842-3357-3_6

221

Chapter 6

Communication inaMulti-Agent System Using WCF

and direct support for service orientation. WCF represents Microsoft’s alternative to a platform that collects a set of industry standards that define different protocols, service interactions, type conversion, marshalling, and so forth. It provides developers with the fundamental predesigned tools that every network application might require, and its first release included many useful facilities for creating services (hosting, service-instance management, asynchronous calls, reliability, transaction management, disconnected queued calls, security, and so on). Applications built using WCF as the runtime environment will allow us to use Common Language Runtime (CLR) types as services and will allow us to consume other services as CLR types. Concepts such as service, contract, binding, endpoint, and others will be explained throughout this chapter as we develop our MAS communication example.

Note Windows Communication Foundation (WCF) is a framework for developing and deploying services on Windows. Using WCF, we can build service-oriented applications (SOAs). WCF replaced the older ASMX web services technologies.

Services A service is a functional component made accessible to its consumers via a network that could be the internet or a local internal network. A calculator could very well be a service offered to different clients in a network so they can connect to the service and request any operation between any given numbers. In a service-oriented application (SOA) we aggregate services the same way we aggregate objects when developing an object-oriented application; the service becomes the first-class citizen in this type of application.

222

Chapter 6

Communication inaMulti-Agent System Using WCF

Services communicate using any communication protocol previously agreed on, and they can use any language, platform, versioning, or framework without needing to have any agreement on those. Thus, one can say that services are dependent on the communication protocol applied but independent in any other area. The client of a service is the part making use of the service’s functionality. In the calculator service example the client would be the program requesting that the calculator solve mathematical expressions. The client can be any type of program, from a console application to a Windows Forms, an ASP.NET MVC site, a WPF program, or another service. In WCF, the client never interacts with the service directly, not even with a local service. Instead, the client always uses a proxy to forward calls to the service. The proxy acts as a middle man, presenting the same operations as the service in addition to some proxy-related methods.

Note There has been an evolution from applications where functions were the first-class citizen to applications where objects were the first-class citizen (object-oriented programming), passing through component-oriented applications (component-object model, COM) and leading to the most recent step in this evolution, service-oriented applications (SOAs). WCF most often uses Simple Object Access Protocol (SOAP) messages to communicate; SOAP is a protocol for data exchange. It can be seen as a set of components that can be invoked, published, and discovered. These messages are independent of transport protocols, and, contrasting with web services, WCF services can communicate over a variety of transports, not just HTTP.WCF clients are capable of interoperating with non-WCF services, and WCF services can interact with non-WCF clients.

223

Chapter 6

Communication inaMulti-Agent System Using WCF

Contracts We deal with contracts often in our daily life, especially in business-related affairs, to make sure parts engaging in a relationship agree on various points. In WCF, a contract is a standard way of describing what a service does; it’s a way for service consumers and providers to correlate correctly. In an SOA application, having a properly defined contract can give its consumers a pretty good idea of how to work with the service even though they might not know how it’s implemented. WCF defines various types of contracts: •

Service Contract, Operation Contract: used to represent a service and describe the operations that the client can perform on the service

•

Data Contract: used to represent an agreement on the data that will be exchanged between the service and the client. WCF defines implicit contracts for built-in types such as int and string and gives you the option of defining explicit data contracts for custom types.

•

Fault Contract: used to define which errors are raised by the service by associating custom exception types with certain service operations and describing how the service handles and propagates errors to its clients

•

Message Contract: used by the service to interact directly with messages, altering its format or manipulating the service messages to modify other features like the SOAP header and so forth

There are different ways or patterns for defining contracts in WCF; we could define them using the One-Way pattern, the Request–Response pattern, or the Duplex pattern. These are all message-exchange patterns.

224

Chapter 6

Communication inaMulti-Agent System Using WCF

•

One-Way: When an operation has no returned values and the client application is not interested in the success or failure of the invocation, we may have this “fire & forget” invocation called One-Way. After the client issues the call, WCF generates a request message, but no reply message will ever head back to the client. Consequently, One-Way operations can’t return values, and any exception thrown on the service side will not make its way back to the client.

•

Request–Response: In this pattern, a service operation call consists of a message sent and a reply expected from the service. Operations using this pattern have an input parameter and an associated return value. The client is always the one to initiate communication between the parties.

•

Duplex: This exchange pattern allows for a random number of messages to be sent by a client and received in any order. It resembles a conversation where each word spoken is seen as a message. Any part can initiate communication.

225

Chapter 6

Communication inaMulti-Agent System Using WCF

In order to implement a service in WCF you typically go through the following steps: 1. Define the service contract. A service contract specifies the signature of a service, the data it exchanges, and other contractually required data. The following code shows the service version of the very classic Hello World program: [ServiceContract] interface IHelloWorld { [OperationContract(IsOneWay = true)] void HelloMessage(); } 2. Implement the contract by inheriting from the service contract definition (prearrangement interface) and create the class that implements the contract: public class Hello: IHelloWorld { public void HelloMessage() { Console.WriteLine("Hello World"); } } 3. Configure the service by specifying endpoint information and other behavior information. We’ll see more about this in the next section.

226

Chapter 6

Communication inaMulti-Agent System Using WCF

4. Host the service in IIS or in an application; it could be a console application, Windows Forms, WPF, ASP .NET, etc. 5. Create a client application; it could be a console application, Windows Forms, WPF, ASP .NET, etc. Note that methods declared on the IHelloWorld service contract that do not have the OperationContract attribute will not be considered as WCF methods; in other words, they won’t be invokable over WCF applications. You can mix non-WCF methods with WCF methods with the intention of having some subliminal processing, but only for that purpose.

B indings WCF allows us to send messages using different transport protocols, such as HTTP, HTTPS, TCP, MSMQ, and so on, and using different XML representations, such as text, binary, or MTOM (Message Transport Optimization Mechanism); this last one is known as the message encoding in WCF.Furthermore, we can improve specific messaging interactions using a suite of SOAP protocols, such as the multiple WS-X (WSHttpBinding, WSDualHttpBinding, etc.) specifications. Improvements could be related to security, reliable messaging, and transactions. These communication concepts (transport, message encoding, and protocol) are vital to understanding what happens on the wire at runtime. In WCF, bindings are represented by the System.ServiceModel. Channels.Binding class, and all binding classes must derive from this base class; Table6-1 illustrates some of the built-in bindings that WCF provides.

227

Chapter 6

Communication inaMulti-Agent System Using WCF

Table 6-1. WCF Built-in Bindings Binding Class

Transport

Message Encoding

Message Version

BasicHttpBinding

HTTP

Text

SOAP 1.1

WSHttpBinding

HTTP

Text

SOAP 1.2 WS-Addressing 1.0

WSDualHttpBinding

HTTP

Text

SOAP 1.2 WS-Addressing 1.0

NetTcpBinding

TCP

Binary

SOAP 1.2

NetPeerTcpBinding

P2P

Binary

SOAP 1.2

NetMsmqBinding

MSMQ

Binary

SOAP 1.2

CustomBinding

Up to you

Up to you

Up to you

Bindings like BasicHttpBinding and WSHttpBinding were created for scenarios where interoperability is essential. Thus, they both use HTTP as the transport protocol and simple text for message encoding. On the other hand, bindings that have the Net prefix are optimized to function with the .NET Framework on both ends (service, client). As a result, these bindings are not designed for interoperability and perform better in Windows environments. A binding is part of another component of a WCF application known as an endpoint; endpoints will be the topic of the next section.

Note As of .NET Framework 4.5 the NetPeerTcpBinding binding has been marked as obsolete and may disappear in the future.

228

Chapter 6

Communication inaMulti-Agent System Using WCF

E ndpoints WCF services are exposed through service endpoints that provide access points for clients to exploit the functionality provided by the WCF service. Service endpoints consist of what is known as the ABC of a service. A stands for Address, which defines where the service is (for example, http://localhost:9090/mas/). B stands for Binding, which defines how to communicate with the service, and C stands for Contract, which defines what the service can do. Hence, an endpoint can be seen as a tuple : an address, a binding, and a contract. We must define endpoints in both our service and client applications; this can be done programmatically or through the app.config file, as shown in the next example (Listing 6-1).

Listing 6-1. Defining Two Endpoints in the app.config File

There’s no significant technical difference in the programmatic way and the configuration setting of the app.config file way for defining endpoints in WCF.Eventually .NET will parse the app.config file and execute its defined configuration in a programmatic manner. Now that we 229

Chapter 6

Communication inaMulti-Agent System Using WCF

have been over the basics of WCF, we will look at the Publisher/Subscriber pattern that WCF supports and that we will be using in communicating with various agents.

Publisher/Subscriber Pattern Real-time applications are those that provide a live feed or update (basketball game, baseball game, and so on) of a particular event occurring at a short, prior time from the time the feed is provided. Real-time apps implement one of two possible mechanisms for giving updated information to clients: pushing and pulling. To understand how these mechanisms work, let’s imagine a scenario where we would like to be updated on the results of a baseball game. We are part of a network that consists of a server, which has the updated information (live updates), and several other computers. Assuming we get the live feed in our browser (client) via HTTP, and considering the use of a pulling mechanism, our computer would be constantly sending update requests and pulling new information (if any) from the server. It would basically be like asking the server from time to time “Do you have anything new for me?” On the other hand, if we were to follow a pushing mechanism our client would tell the server, “Keep me updated on the score of this game,” and the server would automatically “push” updates to the client whenever they were available. The Publisher/Subscriber model follows the latter approach, the pushing mechanism; the server plays the role of publisher and the client the role of subscriber, and it requires a duplex service to be established between both parts. A duplex service consists of two contracts, one at the server and another at the client. The contract implemented at the server will be used by the subscriber (client) to subscribe for a particular data feed. The contract implemented at the client will be used by the server to make a call whenever new data needs to be “pushed.” The contract implemented 230

Chapter 6

Communication inaMulti-Agent System Using WCF

at the client side is known as a callback contract. We’ll see more of the Publisher/Subscriber pattern, as well as callback contracts and duplex services, in the following sections when we look at a practical problem that puts all these pieces together in a complete, functional example.

ractical Problem: Communicating P AmongMultiple Agents Using WCF In this section, we will create a WCF application where several agents contribute to a shared message list and each agent is aware of the current message list; in other words, everyone has an updated copy of the actual list. The service in this scenario acts as a message broker, sending new messages coming from a given agent to all other agents. This is an application that clearly follows the Publisher/Subscriber pattern; in Figure6-1 we can see its architecture.

Figure 6-1. An agent adds a message to the list and the service communicates the updated list to all other agents Beginning with the implementation process, we first need to define our service contract. Since we are going to create a duplex application, the service contract definition will need to be accompanied by a callback contract. The callback contract specifies the operations that the service 231

Chapter 6

Communication inaMulti-Agent System Using WCF

can invoke on the client. To create a WCF service in Visual Studio, go to the Solution Explorer and right-click in the project or folder you wish to be the container of the project; select “Add a New Item,” then look for the “WCF Service” option (Figure6-2).

Figure 6-2. Adding a WCF service to our project Once you add the service you will see two files have been added to your project—a class (contract implementation) and an interface (service contract). You’ll also notice the addition of references to namespaces System.ServiceModel and System.ServiceModel.Description, which are the namespaces containing the binding classes, the ServiceHost class, and so forth.

Note Operations on a duplex service are usually marked as OneWay = true to prevent deadlocks. A deadlock occurs when various units are waiting on the others to finish and as a consequence neither ever does. The implementations of both the service and callback contracts are illustrated in Listing 6-2.

232

Chapter 6

Communication inaMulti-Agent System Using WCF

Listing 6-2. Service and Callback Contracts [ServiceContract(CallbackContract = typeof(IAgentCommunication Callback))] public interface IAgentCommunicationService { [OperationContract(IsOneWay = true)] void Subscribe(); [OperationContract(IsOneWay = true)] void Send(string from, string to, string message); } public interface IAgentCommunicationCallback { [OperationContract(IsOneWay = true)] void SendUpdatedList(List messages); } Notice that in the previous code we are defining a relationship by specifically telling the service contract that its callback contract is IAgentCommunicationCallback. Thus, we are telling the service to use that callback contract to notify the client (notification will be achieved by calling the SendUpdateList() method on the callback) whenever new updates are available. The service contract contains two operations: Subscribe(), which subscribes the agent to the service, and Send(), which sends a new message to the message list. The callback contract has an operation named SendUpdatedList(), which is used to send the latest message list to all agents.

233

Chapter 6

Communication inaMulti-Agent System Using WCF

Note All operations in IAgentCommunicationService and IAgentCommunicationCallback return void because that’s a requirement of the attribute setting IsOneWay = true. One-Way operations will block until the outbound data has been written to the network connection. Now that we know the agreement on operations established by the service and callback, let’s look at their concrete implementations. Listing 6-3 shows the service implementation.

Listing 6-3. Service Implementation [ServiceBehavior(InstanceContextMode = InstanceContextMode. Single, ConcurrencyMode = ConcurrencyMode.Multiple)] public class AgentCommunicationService : IAgentCommunicationService { private static List _callback Channels = new List(); private static List _messages = new List(); private static readonly object _sycnRoot = new object(); public void Subscribe() { try { var callbackChannel = OperationContext.Current.GetCallbackChannel (); lock (_sycnRoot) { 234

Chapter 6

Communication inaMulti-Agent System Using WCF

if (!_callbackChannels.Contains(callbackChannel)) { _callbackChannels.Add(callbackChannel); Console.WriteLine("Added Callback Channel: {0}", callbackChannel.GetHash Code()); callbackChannel.SendUpdatedList(_messages); } } } catch { } } public void Send(string from, string to, string message) { lock (_sycnRoot) { _messages.Add(message); Console.WriteLine("-- Message List --"); _messages.ForEach(listItem => Console. WriteLine(listItem)); Console.WriteLine("------------------"); for (int i = _callbackChannels.Count- 1; i >= 0; i--) { if (((ICommunicationObject)_callback Channels[i]).State != CommunicationState. Opened) { 235

Chapter 6

Communication inaMulti-Agent System Using WCF

Console.WriteLine("Detected Non-Open Callback Channel: {0}", _callback Channels[i].GetHashCode()); _callbackChannels.RemoveAt(i); continue; } try { _callbackChannels[i].SendUpdatedList (_messages); Console.WriteLine("Pushed Updated List on Callback Channel: {0}", _callback Channels[i].GetHashCode()); } catch (Exception ex) { Console.WriteLine("Service threw exception while communicating on Callback Channel: {0}", _callback Channels[i].GetHashCode()); Console.WriteLine("Exception Type: {0} Description: {1}", ex.GetType(), ex.Message); _callbackChannels.RemoveAt(i); } } } } }

236

Chapter 6

Communication inaMulti-Agent System Using WCF

Notice the AgentCommunicationService class has the attributes InstanceContextMode = InstanceContextMode.Single, ConcurrencyMode = ConcurrencyMode.Multiple defined by the ServiceBehavior class; as its name suggests, this class allows us to define various behaviors for the service. The first sets it as a Singleton class; thus, all service calls will be handled by the same service instance, and all agents will refer to the same message and client callback channel list, as those fields were declared static. The latter allows for concurrency to occur and for you to have a multi-thread service, thus permitting each call to be handled in parallel. The synchronization of the service object will be handled using the SyncRoot pattern and the lock statement in C#.

Note Locking public objects is not a good practice. A public object can be locked by anyone, creating unexpected deadlocks. As a result, you should use caution when locking an object that is exposed to the outside world. The SyncRoot pattern guarantees that this scenario does not occur by using a private, internal object to do the locking. The lock statement acts as a key for objects. Imagine a man who wants to enter a room and obtains a key from the owner, and while he is in the room no one else can access it. When he leaves he gives the key back to the owner so the next person in the line can obtain the key and enter the room. The code that prevents multiple threads from accessing and modifying data simultaneously is called thread-safe code. The Subscriber() method (operation) gets the callback channel of the client and checks whether it has been added in the callback channel list, adding it in case it has not been. If the client has not accessed the service before it sends it the latestMessage list. In the Send() method we must ensure that only one thread at a time obtains access to the list, as that’s the reason for the lock statement. Once we have added the message we loop through every callback channel and 237

Chapter 6

Communication inaMulti-Agent System Using WCF

inform the rest of the agents (clients) of the new addition by calling their SendUpdatedList() method. This iteration process is done backward because we will need to remove any channel that may have changed its state to close or throw an exception. As mentioned before, we need to create a Proxy class to interact with the service. To create a duplex proxy we need to design a class that inherits from DuplexClientBase and then create the service contract (Listing 6-4).

Listing 6-4. Proxy Implementation public class AgentCommunicationServiceClient : DuplexClientBase , IAgentCommunicationService { public AgentCommunicationServiceClient(Instance Context callbackInstance, WSDualHttpBinding binding, EndpointAddress endpointAddress) : base(callbackInstance, binding, endpointAddress) { } public void Subscribe() { Channel.Subscribe(); } public void Send(string from, string to, string message) { Channel.Send(from, to, message); } } As we can see from Listing 6-4, the implementation of the proxy class is pretty straightforward—simply forward every call to the Channel property (of type IAgentCommunicationService) provided by the parent class

238

Chapter 6

Communication inaMulti-Agent System Using WCF

DuplexClientBase. In the Send method, we included arguments string from, string to. We’ll use these arguments in the next chapter to filter messages from and to agents. The concrete implementation of the callback contract class is shown in Listing 6-5.

Listing 6-5. Callback Contract Implementation [CallbackBehavior(UseSynchronizationContext = false)] public class AgentCommunicationCallback : IAgent CommunicationCallback { public event EventHandler ServiceCallbackEvent; privateSynchronizationContext _syncContext = AsyncOperation Manager.SynchronizationContext; public void SendUpdatedList(List items) { _syncContext.Post(new SendOrPostCallback(OnService CallbackEvent), new UpdatedListEventArgs(items)); } private void OnServiceCallbackEvent(object state) { EventHandler handler = ServiceCallbackEvent; var e = state as UpdatedListEventArgs; if (handler != null) { handler(this, e); } } } 239

Chapter 6

Communication inaMulti-Agent System Using WCF

Let’s remember that the callback contract is the one handling the “push updates” received from the service contract. By default, the callback contract synchronizes all calls on the current synchronization context. If your client is a Windows Forms application this behavior would result in the code’s being executed on the user-interface thread, which is not a good idea. In order to communicate the results obtained on the operation thread to the UI thread we will use AsyncOperationManager, a class that .NET incorporates for concurrency management. This class contains a SynchronizationContext property, which returns the synchronization context for the application calling it. The purpose, in the end, for using these classes is sharing data between the UI thread and the operation thread.

Note A synchronization context provides a way to queue a unit of work to a particular context. It could allow worker threads to dispatch messages to the UI synchronization context. Only the UI synchronization context is allowed to manipulate the UI controls; therefore, if we attempted to update the UI from another context it would result in an illegal operation, causing an exception to be thrown. We’ll use the Post method of the SynchronizationContext class to asynchronously queue messages to the UI synchronization context. The Post method takes two arguments: a delegate called SendOrPostCallback representing the callback method we need to execute after the message is dispatched to the UI synchronization context, and an object that is submitted to the delegate. We create the SendOrPostCallback delegate by passing in the OnServiceCallbackEvent method that has been implemented in the Callback class. We also create an instance of the UpdatedListEventArgs (Listing 6-6) class and submit the new list of messages in the constructor. The delegate and the event arguments class 240

Chapter 6

Communication inaMulti-Agent System Using WCF

instance are used as arguments to the Post method. In this manner, our event-invocation method is capable of obtaining the event arguments when it is being marshalled from the worker thread to the UI thread. Subscribers (clients such as Windows Forms, console application, and so on) to our ServiceCallbackEvent can then handle the event when it is triggered. Setting the UseSynchronizationContext attribute to false allows the callback operations to be distributed among different threads.

Listing 6-6. Class Used as Event Argument to Update the Message List on the Client Application (Windows Forms) public class UpdatedListEventArgs : EventArgs { public List MessageList { get; set; } public UpdatedListEventArgs(List messages) { MessageList = messages; } } Now that we have presented concrete implementations for all contracts, let’s present the application acting as host for the service (Listing 6-7).

Listing 6-7. Service Being Hosted in a Console Application static void Main(string[] args) { // Step 1 Create a URI to serve as the base address. var baseAddress = new Uri("http://localhost:9090/");

241

Chapter 6

Communication inaMulti-Agent System Using WCF

// Step 2 Create a ServiceHost instance var selfHost = new ServiceHost(typeof(Agent CommunicationService), baseAddress); try { // Step 3 Add a service endpoint. selfHost.AddServiceEndpoint(typeof(IAgent CommunicationService), new WSDualHttpBinding(WSDualHttpSecurity Mode.None), "AgentCommunicationService"); // Step 4 Enable Metadata Exchange and Add MEX endpoint var smb = new ServiceMetadataBehavior { HttpGetEnabled = true }; selfHost.Description.Behaviors.Add(smb); selfHost.AddServiceEndpoint(ServiceMetadata Behavior.MexContractName, MetadataExchangeBindings. CreateMexHttpBinding(), baseAddress + "mex"); // Step 5 Start the service. selfHost.Open(); Console.WriteLine("The service is ready."); Console.WriteLine("Listening at: {0}", baseAddress); Console.WriteLine("Press to terminate service."); Console.WriteLine(); Console.ReadLine(); // Close the ServiceHostBase to shut down the service. 242

Chapter 6

Communication inaMulti-Agent System Using WCF

selfHost.Close(); } catch (CommunicationException ce) { Console.WriteLine("An exception occurred: {0}", ce.Message); selfHost.Abort(); } } The steps for creating the service are clearly presented in Listing 6-7. In this case, we are hosting our service in a console application. Notice that we will not be using or editing the app.config file; on the contrary, all binding, address, and contract configuration is made programmatically.

Note The WCF bindings supporting duplex services are WSDualHttp Binding, NetTcpBinding, and NetNamedPipeBinding. The client application will be a Windows Forms application that has the code shown in Listing 6-8.

Listing 6-8. Client Application public partial class AgentClient : Form { private const string ServiceEndpointUri = "http:// localhost:9090/AgentCommunicationService"; public AgentCommunicationServiceClient Proxy { get; set; } public AgentClient() {

243

Chapter 6

Communication inaMulti-Agent System Using WCF

InitializeComponent(); InitializeClient(); } private void InitializeClient() { if (Proxy != null) { try { Proxy.Close(); } catch { Proxy.Abort(); } } var callback = new AgentCommunicationCallback(); callback.ServiceCallbackEvent += HandleServiceCallbackEvent; var instanceContext = new InstanceContext(callback); var dualHttpBinding = new WSDualHttpBinding(WSDual HttpSecurityMode.None); var endpointAddress = new EndpointAddress(Service EndpointUri); Proxy = new AgentCommunicationServiceClient(instance Context, dualHttpBinding, endpointAddress); Proxy.Open(); Proxy.Subscribe(); }

244

Chapter 6

Communication inaMulti-Agent System Using WCF

private void HandleServiceCallbackEvent(object sender, UpdatedListEventArgs e) { List list = e.MessageList; if (list != null && list.Count > 0) messageList.DataSource = list; } private void SendBtnClick(object sender, EventArgs e) { Proxy.Send("", "", wordBox.Text.Trim()); wordBox.Clear(); } } As expected, the client application (Figure6-3) contains a field of type AgentCommunicationServiceClient, which represents the proxy it will be using for subscribing and communicating with the service. The HandleServiceCallbackEvent is the event that will be triggered when a new message is added to the list; this is directly related to the callback contract and the OnServiceCallbackEvent event we recently described. The SendBtnClick event is fired when a user clicks the Send button of the client’s UI and sends a new message.

Figure 6-3. Client UI in Windows Forms 245

Chapter 6

Communication inaMulti-Agent System Using WCF

Now that we have all the pieces together, let’s test the application and see how different agents communicate and receive messages. First, let’s run the console application that is hosting the service.

Note You would typically need administrator rights to launch the service application. If you are experiencing any issues running the application, try running it as administrator. Then, let’s run as many clients as we want. In this case, we would be satisfied by just executing three clients. The scenario described would be the one illustrated in Figure6-4.

Figure 6-4. Executing the service and three clients

246

Chapter 6

Communication inaMulti-Agent System Using WCF

Now we can play with the application and send messages from any of the clients. The result will be a shared list of all messages as seen in Figure6-5.

Figure 6-5. Agents exchanging messages in a WCF Publisher/ Subscriber application In the next chapter, we will slightly modify the WCF communication application introduced in the last few sections to adjust it to our multi- agent system of cleaning agents example. In the cleaning agents MAS program, clients will be agents that will be communicating through a WCF service that is acting as a message broker (publisher). Concepts examined in Chapter 5, such as cooperation, coordination, Contract Net, and social laws will be covered again in the cited example and in a practical manner will be implemented via classes and methods in C#.

247

Chapter 6

Communication inaMulti-Agent System Using WCF

Summary In this chapter, we explained some of the basics of WCF (services, contracts, addresses, bindings, and endpoints) and also a common pattern in network applications, the Publisher/Subscriber model. We introduced and described duplex services and some of their features, like the callback contract. We implemented a WCF program that simulated the communication of several agents, using a service hosted on a console application as message broker and a Windows Forms application for clients. In the following chapter, we will insert this application into a much bigger program that simulates the process of a multi-agent system whose task is getting rid of all the dirt in an n x m room.

248

CHAPTER 7

Cleaning Agents: AMulti-Agent System Problem Throughout Chapters 5 and 6 we studied multi-agent systems (MAS) and multi-agent communication. We introduced concepts such as agent platform, agent architecture, coordination, cooperation, social laws, and much more; we also detailed a practical problem where we created a multi-agent communication module using Windows Communication Foundation (WCF). In this chapter, we’ll analyze a complete practical problem where we will put all the pieces together and develop an MAS where n cleaning agents will be dealing with the task of cleaning an n x m room of its dirt. This problem will allow us to include many of the concepts and definitions studied before and also attach the WCF communication module created in Chapter 6 as the MAS communication module that every agent in the system will integrate. The cleaning problem is a great benchmark or scenario by which to understand how we can use an MAS to solve a task, such as cleaning, in a much shorter time and using fewer resources than with just a single agent.

© Arnaldo Pérez Castaño 2018 A. Pérez Castaño, Practical Artificial Intelligence, https://doi.org/10.1007/978-1-4842-3357-3_7

249

Chapter 7

Cleaning Agents: AMulti-Agent System Problem

Note Every robot in the cleaning problem will be using WCF at the core of their communication module and Windows Forms to display messages received.

P rogram Structure The application will have a structure like the one depicted in Figure7-1. The program comprises Communication, GUI (Graphical User Interface), Negotiation, Planning, and Platform modules. The Communication and Planning modules will not be analyzed in this chapter (except for the communication language, FipaAcl C# class) as they were previously studied. For further reference please download the source code associated with this book.

Figure 7-1. Program structure The GUI module will contain two Windows Forms applications—one for graphically representing the room with every agent on it and their interactions, the other for representing the agent message board.

250

Chapter 7

Cleaning Agents: AMulti-Agent System Problem

The Negotiation module will contain an implementation of the Contract Net task-sharing method, with every stage implemented as a static C# method within a ContractNet class. The Platform module will contain an implementation of an agent platform and some of its functionalities (agent location via dictionary, Decide Roles service for task sharing, references to manager and contractors, and so forth). It will serve as support for other classes. Within the Communication module we’ll include the Agent Communication Language (ACL) module, which contains a tiny, simplified version of a FIPA-ACL, including a few performatives.

Note In order to simplify the planning task in this MAS example, we will assume that the number of columns (M ) is always divisible by the number of agents (S ) in the MAS, i.e., M % S == 0. This will allow us to simply assign M / S columns to each agent for cleaning.

C leaning Task To represent and encode the cleaning task we have created the class illustrated in Listing 7-1.

Listing 7-1. CleaningTask Class public class CleaningTask { public int Count { get; set; } public int M { get; set; } public List SubDivide { get; set; } public IEnumerable SubTasks { get; set; }

251

Chapter 7

Cleaning Agents: AMulti-Agent System Problem

public CleaningTask(int m, int agents) { M = m; Count = agents; SubDivide = new List(); Divide(); SubTasks = BuildTasks(); } /// /// For the division we assume that M % Count = 0, i.e. /// the number of columns is always divisible by the number of agents. /// private void Divide() { var div = M / Count; for (var i = 0; i < M; i += div) SubDivide.Add(new Tuple(i, i + div - 1)); } private IEnumerable BuildTasks() { var result = new string[SubDivide.Count]; for (var i = 0; i < SubDivide.Count; i++) result[i] = "clean(" + SubDivide[i].Item1 + "," + SubDivide[i].Item2 + ")"; return result; } }

252

Chapter 7

Cleaning Agents: AMulti-Agent System Problem

The class contains the following fields or properties: •

Count: integer representing the number of agents participating in the cleaning task

•

M: integer representing the number of columns in the room

•

SubDivide: List representing the equitable column division made considering the number of agents and columns

•

SubTasks: IEnumerable representing the set of tasks that need to be executed in order to complete the global task (cleaning the entire room). Each task is defined in a self-created inner language that our mini FipaAcl will be using.

On the other hand, the CleaningTask class exposes these methods: •

Divide(): divides the global task of cleaning a room into smaller subtasks. Each subtask will consist of a subset of contiguous columns to be cleaned. It stores in the SubDivide property a set of tuples, each defining a range of columns to be cleaned; e.g., (0, 2) will indicate the subtask of cleaning columns 0 up to 2.

•

BuildTasks(): returns an IEnumerable containing every subtask in a self-created language that will be used later for transmitting information via the communication module and using FIPA as ACL.

In trying to maintain a well-modularized application, the CleaningTask class merely deals with operations related to cleaning issues. In the next section, we’ll take a look at the Cleaning Agent platform.

253

Chapter 7

Cleaning Agents: AMulti-Agent System Problem

Cleaning Agent Platform The Cleaning Agent platform is represented by the CleaningAgentPlatform class, whose code can be seen in Listing 7-2.

Listing 7-2. CleaningAgentPlatform Class public class CleaningAgentPlatform { public Dictionary Directory { get; set; } public IEnumerable Agents { get; set; } public IEnumerable Contractors { get; set; } public MasCleaningAgent Manager { get; set; } public CleaningTask Task { get; set; } public CleaningAgentPlatform(IEnumerable agents, CleaningTask task) { Agents = new List(agents); Directory = new Dictionary(); Task = task; foreach (var cleaningAgent in Agents) { Directory.Add(cleaningAgent.Id, cleaningAgent); cleaningAgent.Platform = this; } DecideRoles(); }

254

Chapter 7

Cleaning Agents: AMulti-Agent System Problem

public void DecideRoles() { // Manager Role Manager = Agents.First(a => a.CleanedCells.Count == Agents.Max(p => p.CleanedCells.Count)); Manager.Role = ContractRole.Manager; // Contract Roles Contractors = new List(Agents. Where(a => a.Id != Manager.Id)); foreach (var cleaningAgent in Contractors) cleaningAgent.Role = ContractRole.Contractor; (Contractors as List).Add(Manager); } } This class contains the following properties or fields: •

Directory: dictionary containing the ID of the agent and a reference to it as key–value pairs

•

Agents: IEnumerable containing the set of agents

•

Contractors: IEnumerable containing the set of contractors in a Contract Net

•

Manager: reference to the manager in a Contract Net

•

Task: cleaning task to be executed

This class contains two functions: a constructor and the DecideRoles() method. In the constructor, we initialize every property and then add every agent to the directory, referencing the Platform property of agents to point to this platform. The DecideRoles() method decides which agent is selected as manager, while the rest are regarded as contractors. In this case, the criteria for manager selection is to select the agent with the highest number of cells cleaned; this is equivalent to saying “Pick the most experienced agent, the one who has worked the most.” 255

Chapter 7

Cleaning Agents: AMulti-Agent System Problem

Note In this case, we also add the manager to the list of contractors because we would like him not only to direct the operation but also to take part in it and clean a range of columns of the room as any other contractor would do.

C ontract Net The Contract Net task-sharing mechanism is represented by the ContractNet class; the role assumed by each agent is defined in the ContractRole enum. Both are described in Listing 7-3.

Listing 7-3. ContractNet Class public class ContractNet { public static IEnumerable Announcement(CleaningTask cleaningTask, MasCleaningAgent manager, IEnumerable contractors, FipaAcl language) { var tasks = cleaningTask.SubTasks; foreach (var contractor in contractors) { foreach (var task in tasks) language.Message(Performative.Cfp, manager.Id.ToString(), contractor.Id.ToString(), task); }

256

Chapter 7

Cleaning Agents: AMulti-Agent System Problem

return tasks; } public static void Bidding(IEnumerable tasks, IEnumerable contractors) { foreach (var contractor in contractors) contractor.Bid(tasks); } public static void Awarding(List messages, MasCleaningAgent manager, IEnumerable contractors, CleaningTask task, FipaAcl language) { var agentsAssigned = new List(); var messagesToDict = messages.ConvertAll(FipaAcl. MessagesToDict); // Processing bids foreach (var colRange in task.SubDivide) { var firstCol = colRange.Item1; var secondCol = colRange.Item2; // Bids for first column var bidsFirstCol = new List(); // Bids for second column var bidsSecondCol = new List();

257

Chapter 7

Cleaning Agents: AMulti-Agent System Problem

foreach (var contractor in contractors) { // Skip agents that have been already assigned if (agentsAssigned.Exists(tuple => tuple. Item1.Id == contractor.Id)) continue; var c = contractor; // Get messages from current contractor var messagesFromContractor = messagesToDict. FindAll(m => m.ContainsKey("from") && m["from"] == c.Id.ToString()); var bids = FipaAcl.GetContent(messagesFrom Contractor); // Bids to first column in the range column var bidsContractorFirstCol = bids.FindAll(b => b.Item2.Item2 == firstCol); // Bids to second column in the range column var bidsContractorSecondCol = bids.FindAll(b => b.Item2.Item2 == secondCol); if (bidsContractorFirstCol.Count > 0) { bidsFirstCol.Add( new KeyValuePair(contractor, bidsContractorFirstCol)); } if (bidsContractorSecondCol.Count > 0) {

258

Chapter 7

Cleaning Agents: AMulti-Agent System Problem

bidsSecondCol.Add( new KeyValuePair(contractor, bidsContractorSecondCol)); } } // Sorts to have at the beginning of the list the best bidders (closest agents) bidsFirstCol.Sort(Comparison); bidsSecondCol.Sort(Comparison); var closestAgentFirst = bidsFirstCol. FirstOrDefault(); var closestAgentSecond = bidsSecondCol. FirstOrDefault(); // Sorts again to find closest end if (closestAgentFirst.Value != null) closestAgentFirst.Value.Sort(Comparison); if (closestAgentSecond.Value != null) closestAgentSecond.Value.Sort(Comparison); // Assigns agent to column range if (closestAgentFirst.Value != null && closestAgentSecond.Value != null) { if (closestAgentFirst.Value.First().Item1 >= closestAgentSecond.Value.First().Item1) agentsAssigned.Add(new Tuple(closestAgentSecond.Key, closestAgentSecond.Value.First().Item2)); 259

Chapter 7

Cleaning Agents: AMulti-Agent System Problem

else agentsAssigned.Add(new Tuple(closestAgentFirst.Key, closestAgentFirst.Value.First().Item2)); } else if (closestAgentFirst.Value == null) agentsAssigned.Add(new Tuple(closestAgentSecond.Key, closestAgentSecond.Value.First().Item2)); else agentsAssigned.Add(new Tuple(closestAgentFirst.Key, closestAgentFirst.Value.First().Item2)); } // Transmits the accepted proposal for each agent. foreach (var assignment in agentsAssigned) language.Message(Performative.Accept, manager. Id.ToString(), assignment.Item1.Id.ToString(), "clean(" + assignment.Item2.Item1 + "," + assignment. Item2.Item2 + ")"); } private static int Comparison(Tuple tupleA, Tuple tupleB) { if (tupleA.Item1 > tupleB.Item1) return 1; 260

Chapter 7

Cleaning Agents: AMulti-Agent System Problem

if (tupleA.Item1 < tupleB.Item1) return -1; return 0; } private static int Comparison(KeyValuePair bidsAgentA, KeyValuePair bidsAgentB) { if (bidsAgentA.Value.Min(p => p.Item1) > bidsAgentB.Value.Min(p => p.Item1)) return 1; if (bidsAgentA.Value.Min(p => p.Item1) < bidsAgentB.Value.Min(p => p.Item1)) return -1; return 0; } } public enum ContractRole { Contractor, Manager, None } This class contains the following static methods: •

Announcement(): a message is sent from the manager to every contractor, announcing every task to be completed

•

Bidding(): each agent is asked for a bid that considers the set of tasks to be completed. Bidding on the agent side is executed in the Bid() method of the MasCleaningAgent class. 261

Chapter 7

Cleaning Agents: AMulti-Agent System Problem

•

Awarding(): method executing the final stage of the task-sharing mechanism. To award a range of columns x- x' to a contractor (agent), it calculates the distance of every agent to the four ends of that column range—i.e., cells(0, x), (n- 1, x) at the first column and cells(0, x'), (n- 1, x') at the second column—and then awards that column range to the agent that is the closest (minimum Block or Manhattan distance) to any of the four ends. The bid of the agent contains a tuple defining the closest end and a double representing the distance to that end. Refer to the code comments for more details.

•

Comparison(): Both methods relate to sorting a list of elements by considering a double value that indicates its distance to a column.

Every method was created as a service of the class; in other words, as a static method that requires no instance of the class to be called.

F IPA-ACL In order to communicate cleaning-related issues among agents, we created a tiny language for processing these types of commands. This mini-language resembles the FIPA language and contains an inner language that merely includes the clean(x, y) statement telling agents to clean all columns from x to y. The FipaAcl class and the Performative enum are both illustrated in Listing 7-4.

262

Chapter 7

Cleaning Agents: AMulti-Agent System Problem

Listing 7-4. FipaACL Class public class FipaAcl { public AgentCommunicationServiceClient Communication { get; set; } public FipaAcl(AgentCommunicationServiceClient communication) { Communication = communication; } public void Message(Performative p, string senderId, string receiverId, string content) { switch (p) { case Performative.Accept: ThreadPool.QueueUserWorkItem(delegate { Communication.Send(senderId, receiverId, "accept[content:" + content + ";]"); }); break; case Performative.Cfp: ThreadPool.QueueUserWorkItem(delegate { Communication.Send(senderId, receiverId, "cfp[content:" + content + ";]"); }); break;

263

Chapter 7

Cleaning Agents: AMulti-Agent System Problem

case Performative.Proposal: ThreadPool.QueueUserWorkItem(delegate { Communication.Send(senderId, receiverId, "proposal[from:" + senderId + ";content:" + content + "]"); }); break; } } public static string GetPerformative(string task) { return task.Substring(0, task.IndexOf('[')); } public static string GetInnerMessage(string task) { return task.Substring(task.IndexOf('[') + 1, task.LastIndexOf(']')- task.IndexOf('[')- 1); } public static Dictionary MessageToDict(string innerMessage) { var result = new Dictionary(); var items = innerMessage.Split(';'); var contentItems = new List(); foreach (var item in items) if (!string.IsNullOrEmpty(item)) contentItems.AddRange(item.Split(':')); for (int i = 0; i < contentItems.Count; i += 2) result.Add(contentItems[i], contentItems[i + 1]); return result; } 264

Chapter 7

Cleaning Agents: AMulti-Agent System Problem

public static Dictionary MessagesToDict(string message) { return MessageToDict(GetInnerMessage(message)); } public static List GetContent(List messagesFromContractor) { var result = new List(); foreach (var msg in messagesFromContractor) { var content = msg["content"]; var values = content.Split(','); result.Add(new Tuple(double.Parse(values[0]), new Tuple(int.Parse(values[1]), int.Parse(values[2])))); } return result; } } public enum Performative { Accept, Cfp, Inform, Proposal }

265

Chapter 7

Cleaning Agents: AMulti-Agent System Problem

Notice that every agent communication is executed using the QueueUserWorkItem method of the ThreadPool class. Starting a new thread can be a very expensive operation; therefore, we use the threadpool facilities to reuse threads and reduce cost. In this manner, we queue methods for execution under different threads that are drawn from the thread pool. The FipaACL class includes an AgentCommunicationServiceClient communication property (recall from Chapter 6 that AgentCommunicationServiceClient is the proxy that establishes communication between client and service) that is used to transmit messages to other agents. FipaACL incorporates the following methods:

266

•

Message(): depending on the type of performative, creates and sends a new message using the senderId, receiverId, and content strings provided as arguments.

•

GetPerformative(): gets the performative of the message provided as argument; e.g., for a message such as cfp[content: clean(0,2)] the performative would be cfp

•

GetInnerMessage(): gets the inner message; e.g., if the entire message is something like cfp[from: 2312; content: clean(0,2)] then from: 2312; content: clean(0,2) represents the inner message

•

MessageToDict(): assuming an inner message is supplied as argument, it translates that inner message into a dictionary; e.g., from an inner message such as from: 2312; content: clean(0,2) the resulting dictionary would be { 'from': 2312, 'content': 'clean(0,2)' }

Chapter 7

Cleaning Agents: AMulti-Agent System Problem

•

MessagesToDict(): gets the inner message of a message submitted as an argument and returns the dictionary resulting from the MessageToDict() method

•

GetContent(): gets the set of values contained within the content label of the inner message. It assumes each message corresponds to a contractor’s bid; therefore, it contains three elements: a distance double and a pair of integers matching a column range; e.g., 2.0, 1, 1 will add the tuple

The only components of the MAS cleaning example presented in this chapter that use the FipaAcl class are the ContractNet and MasCleaningAgent classes; the latter will be the topic of the next section.

MAS Cleaning Agent Agents in the cleaning MAS example are objects of the MasCleaningAgent class, which contains the set of properties, fields, and constructor shown in Listing 7-5.

Listing 7-5. MasCleaningAgent Class, Including Fields, Properties, and Constructor public class MasCleaningAgent { public Guid Id { get; set; } public int X { get; set; } public int Y { get; set; } public bool TaskFinished { get; set; } public Timer ReactionTime { get; set; } public FipaAcl Language { get; set; }

267

Chapter 7

Cleaning Agents: AMulti-Agent System Problem

public CleaningAgentPlatform Platform { get; set; } public List CleanedCells; public ContractRole Role { get; set; } public Color Color; public bool AwaitingBids { get; set; } public bool AwaitingTaskAssignment { get; set; } public bool AnnouncementMade { get; set; } public bool TaskDistributed { get; set; } public Plan Plan { get; set; } public bool InCleaningArea { get; set; } public List AreaTobeCleaned; private readonly int[,] _room; private readonly Form _gui; private Messaging _messageBoardWin; private readonly List _wishList; public MasCleaningAgent(Guid id, int[,] room, Form gui, int x, int y, Color color) { Id = id; X = x; Y = y; _room = room; CleanedCells = new List(); Role = ContractRole.None; _wishList = new List(); Color = color; _gui = gui; Run(); } } 268

Chapter 7

Cleaning Agents: AMulti-Agent System Problem

This class exposes the following properties and fields: •

Id: represents a unique identifier for the agent

•

X: integer representing the x-coordinate of the agent in the room

•

Y: integer representing the y-coordinate of the agent in the room

•

TaskFinished: Boolean value indicating whether the task has been completed

•

ReactionTime: timer defining the reaction time of the agent; i.e., the frequency by which it executes an action

•

Language: mini-Fipa language represented by the FipaAcl class that will be used for parsing and transmitting messages

•

Platform: agent platform used for different services (agent location) and for deciding the role (manager or contractor) of each agent. It’s represented by the CleaningAgentPlatform class.

•

CleanedCells: list of Tuple indicating the cells on the terrain that have already been cleaned by the agent

•

Role: role assumed by the agent (contractor, manager, none)

•

Color: color used by the agent on the room; i.e., on the Windows Forms picture box representing the room

•

AwaitingBids: Boolean value indicating whether the agent is awaiting a bid (for the manager role)

269

Chapter 7

270

Cleaning Agents: AMulti-Agent System Problem

•

AwaitingTaskAssignment: Boolean value indicating whether the agent is awaiting a task assignment (for the contractor role)

•

AnnouncementMade: Boolean value indicating whether an announcement has been made (for the manager role)

•

TaskDistributed: Boolean value indicating whether tasks have been distributed (for the manager role)

•

Plan: instance of the Plan class used for executing path-finding algorithms. This is the Plan class presented in Chapter 4, “Mars Rover.”

•

InCleaningArea: Boolean value indicating whether the agent is in the cleaning area assigned by the manager after a Contract Net task-sharing mechanism has been executed

•

AreaTobeCleaned: list of cells the agent must clean

•

_room: reference to the integer matrix representing the room to be cleaned. A value greater than 0in any cell represents dirt; a value of 0 indicates the cell is clean.

•

_gui: reference to the Windows Forms object that represents the room

•

_messageBoardWin: reference to the Windows Forms representing the message board where all messages received by the agent will be displayed

•

_wishList: list of Tuple representing the wish list or bid list (for the contractor role) of the agent. The second item indicates a cell of the room, and the first item indicates the distance to that cell. This field is used in the bidding process to find the closest column end.

Chapter 7

Cleaning Agents: AMulti-Agent System Problem

In the constructor, we initialize various fields and properties and eventually call the Run() method (Listing 7-6), which will set up everything to start running the agent.

Listing 7-6. Run() Method Starts the Agent by Enabling the Timer and Connecting the Tick Event to the ReactionTimeOnTick() Method private void Run() { _messageBoardWin = new Messaging (Id.ToString()) { StartPosition = FormStartPosition. WindowsDefaultLocation, BackColor = Color, Size = new Size (300, 300), Text = Id.ToString(), Enabled = true }; Language = new FipaAcl(_messageBoardWin.Proxy); _messageBoardWin.Show(); ReactionTime = new Timer { Enabled = true, Interval = 1000 }; ReactionTime.Tick += ReactionTimeOnTick; } In the Run() method we initialize the _messageBoardWin variable as an instance of the Messaging class (Form class that will contain all messages received by the agent). We also initialize the Language property, passing as an argument the proxy created in the Messaging class. Finally, the Timer

271

Chapter 7

Cleaning Agents: AMulti-Agent System Problem

of the agent is enabled and subscribed to the ReactionTimeOnTick (Listing 7-7). This method, which will be executed every second, causes the agent to take action.

Listing 7-7. ReactionTimeOnTick() Method Executed private void ReactionTimeOnTick(object sender, EventArgs eventArgs) { // There's no area assigned for cleaning if (AreaTobeCleaned == null) { if (Role == ContractRole.Manager && AnnouncementMade && !TaskDistributed) { ContractNet.Awarding(_messageBoardWin. Messages, Platform.Manager, Platform. Contractors, Platform.Task, Language); TaskDistributed = true; } if (Role == ContractRole.Manager && !AnnouncementMade) { ContractNet.Announcement(Platform.Task, Platform.Manager, Platform.Contractors, Language); AnnouncementMade = true; Thread.Sleep(2000); } if (Role == ContractRole.Contractor && AwaitingTaskAssignment || Role == ContractRole. Manager && TaskDistributed)

272

Chapter 7

Cleaning Agents: AMulti-Agent System Problem

{ AreaTobeCleaned = SetSocialLaw (_messageBoardWin.Messages); } if (Role == ContractRole.Contractor && !AwaitingTaskAssignment) { Thread.Sleep(2000); ContractNet.Bidding(_messageBoardWin. Messages, Platform.Contractors); AwaitingTaskAssignment = true; } } else { if (!InCleaningArea) { if (Plan == null) { Plan = new Plan(TypesPlan.PathFinding, this); Plan.BuildPlan(new Tuple(X, Y), AreaTobeCleaned.First()); } else if (Plan.Path.Count == 0) InCleaningArea = true; } Action(Perceived()); } _gui.Refresh(); } 273

Chapter 7

Cleaning Agents: AMulti-Agent System Problem

Notice that we put the thread to sleep for 2000 milliseconds to wait for certain operations of other agents to complete. This time may need to be increased as the cardinality of the set of agents increases. The ReactionTimeOnTick() method uses a logic that depends on two scenarios: the agent has a cleaning area assigned or no area has been assigned. If no area has been assigned, that indicates no task sharing has been accomplished among agents, and so a Contract Net mechanism must be started. The different scenarios for when no cleaning area has been defined for the agent are the following: •

If the agent is a manager and an announcement has been made and tasks have not been distributed yet then the agent must enter an awarding phase.

•

If the agent is a manager and no announcement has been made then the agent must enter an announcement phase.

•

If the agent is a contractor and is awaiting a task assignment or the agent is a manager and tasks have been distributed then it should assign an area to be cleaned by setting a social law; we will detail this social law soon.

•

If the agent is a contractor and is awaiting a task assignment then it must enter a bidding phase.

The bidding process of the agent follows the logic described by the code shown in Listing 7-8.

Listing 7-8. Bid Method of the Agent public void Bid(IEnumerable tasks) { var n = _room.GetLength(0); _wishList.Clear();

274

Chapter 7

Cleaning Agents: AMulti-Agent System Problem

foreach (var task in tasks) { var innerMessage = FipaAcl.GetInnerMessage(task); var messageDict = FipaAcl. MessageToDict(innerMessage); var content = messageDict["content"]; var subtask = content.Substring(0, content. IndexOf('(')); var cols = new string[2]; switch (subtask) { case "clean": var temp = content.Substring(content. IndexOf('(') + 1, content.Lengthcontent.IndexOf('(')- 2); cols = temp.Split(','); break; } var colRange = new Tuple(int. Parse(cols[0]), int.Parse(cols[1])); for (var i = colRange.Item1; i < colRange. Item2; i++) { // Distance to extreme points for each column var end1 = new Tuple(0, i); var end2 = new Tuple(n- 1, i); var dist1 = ManhattanDistance(end1, new Tuple(X, Y));

275

Chapter 7

Cleaning Agents: AMulti-Agent System Problem

var dist2 = ManhattanDistance(end2, new Tuple(X, Y)); _wishList.Add(new Tuple(dist1, end1)); _wishList.Add(new Tuple(dist2, end2)); } } _wishList.Sort(Comparison); foreach (var bid in _wishList) Language.Message(Performative.Proposal, Id.ToString(), Platform.Manager.Id.ToString(), bid.Item1 + "," + bid.Item2.Item1 + "," + bid. Item2.Item2); } The Bid() method receives the list of tasks as input, parses every task message contained in the list, and then, having the column range detailed in each incoming message task, finds the distance to the four possible column ends. Finally, it sorts the _wishList of all possible distances to column ends and transmits them (as proposals) to the manager ordered from lowest to highest. When a cleaning area has been assigned, the agent must design a plan (path-finding technique from Chapter 4) to reach its cleaning area. Once in its cleaning area, the agent will follow a social law defined by the method illustrated in Listing 7-9.

276

Chapter 7

Cleaning Agents: AMulti-Agent System Problem

Listing 7-9. SetSocialLaw() Method private List SetSocialLaw(List messages) { if (!messages.Exists(m => FipaAcl. GetPerformative(m) == "accept")) return null; var informMsg = messages.First(m => FipaAcl. GetPerformative(m) == "accept"); var content = FipaAcl.MessageToDict(FipaAcl. GetInnerMessage(informMsg)); var directive = content["content"]; var temp = directive.Substring(directive.IndexOf('(') + 1, directive.Length- directive.IndexOf('(')- 2); var pos = temp.Split(','); var posTuple = new Tuple(int.Parse(pos[0]), int. Parse(pos[1])); var colsTuple = new Tuple(posTuple.Item2, posTuple. Item2 + _room.GetLength(1) / Platform.Directory.Count- 1); var result = new List(); var startRow = _room.GetLength(0)- 1; var dx = -1; // Generate path to clean for (var col = colsTuple.Item1; col = 0; row+=dx) result.Add(new Tuple(row, col)); } return result; } While in their cleaning area, and for the purpose of having an ordered, uniform way of executing their cleaning task, the SetSocialLaw() method will define the path followed by agents during their cleaning process; this social law is illustrated in Figure7-2.

Figure 7-2. Social law followed by agents If there’s an active plan (for going to the designated cleaning area) then a move from this plan is executed and deleted from the plan’s path. According to the percepts received (clean, dirty), the agent will choose to update its state or clean the dirty cell. If the area to be cleaned still contains some unvisited cells, then we move to that cell. If the area to be cleaned has no more cells, then the task can be considered finished. This is the process executed by the Action() method seen in Listing 7-10. 278

Chapter 7

Cleaning Agents: AMulti-Agent System Problem

Listing 7-10. Action() Method public void Action(List percepts) { if (Plan.Path.Count > 0) { var nextAction = Plan.NextAction(); var percept = percepts.Find(p => p.Item1 == nextAction); Move(percept.Item1); return; } if (percepts.Exists(p => p.Item1 == TypesPercept. Clean)) UpdateState(); if (percepts.Exists(p => p.Item1 == TypesPercept. Dirty)) { Clean(); return; } if (AreaTobeCleaned.Count > 0) { var nextCell = AreaTobeCleaned.First(); AreaTobeCleaned.RemoveAt(0); Move(GetMove(nextCell)); }

279

Chapter 7

Cleaning Agents: AMulti-Agent System Problem

else { if (!TaskFinished) { TaskFinished = true; MessageBox.Show("Task Finished"); } } } Other methods of the MasCleaningAgent class, such as Clean(), IsDirty(), Move(), GetMove(), UpdateState(), ManhattanDistance(), MoveAvailable(), and Perceived() share a high degree of similitude with methods of the same name defined in the example from Chapter 2; thus, we will not be including their codes in this chapter. For further reference please consult the source code associated with this book.

G UI As mentioned before, we will include in the project two Windows Forms applications—one for showing a list of messages received by the agent and another for graphically representing the room. The Messaging class of the message board acts as a client; it incorporates the code presented in the last chapter in the client’s Windows Forms application. The service in this case is called from a console application in similar fashion to the one we detailed in Chapter 6. Even though the code of the Room class is merely a Windows Forms code, we present it in Listing 7-11 to serve as reference.

280

Chapter 7

Cleaning Agents: AMulti-Agent System Problem

Listing 7-11. Room Class public partial class Room : Form { public List CleaningAgents; private int _n; private int _m; private int[,] _room; public Room(int n, int m, int[,] room) { _n = n; _m = m; _room = room; CleaningAgents = new List(); InitializeComponent(); } private void RoomPicturePaint(object sender, PaintEventArgs e) { var pen = new Pen(Color.Wheat); var cellWidth = roomPicture.Width / _m; var cellHeight = roomPicture.Height / _n; // Draw room grid for (var i = 0; i < _m; i++) e.Graphics.DrawLine(pen, new Point (i * cellWidth, 0), new Point(i * cellWidth, i * cellWidth + roomPicture.Height));

281

Chapter 7

Cleaning Agents: AMulti-Agent System Problem

for (var i = 0; i < _n; i++) e.Graphics.DrawLine(pen, new Point(0, i * cellHeight), new Point(i * cellHeight + roomPicture.Width, i * cellHeight)); // Draw agents for (var i = 0; i < CleaningAgents.Count; i++) e.Graphics.FillEllipse(new SolidBrush (CleaningAgents[i].Color), CleaningAgents[i].Y * cellWidth, CleaningAgents[i].X * cellHeight, cellWidth, cellHeight); // Draw Dirt for (var i = 0; i < _n; i++) { for (var j = 0; j < _m; j++) if (_room[i, j] > 0) e.Graphics.DrawImage(new Bitmap("rocktransparency.png"), j * cellWidth, i * cellHeight, cellWidth, cellHeight); } } private void RoomPictureResize(object sender, EventArgs e) { Refresh(); } } In the Room class, we implemented the Paint event and the PictureResize event of the PictureBox, where all elements (dirt, agents) are graphically represented. Agents are drawn as ellipses of a color defined

282

Chapter 7

Cleaning Agents: AMulti-Agent System Problem

by the Color agent property, and dirt is drawn as images. When agents clean dirty cells, the dirt will vanish (image no longer painted), and the global task will end when no cell contains a picture of dirt.

R unning theApplication Now that we’ve finished building an MAS program that incorporates all topics described in the preceding three chapters, let us run and look at the complete application and how the agents cooperate, coordinate, and are actually capable of cleaning an n x m room. Remember we are assuming the number of columns is divisible by the number of agents, which simplifies our planning process. The reader can easily change this strategy, transforming it into a more general strategy—one that will allow him to plan the cleaning task for any number of agents. We embed the WCF service in the console application where we also declare all agents, platform, and the room GUI (Listing 7-12).

Listing 7-12. Setting Up and Starting the Application in a Console Application Project var room = new [,] { {0, {0, {0, {0, {2, {0, {0, {0, {0, {0, };

0, 0, 0, 0, 0, 0, 0, 0, 0, 0,

0, 0, 0, 0, 0, 0, 0, 0, 0, 0,

0, 0, 0, 0, 1, 0, 0, 0, 0, 0,

0, 0, 0, 0, 0, 0, 0, 0, 0, 0,

0, 0, 0, 0, 0, 0, 0, 0, 0, 0,

0, 0, 1, 0, 0, 0, 0, 0, 0, 0,

0, 0, 0, 0, 0, 0, 0, 1, 0, 0,

0, 0, 0, 0, 0, 0, 0, 0, 0, 0,

0}, 0}, 0}, 0}, 0}, 1}, 0}, 0}, 0}, 0} 283

Chapter 7

Cleaning Agents: AMulti-Agent System Problem

Application.EnableVisualStyles(); Application.SetCompatibleTextRenderingDefault(false); const int N = 10; const int M = 10; var roomGui = new Room(N, M, room); // Starts the WCF service. InitCommunicationService(); var clAgent1 = new MasCleaningAgent(Guid.NewGuid(), room, roomGui, 0, 0, Color.Teal); var clAgent2 = new MasCleaningAgent(Guid.NewGuid(), room, roomGui, 1, 1, Color.Yellow); var clAgent3 = new MasCleaningAgent(Guid.NewGuid(), room, roomGui, 0, 0, Color.Tomato); var clAgent4 = new MasCleaningAgent(Guid.NewGuid(), room, roomGui, 1, 1, Color.LightSkyBlue); var clAgent5 = new MasCleaningAgent(Guid.NewGuid(), room, roomGui, 1, 1, Color.Black); roomGui.CleaningAgents = new List { clAgent1, clAgent2, clAgent3, clAgent4, clAgent5 }; var platform = new CleaningAgentPlatform(roomGui. CleaningAgents, new CleaningTask(M, roomGui. CleaningAgents.Count)); Application.Run(roomGui); The InitCommunicationService() method contains the exact lines of code as in the agent service detailed in Chapter 6. The result is the one shown in Figure7-3, where the MAS application starts by having all agents exchange messages in a Contract Net mechanism.

284

Chapter 7

Cleaning Agents: AMulti-Agent System Problem

Figure 7-3. Agents exchanging messages in a Contract Net mechanism; messages received are shown in their Message Board windows 285

Chapter 7

Cleaning Agents: AMulti-Agent System Problem

Once an agreement has been reached and every agent is aware of its designated cleaning area, the cleaning process starts by following the social law previously described. When they complete their subtask, a message box with a “Task Finished” message is displayed (Figure7-4). Each agent thread is put to sleep for a certain time while cleaning a unit of dirt from the room; that way we simulate the cleaning process as it would occur in real life.

286

Chapter 7

Cleaning Agents: AMulti-Agent System Problem

Figure 7-4. Agents cleaning their designated area and displaying the “Task Finished” message once they have completed cleaning their area 287

Chapter 7

Cleaning Agents: AMulti-Agent System Problem

We have finally reached the closing stages of our cleaning agent MAS application. In this particular example, a 10 x 10 room was successfully cleaned by five agents, which distributed the global task of cleaning the entire room into subtasks of cleaning just portions of it; these portions were defined by column ranges. Moreover, communication via a WCF service resulted in a coordination and cooperation strategy. As occurred with the Mars Rover program from Chapter 4, the reader can use this example in an experimental application or improve it with new strategies or methods. The cleaning MAS developed in this book can serve as the foundation or base application for solving other problems that require a more efficient solution when various agents interact and collaborate.

S ummary Chapter 7 ends for now the “Agents” topic of this book, the closing practical problem not only encompassing many of the points studied in Chapters 5 and 6 but also going beyond the scope of detail included in those chapters to be the most thorough, precise chapter up to this point. Going back to the cleaning MAS application, you’ll notice that topics such as logic, firstorder logic, and agents are incorporated as inevitable components of a multi-agent program. In Chapter 8, we’ll begin describing an area that is deeply related to probability and statistics—the very interesting topic of simulation.

288

CHAPTER 8

Simulation Modeling is a basic tool of the human mind that provides us with the ability to create abstract versions of the world, or part of it. These abstract versions can embody a convenient, simplified representation of a situation, object, and so forth and can be used to find a solution to a given problem. Modeling involves imagination and creativity; it underlies our capacity to communicate, generalize, and express meaning or patterns in an intelligent manner. It is usually accepted that modeling is a way of making decisions and predictions about the world and that the purpose of a model must be well defined and understood before the model is created. Models are typically classified as descriptive (they explain or describe the world) or prescriptive (they formulate optimal solutions to problems and are related to the area of optimization). Examples of models of the first type are maps, 3D objects created using computer graphics, or video games. Models of the latter type are heavily related to math and specifically to optimization; in these models, we define a set of constraints for a problem and a goal function to be optimized. Every model possesses three basic features: •

Reference: It represents something, either from the real world or an imaginary world; e.g., building, city.

•

Purpose: It has a logical intention with respect to that which it references; e.g. study, analysis.

•

Cost-effectiveness: It is more effective to use the model than the reference, e.g. blueprint vs. real building, map vs. real city.

© Arnaldo Pérez Castaño 2018 A. Pérez Castaño, Practical Artificial Intelligence, https://doi.org/10.1007/978-1-4842-3357-3_8

289

Chapter 8

Simulation

Simulation is considered a variety of modeling whose purpose is comprehension, planning, prediction, and manipulation. It can be defined broadly as a behavioral or phenomenological approach to modeling; that is, a simulation is an active behavioral analog of its referent.

Note Modeling is one of the most important processes that occurs in the human mind. When modeling we try to create abstract versions of our reality, simplifying it many times to help us solve a problem. Examples of models are maps (such as Google Maps), which represent abstract versions of the world.

What Is Simulation? As occurs with the logic and agent words (it seems like the AI community should get together and try to agree on several definitions), there’s no consensus on what the word simulation means. There is, however, a consensus on the fact that simulation is an imitative and dynamic type of modeling used to model phenomena that must be researched or understood for some reason. When we implement a simulation as a computer program we obtain high flexibility; being in a programming-language environment means that in principle it is possible to refine, maintain, evolve, and extend a computer simulation in ways that are difficult to match in any other environment. Modern programming languages such as C# facilitate the development of modular data and program code, allowing new simulations to be built using pieces or modules of existing ones. Computer simulation is usually divided into analytic and discrete- event approaches. The analytic approach involves mathematical analysis and problems that can be understood or approximated from an analytic

290

Chapter 8

Simulation

perspective. For instance, if the reality being modeled can be accurately described by a set of differential equations (as in the flow of heat over a surface), analytic solutions for those equations can be used to generate the time-dependent behavior required for the simulation. The mathematical elegance of analytic simulation makes it in many scenarios cryptic and incomprehensible; by reducing reality to an abstract mathematical relationship the understanding required could get obscured. There are also cases in which analytic solutions are known but feasible means of computing these solutions are not available. Nonetheless, analytic simulations are indispensable in many situations, particularly when dealing with complex physical phenomena involving enormous numbers of relatively small and relatively similar entities whose individual interactions are relatively simple and whose aggregate interactions follow the “law of large numbers”; in other words, they permit statistical treatment. In such cases, analytic models often represent at least one form of complete understanding.

Note There is a large class of problems that are not well enough understood to be handled analytically—i.e., for which no formal mathematical solutions exist. These problems are modeled and simulated by means of discrete-event simulations (DES). When we have a system that is composed of several entities, and we understand each entity in isolation and also their pairwise interactions, but fail to comprehend the behavior and relations of the system as a whole, then we can make use of a simulation to encode the pairwise interactions and then run the simulation to try to approximate the relations or behavior of the system as a whole; one of these simulations is known as a discrete-event simulation (DES).

291

Chapter 8

Simulation

Discrete-Event Simulation Time is essential in a DES, and the simulation can be seen as a succession of discrete events in which entities interact. Time advances in a discrete manner by means of fixed ticks or a simulated clock. A DES is often the last alternative for modeling certain kinds of intractable problems. Its power lies in its capacity to expose patterns of interaction for the whole system that cannot be acknowledged in other ways. It’s frequently possible to enumerate and describe a collection of entities and their properties, relations, and immediate interactions without knowing where these interactions lead. If this knowledge is encoded in a DES simulation and the behavior of the resulting model is observed, then we could acquire a better understanding of the system and the interaction among its entities; this is typically the main purpose behind the development of a DES. When developing a DES there are six key elements to consider:

292

•

Objects, which represent elements of the system, have properties, relate to events, consume resources, and enter and leave queues over time. In an airport simulation (soon to be examined), objects would be airplanes. In a health-care system, objects might be patients or organs. In a warehouse system, objects would be products in stock. Objects are supposed to interact with each other or the system and can be created at any time during the simulation.

•

Properties, which are features particular to every object (size, takeoff time, landing time, sex, price, and so on), are stored in some manner and help determine a response to a variety of scenarios that might arise during the simulation; such values can be modified.

Chapter 8

Simulation

•

Events, which are incidents that can occur in the system and are usually related to objects, can be things like the landing of an airplane, the arrival of a product to a warehouse, the appearance of a particular disease, and so forth. Events can occur and reoccur in any order.

•

Resources, which are elements that provide services to objects (for example, a runway at the airport, storage cells in a warehouse, and doctors at a clinic), are finite. When a resource is occupied and an object needs it, the object must queue and wait until the resource is available. We’ll see such a scenario in the practical problem of this chapter.

•

Queues, which are the means by which objects are organized to await the release of some resource that’s currently occupied, can have a maximum capacity and can have different calling approaches: First-In- First-Out (FIFO), Last-In-First-Out (LIFO), or based on some criteria or priority (disease progression, fuel consumption, and the like).

•

Time (as mentioned before and occurs in real life) is essential in simulation. To measure time, a clock is started at the beginning of the simulation and can be used to track particular periods of time (departure or arrival time, transportation time, time spent with certain symptoms, and so on). Such tracking is fundamental because it allows you to know when the next event should occur.

Discreet Events Simulation (DES) are closely related to probability and statistics because they model real-life scenarios where randomized and probabilistic events occur; DES must rely on probabilistic distributions, random variables, and other statistics and probability tools for events generation. 293

Chapter 8

Simulation

P robabilistic Distributions A discrete random variable is one whose set of values is finite or countably infinite; in other words, its values can be listed as a finite or infinite sequence, such as 1, 2, 3 . . . and so on. The probability distribution for a discrete random variable is any graph, table, or formula that assigns a probability to each possible value. The sum of all probabilities must be 1, and each individual probability must be between 0 and 1. For example, when we throw a fair die (all sides equally probable), the discrete random variable X representing the possible outcomes will have the probability distribution X(1) = 1/6, X(2) = 1/6, …, X(6) = 1/6. All sides are equally probable, so the assigned probability for every value of the random variable is 1/6. Parameter μ will indicate the mean (expected value) in their corresponding distributions. The mean represents the value that the random variable takes on average. In other words, it’s the sum E=[(each possible outcome) × (probability of that outcome)], where E denotes the mean. In the case of the die, the mean would be E = 1/6 + 2/6 + 3/6 + 4/6 + 5/6 + 6/6 = 3.5. Notice that the result 3.5 is actually halfway between all possible values the die can take; it’s the expected value when the die is rolled a large number of times. Parameter σ2 will indicate the variance of the distribution. Variance represents the dispersion of possible values of the random variable; it’s always non-negative. Small variances (close to 0) indicate values are close to each other and the mean; high variances (close to 1) indicate great distance among values and from the mean. Poisson is a discrete distribution expressing probabilities concerning the number of events per time unit (Figure8-1). It’s usually applied when the probability of an event is small and the number of opportunities for its occurrence is large. The number of misprints in a book, airplanes arriving at an airport, cars arriving at traffic lights, and deaths per year in a given age group are all examples of applications of the Poisson distribution. 294

Chapter 8

Simulation

Figure 8-1. Poisson distribution An exponential distribution expresses time between events in a Poisson process (Figure8-2). For instance, if you’re dealing with a Poisson process describing the number of airplanes arriving at an airport during a certain time then you may be interested in a random variable that would indicate how much time passed before the first plane arrived. An exponential distribution can serve this purpose, and it could also be applied to physics processes; for example, to represent the lifetime of particles where the λ parameter would indicate the rate at which the particle ages.

Figure 8-2. Exponential distribution 295

Chapter 8

Simulation

The normal distribution describes a probability that converges around a central value, no bias left or right, as shown in Figure8-3. Normal distributions are symmetric and possess bell-shaped density curves with a single peak at the mean. Fifty percent of the distribution lies to the left of the mean and fifty percent to the right. The standard deviation indicates the spread or belt of the bell curve; the smaller the standard deviation the more concentrated the data. Both the mean and the standard deviation must be defined as parameters of the normal distribution. Many natural phenomena strongly follow a normal distribution: blood pressure, people’s height, errors in measurements, and many more.

Figure 8-3. Normal distribution So far we have described what a DES is, its components, and some of the most important probability distributions that can be applied for event time-generation in this type of simulation. In the next section, we will start looking at a practical problem, where we will see how to put all the pieces together in an airport simulation example.

296

Chapter 8

Simulation

Practical Problem: Airport Simulation Let’s imagine a scenario in which we would like to simulate the operation of a five-runway airport where airplanes transporting a certain number of passengers arrive, spend some time at the airport to refuel, and eventually depart in a timeframe that depends, among others, on the probability that the airplane might have gotten broken up. This is the airport simulation that we will be implementing in this chapter. The IDistribution, Poisson, and Continuous classes (interfaces) seen in future code are part of the MathNet.Numerics package. The time between arrival to the airport of one plane and another distributes as a Poisson function with a lambda parameter specified by Table8-1.

Table 8-1. Arrivals of Airplanes at the Airport According to Timeframes Time

Lambda

06:00–14:00

7 mins

14:00–22:00

10 mins

22:00–06:00

20 mins

When an airplane arrives at the airport it lands on an available runway, selecting it uniformly from any of the available runways. If there’s no runway available, the airplane is enqueued into a line of airplanes asking permission to land. Once the airplane finally lands, it processes its cargo in an amount of time that distributes by an exponential function whose parameter gets its value by considering the number of passengers traveling on the plane, as shown in Table8-2.

297

Chapter 8

Simulation

Table 8-2. Time to Process Cargo for Any Airplane and Dependant on Number of Passengers Passengers

Lambda

0–150

50 mins

150–300

60 mins

300–450

75 mins

While an airplane is processing its cargo, it’s considered to be occupying the runway. An airplane can get broken down with a probability of 0.15, in which case the reparation will distribute by an exponential function with parameter lambda = 80 mins. In order to start analyzing the code of our airport simulation, let’s consider the Airplane class as described in Listing 8-1.

Listing 8-1. Airplane Class public class Airplane { public Guid Id { get; set; } public intPassengersCount{ get; set; } public double TimeToTakeOff{ get; set; } public intRunwayOccupied{ get; set; } public bool BrokenDown{ get; set; } public Airplane(int passengers) { Id = Guid.NewGuid(); PassengersCount = passengers; RunwayOccupied = -1; } } 298

Chapter 8

Simulation

The Airplane class contains the following properties: •

Id: It’s initialized in the constructor and will uniquely identify every airplane.

•

PassengersCount: defines the number of passengers in the airplane

•

TimeToTakeOff: defines the time (in minutes) at which the airplane is supposed to take off from the landing strip

•

RunwayOccupied: identifies whether an airplane is occupying a runway at the airport, and, if so, this property matches the index of the runway being occupied. When its value is less than 0 it means the airplane is not occupying any runway.

•

BrokenDown: has value True if the airplane has broken down, False otherwise

In Listing 8-2 we can see the AirportEvent abstract class, which will serve as the parent of the other three classes representing different events taking place in the AirportSimulation. The intention is to shorten the code, compacting all lines that can be logically compacted or included in one single parent class, thus taking advantage of inheritance in C#.

Listing 8-2. AirportEvent Abstract Class public abstract class AirportEvent where T: IComparable { protected double[] Parameters; protected List Frames; public double[] DistributionValues; public List Distributions;

299

Chapter 8

Simulation

protected AirportEvent(params double[] lambdas) { Distributions = new List(); DistributionValues = new double[lambdas.Length]; Frames = new List(); Parameters = lambdas; } public virtual void SetDistributionValues(Distribution Type type) { foreach (var lambda in Parameters) { switch (type) { case DistributionType.Poisson: Distributions.Add(new Poisson(lambda)); break; case DistributionType.Exponential: Distributions.Add(new Exponential(lambda)); break; } } // Sampling distributions for (vari = 0; i= 1 and red points satisfy equation wx + b 0 î Note that if wx + b >= 1 then data point x belongs to class 1; otherwise, x belongs to class -1. As we can see, merely having w, b as the weight vector and bias of the optimal classifying hyperplane will allow us to classify new incoming data. Even though at the moment we have reached a formulation for an optimization problem whose solution would indeed lead us to finding the maximum margin of a classifying hyperplane, this formulation is typically disregarded for one that facilities the computational effort and the optimization itself. This new formulation is based on Lagrange multipliers and the Wolfe dual-problem equivalence. Duality represents a key role in optimization theory, and many optimization problems have an associated optimization problem called the dual. This alternative formulation of the problem possesses a set of solutions that are related to the solutions of the original (known as primal) problem. In particular, for a broad class of problems the primal solutions can be easily calculated from the dual ones. Moreover, in the specific case of the problem we are dealing with in this chapter, the dual formulation provides us with easier-to-handle constraints that are also well suited for kernel functions (we’ll examine them soon). A constrained optimization problem such as ours can be solved by means of the Lagrangian method. This method allows us to find the maximum or minimum of a multi-variable function subject to a set of constraints. It reduces the constrained problem to an unconstrained problem by adding n + k variables, k being the number of constraints of the original problem. These new variables are known as Lagrange multipliers. Using this transformation, the resulting problem will include equations that are easier to solve than the ones in the original problem. 325

Chapter 9

Support Vector Machines

The Lagrangian of a function f(x) having constraints g i ( x ) = 0(i = 1¼m) is the following: m

L ( x , a ) = f ( x ) + åa i g i ( x ) i =1

Notice the new formulation has no constraints; they have been encapsulated in the only function now present, L(x,α). In this case, the αi represents the Lagrangian multipliers. Let’s substitute the objective function and constraints of our primal problem into L(w,b,α): L ( w , b, a ) =

w 2

2

m

- åa i ( y i ( wx i + b ) - 1) i =1

The previous expression uses the generalized Lagrangian form that not only encompasses equality constraints but also inequalities g i ( x ) £ 0 or equivalently -g i ( x ) ³ 0. Once we have introduced the Lagrangian multipliers, we just need to find the dual form of the problem. In particular, we’ll find the Wolfe dual form of the problem. For this purpose, we minimize L with respect to w, b, which is achieved by solving the following equations where Ñ x L ( w , b, a ) denotes the gradient of L with respect to x: Ñ w L ( w , b, a ) = 0 Ñ b L ( w , b, a ) = 0 The derivative of L with respect to w yields the following result: m

Ñ w L ( w , b, a ) = w - åa i y i x i = 0 i =1

326

Chapter 9

Support Vector Machines

This implies the following: m

w = åa i y i x i i =1

As for the gradient with respect to b, the result is as follows: m

Ñ bL ( w , b, a ) = åa i y i = 0 i =1

Substituting the new formula obtained for w and considering that åai y i = 0, we can adjust L(w,b,α) as follows: m

i =1

m

L ( w , b, a ) = åa i i =1

1 m å y i y ja i a j x i x j 2 i , j=1

Notice that since xi,xj are vectors, xixj denotes their inner product. So, finally, we have reached the expression of the dual problem, and in fact the optimization problem that most SVM libraries and packages solve because of the advantages previously mentioned. The complete optimization problem would be as follows: m

max L ( w , b, a ) = åa i a

i =1

m

s.t

åa y i =1

i

i

1 m å y i y ja i a j x i x j 2 i , j=1 =0

a i ³ 0 i = 1, ¼m In the next section, we’ll see a practical problem where the previous problem (dual) will be solved using an optimization library in C#. Such a problem will help us understand some of the concepts and ideas that have been introduced in this chapter.

327

Chapter 9

Support Vector Machines

Note The gradient of a function f is usually denoted by the symbol Ñ preceding the function name ( Ñf ). It’s a vector formed by the derivatives of f with respect to every variable and indicates the direction of the maximum increment of f at a given point. For instance, assuming f is the function that maps every point in space with a given pressure, then the gradient will indicate the direction in which pressure will change more quickly from any point (x, y, z).

Practical Problem: Linear SVM inC# To develop our Linear SVM, we will create a class named LinearSvmClassifier that has the following fields or properties (Listing 9-1).

Listing 9-1. Properties and Fields of Our Linear SVM public class LinearSvmClassifier { public ListTrainingSamples{ get; set; } public double[] Weights; public double Bias; public ListSetA{ get; set; } public ListSetB{ get; set; } public List Hyperplane { get; set; } private readonlydouble[] _alphas; public intModelToUse = 1;

328

Chapter 9

Support Vector Machines

public LinearSvmClassifier(IEnumerabletraining Samples) { TrainingSamples = new List(trainingSamples); Weights = new double[TrainingSamples.First(). Features.Length]; SetA = new List(); SetB = new List(); Hyperplane = new List(); _alphas = new double[TrainingSamples.Count]; } } public class TrainingSample { public int Classification { get; set; } public double[] Features { get; set; } public TrainingSample(double [] features, int classification) { Features = new double[features.Length]; Array.Copy(features, Features, features.Length); Classification = classification; } } Each property or field is described as follows: •

TrainingSamples: list of TrainingSample objects; each object represents a data point accompanied by its classification. The TrainingSample class illustrated in Listing 9-1 merely consists of a Features array of doubles and an integer Classification. 329

Chapter 9

Support Vector Machines

•

Weights: double array representing the weights in an SVM model

•

Bias: double value representing the bias or intercept in an SVM model

•

SetA: list of Tuple representing points in the training data that satisfy wx + b >= 1. It’s only used in the prediction stage.

•

SetB: list of Tuple representing points in the training data that satisfy wx + b 0) . Hence, the new value for x, let it be x', will be shifted to the left, and the new p2 = f(x') will satisfy that p2 < p1. This procedure will continue until we reach the minimum, assuming α is small enough and will take smaller steps as it approaches the minimum; in other words, x will be slowly shifted to the left on new iterations. Going back to the general case, and in order to find the steepest decrease of the error, we express E (the sum of all errors upon classifying each training data) in terms of w (weight vector). Notice that setting E in terms of w is always possible because y ¢x = w i x i for any given training data x; therefore, the function to minimize will be the following: n

E(w ) =

å( y i =1

i

- y ¢i )

2

2

429

Chapter 11

Neural Networks

Hence, we will find the gradient of E(w)—let it be ÑE ( w ) —and we will consider it in Adaline’s learning rule, which would be the following: w = w - a * ÑE ( w ) Notice the sign on the rule is a minus and not a plus. That’s because we must negate the gradient, -ÑE ( w ) , in order to minimize E(w). As it was previously defined in the Perceptron, α is the learning rate that controls how fast we move toward a solution. The previous formula relates to the way we update the weight vector w, but how should we update a single weight? The rule for a single weight would be this: wi = wi - a *

¶E ¶w i

We substituted the gradient with its equivalent, the partial derivatives with respect to every weight wi. After developing the term ¶E by ¶w i calculating some derivatives and applying the chain rule, we will finally have the complete learning rule for GDS: n ¶E = å ( y j - y ¢j ) * ( - x ij ) ¶w i j=1

As before, yj represents the correct classification of training data j, yj′ represents the classification outputted by the NN, and xij represents the ith input value of training data j—the input of training data j associated with weight wi. Even though GDS is, from a theoretical or mathematical perspective, an elegant method for finding a local minimum of a function, in practice it tends to be quite slow. Notice that to update a single weight you would need to go over the entire training data set, which could contain tens of 430

Chapter 11

Neural Networks

thousands of training examples, so that would imply a lot of computations. Thus, for this practical reason, we typically use an approximated variant of GDS as the learning rule of Adaline; this variant is presented in the next section.

Stochastic Approximation Stochastic gradient descent (SGD) or incremental gradient descent is an approximation procedure supplemental to GDS where weights are updated incrementally after the calculation of the error of each training data. Thus, it saves us from the computational trouble of having to loop over the entire training data set to compute the value of every weight. This is, in practice, the method used in Adaline and in other NN algorithms (backpropagation) that minimize the squared error by considering the correct classification of a training data and its output in the NN.The learning rule that uses stochastic approximation is known as the Delta Rule, the Adaline Rule, or the Widrow-Hoff Rule (after its creators). In Figure11-9 we can see a very intuitive idea of the differences between GDS and SGD.In the first we move directly to the minimum of the error surface so we follow a straight path, while in the latter we move like a drunk person would; sometimes we lose balance and move to incorrect positions, but eventually we end up at the same point as GDS.

Figure 11-9. To the left the direct path that GDS would follow over the error surface to get to a minimum; to the right the “unbalanced” path followed by SGD 431

Chapter 11

Neural Networks

The update rule using SGD would be as follows: w i = w i + a * ( y i - y ¢i ) * x i Notice the similarity between this learning rule and the one described before for the Perceptron—it looks very similar. What’s the main difference? The main difference is in the output of the NN while training. In Adaline we do not consider any threshold or activation function; therefore, y ¢i = w i x i .

Note When you combine several Adalines in a multi-layer network you obtain what is known as a Madaline.

Practical Problem: Implementing Adaline NN After examining the theory behind Adaline’s algorithm, it’s time to finally implement the procedure in C#. For this purpose, we will create the class Adaline, shown in Listing 11-4.

Listing 11-4. Adaline Class public class Adaline :SingleNeuralNetwork { public Adaline(IEnumerabletraining Samples, int inputs, double learningRate) : base(trainingSamples, inputs, learningRate) { } public override void Training() { double error;

432

Chapter 11

Neural Networks

do { error = 0.0; foreach (vartrainingSample in TrainingSamples) { var output = LinearFunction(trainingSample.Features); varerrorT = Math.Pow(trainingSample.Classification- output, 2); if (Math.Abs(errorT) < 0.001) continue; for (var j = 0; j < Inputs; j++) Weights[j] +=LearningRate * (trainingSample.Classification- output) * trainingSample.Features[j]; error = Math.Max(error, Math.Abs(errorT)); } } while (error > 0.25); } public double LinearFunction(double [] values) { var summation = (from i in Enumerable.Range(0, Weights.Count) select Weights[i]*values[i]).Sum(); return summation; } public override double Predict(double[] features) { return LinearFunction(features) >0.5 ?1 : 0; } } 433

Chapter 11

Neural Networks

This class inherits from SingleNeuralNetwork and contains three methods. The second method is LinearFunction(), which simply computes the weighted sum wixi. Remember, there’s a difference between the prediction stage and the training stage in an Adaline. In the training or learning phase we compute the output of the NN as a weighted sum, but in the prediction phase we must use a categorical function to classify new incoming data; therefore, the prediction function is different from the learning function. In this case, our prediction function computes the weighted sum of the new data and outputs either 1 or 0 depending on whether the result of the weighted sum outputted a value greater than 0.5 or less than it. The Training() method consists of a do ... while() statement where we verify if the maximum error carried out when classifying any training data exceeds 0.25. If it does, the loop will continue; otherwise, we will consider ourselves as being satisfied, and the method will end. Furthermore, we will not alter the weights if the error when classifying a training data is below 0.001. In Figure11-10 we can see the result obtained after executing our Adaline on a small set of data.

Figure 11-10. Result obtained after executing our Adaline on a small data set

434

Chapter 11

Neural Networks

If we are curious about the functioning of the algorithm, we could set a breakpoint on the line while (error > 0.25); and then see how the maximum error diminishes after each iteration. The following values were the ones obtained on a series of iterations when we executed Adaline on the same training data set used in the Perceptron implementation: 3.2386, 1.7957, 1.0569, 0.6973, 0.5822, 0.5050, 0.4523, 0.4144, 0.3861, 0.3640, 0.3463, 0.3315, 0.3189, 0.3078, 0.2980, 0.2891, 0.2810, 0.2735, 0.2676, 0.2614, 0.2552, and 0.2491.

M ulti-layer Networks A multi-layer network is a type of NN in which we have multiple NNs grouped in layers and connected from one layer to the other. The NNs we have described so far (Perceptron, Adaline) were constituted by two layers: an input layer of multiple nodes and an output layer of a single node. The multi-layer NN shown in Figure11-11 is composed of three layers: input, hidden, and output. It’s also a feed-forward NN; in other words, all signals go from nodes in one layer to nodes in the next layer. Thus, a multi-layer NN is constructed by putting together many of our simple “neurons” arranged into layers and having the output of a neuron as the input of another neuron in the next layer.

435

Chapter 11

Neural Networks

Figure 11-11. Multi-layer, feed-forward, fully connected NN consisting of three layers: one for input units, one for hidden units (gray) and one for output units (green). Sometimes the input layer is not considered as a layer. Except for the input layer, which receives its inputs from the components (xi) of the training data, all other layers receive their inputs from the activation function of the previous layer. Every edge in a multi- layer NN represents a weight, and any edge leaving a node has its weight value multiplied by the activation function value of the node from which it originates. Thus, any node from layer L, where L > 0 (not the input layer), will have its input or activation value computed as follows: æ n ö A l ,i = g ç å w l -1, j,i A l -1, j ÷ è j=1 ø

436

Chapter 11

Neural Networks

where n is the total number of units in layer L- 1, Al,i indicates the activation value of unit i at layer L, w l -1, j,i is the weight or edge going from unit j of layer L– 1 to unit i of layer L, and g is the activation function applied in the NN.Typically, g is chosen as the technically logical sigmoid function whose values range in the interval [0, 1], and it is computed as follows: sigmoid ( x ) =

1 1 + e- x

A very important property of the sigmoid function is that it’s differentiable and continuous; remember that this property is significant to us because we will be calculating gradients and consequently derivatives. One key element with multi-layer NNs is that they are capable of classifying non-linearly separable data sets. As a result, functions like XOR (Figure11-12) that cannot be classified by linear NNs such as the Perceptron can be correctly classified by a simple multi-layer NN containing just one hidden layer.

Figure 11-12. XOR function; there’s no line that would divide the red points from the green points

437

Chapter 11

Neural Networks

We could think of multi-layer NNs as powerful mathematical functions able to approximate any tabular function we may have on the training data set. Each hidden layer represents a function, and the combination of layers can be seen as the composition of functions in mathematics. Thus, having n hidden layers could be seen as having the mathematical function o(h1, h2 ( … hn(i(x)) … )) where o is the output layer, i the input layer, and hi the hidden layers. Traditional NNs have a single hidden layer, and when they have more than one layer we are dealing with deep neural networks and deep learning. Table11-1 illustrates the relationship between the number of hidden layers and the capacity of the resulting NN.

Table 11-1. Relationship Between Number of Layers and Power of NNs Number Hidden Layers

Result

None

Only capable of representing linear separable functions or decisions

1

Can approximate any function that contains a continuous mapping from one finite space to another

2

Can represent an arbitrary decision boundary to arbitrary accuracy with rational activation functions and can approximate any smooth mapping to any accuracy

>2

Additional layers can learn complex representations (sort of automatic feature engineering).

438

Chapter 11

Neural Networks

It has been proven that a multi-layer NN with a single hidden layer is capable of learning any function. Hence, one may ask the question, if with a single hidden layer we can learn any function, then why do we need deep learning? The idea is that while the universal approximation theorem proves that, indeed, having a single hidden layer is enough for learning any continuous function, it does not state how easy it would be to complete this learning. Thus, for efficiency and accuracy reasons, we may need to add complexity to our NN architecture and include additional hidden layers in order to get a decent solution in a decent time. The number of neurons in hidden layers is another important issue to consider when deciding on our NN architecture. Even though these layers do not directly interact with the external environment, they do have a remarkable influence on the final output. Both the number of hidden layers and the number of neurons in hidden layers must be carefully thought out. Using too few neurons in the hidden layers will result in something called underfitting. Underfitting occurs when there are too few neurons in the hidden layers to effectively perceive signals in a complicated data set. Using too many neurons in hidden layers can result in several problems, the best known of which is overfitting, or when the weights adjust too well to the training data set and as a result the NN is unable to correctly predict new incoming data.

Note The universal approximation theorem states that a feed-forward network with a single hidden layer containing a finite number of neurons can approximate any continuous function; this allows NNs to be considered as universal approximators.

439

Chapter 11

Neural Networks

Backpropagation Algorithm As occurs in Adaline NNs, multi-layer NNs using backpropagation typically rely on the gradient descent method, and more specifically on the stochastic gradient approximation method, for adjusting the weights of the NN.They also seek to achieve the same goal as Adaline’s algorithm— minimizing the error in the quadratic difference between the true classification of the data and the network output. The idea with the backpropagation algorithm is that it serves as a mechanism for transporting the error taking place at the output layer to the final hidden layer (adjusting weights on the way), and from there to the previous hidden layer, and so on, backward; in other words, if o is the output layer and h1,h2,…,hn denote the hidden layers, then the backpropagation algorithm carries on the error from the output layer (equivalent to having the weights adjusted or the error minimized), from o to hn, then from hn to h n-1 , and so on until the error adjustment process reaches h1. This functioning justifies the name backpropagation, because the output is computed from the input layer passing through layers h1,h2,…,hn and ending in the output layer, and then, once an output has been obtained, the weights are adjusted backward from output to the first hidden layer. As mentioned before, the backpropagation algorithm relies on the gradient descent method, as does the Adaline method. The first difference we can call out between these two procedures is that with Adaline we only had one output node, but in multi-layer NNs and therefore in backpropagation we could be dealing with multiple output nodes arranged in an output layer; thus, the total error must be calculated as follows: n

E(w ) =

440

k

åå ( y i =1 j=1

ij

2

- y ¢ij )

2

Chapter 11

Neural Networks

where n is the cardinality of the training data set, k the number of units in the output layer, yij the correct classification of training data i at node and position j from the output layer, and y'ij the classification outputted for training data i at node j in the output layer of our NN. The learning rule for every node in a backpropagation procedure resembles that of the Perceptron and Adaline. The rule, according to a stochastic approximation, is as follows: w ij = w ij + a * d j * x ij In this case, wij indicates a weight going from node i into node j, α is the learning rate, xij is the activation value going from node i into node j (in the input layer these values coincide with the input values), and δj is the error at node j. Learning rules previously described did not have two subindices (wij) as they do now in the weight update rule of the backpropagation algorithm. Let’s recall that backpropagation is intended to work on multi- layer NNs; therefore, we will have many nodes connected to other nodes so each edge ij has an associated wij. So, we initially have every variable in the weight update formula except for δj; this term represents the error on classification and is the one we need to derivate with respect to the weights to find the gradient, and as a result the steepest descent with respect to w in the error surface. As stochastic approximation does, we iterate through every training data one at a time, which justifies that dj = -

¶E d ¶ åw ij * x ij i

441

Chapter 11

Neural Networks

where Ed is the error associated with classifying training data d and wij is the weights associated with unit j. We know the formula for the global error E(w), but that’s not the formula we derivate to minimize w. Remember that stochastic approximation works on one training data at a time; therefore, we derivate the following equation: k

Ed =

å( y j=1

j

- y ¢j )

2

2

In this case, k is the total number of nodes in the output layer, yj is the correct classification for node j, and y'j is the value outputted by our NN.Applying the chain rule and considering the case where the node on which we calculate the error term is either an output or a hidden unit, we can reach the next formulas: •

For nodes in the output layer, dj = -

¶E d = ( y j - y ¢j ) * y ¢j * (1 - y ¢j ) ¶ åw ij * x ij i

•

This implies, w ij = w ij + a * ( y j - y ¢j ) * y ¢j * (1 - y ¢j ) * x ij

•

For nodes in the hidden layers, dj = -

¶E d = y ¢j * (1 - y ¢j ) * å dk * w kj ¶ åw ij * x ij kÎStream i

442

Chapter 11

•

Neural Networks

Stream, in this case, is the set of nodes whose inputs correspond to the output of node j. The previous formula implies that w ij = w ij + a * y ¢j * (1 - y ¢j ) *

å

dk * w kj * x ij

kÎStream

Note that the weight-update formulas obtained assume we have sigmoid units; in other words, that we are using the sigmoid function as an activation function in every node of the NN.The general form of the weight-update rule for output and hidden layers respectively would be as follows: w ij = w ij + a * ( y j - y ¢j ) * G ( y ¢j ) * x ij w ij = w ij + a * G ( y ¢j ) *

å

dk * w kj * x ij

kÎStream

where G(yj′) represents the derivative of the activation function evaluated at the value outputted by the activation, as we know this value can be expressed in terms of w. Recall that the sigmoid function’s derivative is F(x) * (1- F(x)); this is very easy to compute and work with and is one of the main reasons the sigmoid function is the classical activation function for multi-layer neural networks. Figure11-13 illustrates another popular activation function, the hyperbolic tangent, a symmetrical function whose output is in the range [-1;1] and that is denoted and calculated as follows: tanh ( x ) =

sinh ( x )

cosh ( x )

=

ex - e- x ex + e- x

443

Chapter 11

Neural Networks

Figure 11-13. Hyperbolic tangent function, which outputs values in the range (-1;1) Nowadays, a popular activation function that is replacing the sigmoid function and other similar smooth functions is the rectified linear unit, or ReLU (Figure11-14). Unlike sigmoid and the smooth functions, ReLU doesn’t have the shortcoming of the vanishing gradient issues seen in deep learning, such as when training a NN of more than three layers. Its equation is extremely simple: ReLU ( x ) = max ( 0 , x ) In other words, ReLUs let all positive values pass through unchanged, but just set any negative values to 0. Although newer activation functions are gaining traction, most deep neural networks these days use ReLU or one of its closely related variants.

444

Chapter 11

Neural Networks

Figure 11-14. ReLU function To comprehend a little bit better the flow backward in the backpropagation algorithm and the nodes or edges in which our variables will reside, let’s examine Figure11-15.

Figure 11-15. Flow backward in the backpropagation algorithm. Weight wij is updated by considering the error term residing in node j. Now that we have a theoretical background on the functioning of the backpropagation algorithm, in the next section we will implement a MultiLayerNetwork class representing multi-layer NNs, and we will develop our backpropagation algorithm as a method of that class.

445

Chapter 11

Neural Networks

ractical Problem: Implementing P Backpropagation & Solving theXOR Problem To properly encode the multi-layer NN paradigm, we will create the class shown in Listing 11-5. We’ll also apply an object-oriented approach to include a Layer class for representing all nodes arranged as a list of sigmoid units.

Listing 11-5. MultiLayerNetwork and Layer Classes public class MultiLayerNetwork { public List Layers { get; set; } public ListTrainingSamples{ get; set; } public intHiddenUnits{ get; set; } public intOutputUnits{ get; set; } public double LearningRate{ get; set; } private double _maxError; public MultiLayerNetwork(IEnumerabletra iningSamples, int inputs, inthiddenUnits, int outputs, double learningRate) { Layers = new List(); TrainingSamples = new List(trainingSamples); LearningRate = learningRate; HiddenUnits = hiddenUnits; OutputUnits = outputs; CreateLayers(inputs); }

446

Chapter 11

Neural Networks

private void CreateLayers(int inputs) { Layers.Add(new Layer(HiddenUnits, TrainingSamples, LearningRate, inputs, TypeofLayer.Hidden)); Layers.Add(new Layer(OutputUnits, TrainingSamples, LearningRate, HiddenUnits, TypeofLayer.OutPut)); } public ListPredictSet(IEnumerable objects) { var result = new List(); foreach (varobj in objects) result.Add(Predict(obj)); return result; } public Layer OutPutLayer { get { returnLayers.Last(); } } public Layer HiddenLayer { get { returnLayers.First(); } } } public class Layer { public List Units { get; set; } public TypeofLayer Type { get; set; }

447

Chapter 11

Neural Networks

public Layer(int number, Listtr ainingSamples, double learningRate, int inputs, TypeofLayertypeofLayer) { Units = new List(); Type = typeofLayer; for (vari = 0; i< number; i++) Units.Add(new SigmoidUnit(trainingSamples, inputs, learningRate)); } } public enumTypeofLayer { Hidden, OutPut } The Layer class contains two properties, a List of SigmoidUnit (we will soon examine this class) and a TypeofLayer Type that is an enum with two possible values: Hidden and OutPut. In the class constructor we simply add as many nodes to the layer as the number argument specifies. In the MultiLayerNetwork class we include properties to obtain the HiddenLayer or, if there’s more than one, the first hidden layer and the OutputLayer. The constructor of the MultiLayerNetwork class receives as arguments the training data set, the number of inputs, hidden nodes, and outputs; and the learning rate. It creates the set of layers by calling the CreateLayers() method. Finally, the PredictSet() method predicts or classifies a set of data received as an argument. The class also includes some properties or fields, most of which are self- descriptive. The _maxError field will be used to indicate the maximum error when classifying any training data in an iteration or epoch of the backpropagation algorithm.

448

Chapter 11

Neural Networks

Note An iteration in a NN’s learning algorithm is typically known as an epoch. The SigmoidUnit class inherits from SingleNeuralNetwork, and its code is very simple (Listing 11-6). It merely overrides the Predict() method to compute the value of the sigmoid function with the features of the input data and the weight vector.

Listing 11-6. SigmoidUnit Class, Which Inherits from the SingleNeuralNetwork Abstract Class public class SigmoidUnit :SingleNeuralNetwork { public double ActivationValue{ get; set; } public double ErrorTerm{ get; set; } public SigmoidUnit(IEnumerabletraining Samples, int inputs, double learningRate) : base(trainingSamples, inputs, learningRate) { } public override double Predict(double [] features) { var result = 0.0; for (vari = 0; i .001) { foreach (vartrainingSample in TrainingSamples) { Predict(trainingSample.Features); // Error term for output layer ... for (vari = 0; iMath.Abs(u.ErrorTerm)); } } In order to make our method as flexible as possible and interact easily with different activation functions, we coded the FunctionDerivative() method (Listing 11-8), which receives an activation value and a type of function (encoded as an enum) and outputs the derivative of the activation function evaluated at the activation value.

451

Chapter 11

Neural Networks

Listing 11-8. FunctionDerivative() Method and Enum Declaration with Activation Functions Previously Mentioned private double FunctionDerivative(double v, TypeFunctionfunction) { switch (function) { case TypeFunction.Sigmoid: return v*(1- v); case TypeFunction.Tanh: return 1- Math.Pow(v, 2); case TypeFunction.ReLu: return Math.Max(0, v); default: return 0; } } public enumTypeFunction { Sigmoid, Tanh, ReLu } By combining the previous method with the following (Listing 11-9) sibling classes of the SigmoidUnit class (shown in Listing 11-6), we can effortlessly change our model from one type of unit (Sigmoid, Tanh, ReLU) to the other and experiment with different types of activation functions.

452

Chapter 11

Neural Networks

Listing 11-9. Hyperbolic Tangent and ReLU Units public class TanhUnit :SingleNeuralNetwork { public double ActivationValue{ get; set; } public double ErrorTerm{ get; set; } public TanhUnit(IEnumerabletraining Samples, int inputs, double learningRate) : base(trainingSamples, inputs, learningRate) { } public override double Predict(double [] features) { var result = 0.0; for (vari = 0; iunit. ActivationValue == max); } In order to test our multi-layer NN, we will see how it correctly classifies data from the XOR problem by having a NN structure composed of a hidden layer of three nodes and an output layer of a single node. We will also add a little modification to our TrainingSample class to contemplate the case where a training data may have a classification vector instead of a single value. A classification vector could be binary; for instance, (1, 0, 0) could indicate that the training data with which it associates is to be classified as red and not green or blue. Both the new TrainingSample class and the setting for testing a multi- layer NN on the XOR problem are illustrated in Listing 11-12.

456

Chapter 11

Neural Networks

Listing 11-12. Slight Modification to TrainingSample Class and Setting Up for Testing Our MultiLayerNetwork Class for the XOR Problem public class TrainingSample { public int Classification { get; set; } public List Classifications { get; set; } public double[] Features { get; set; } public TrainingSample(double [] features, int classification, IEnumerableclasifications = null ) { Features = new double[features.Length]; Array.Copy(features, Features, features.Length); Classification = classification; if (clasifications != null) Classifications = new List(clasifications); } } vartrainingSamplesXor = new List { new TrainingSample (new double[] {0, 0}, -1, new List { 0 } ), new TrainingSample (new double[] {1, 1}, -1, new List { 0 } ),

457

Chapter 11

Neural Networks

new TrainingSample (new double[] {0, 1}, -1, new List { 1 } ), new TrainingSample (new double[] {1, 0}, -1, new List { 1 } ), }; var multilayer = new MultiLayerNetwork(trainingSamplesXor, 2, 3, 1, 0.01); vartoPredict = new List { new double[] {1, 1}, new double[] {1, 0}, new double[] {0, 0}, new double[] {0, 1}, new double[] {2, 0}, new[] {2.5, 2}, new[] {0.5, 1.5}, }; var predictions = multilayer.PredictSet(toPredict); for (vari = 0; i c).ToArray(); trainingDataSet.Add(new TrainingSample(imgVector, @class, classVec)); } 472

Chapter 12

Handwritten Digit Recognition

_handwrittenDigitRecogNn = new HandwrittenDigit RecognitionNn(trainingDataSet, NnInputs, NnHidden, NnOutputs, 0.002); _handwrittenDigitRecogNn.Training(); var fileWeights = new StreamWriter("weights.txt", false); foreach (var layer in _handwrittenDigitRecogNn.Layers) { foreach (var unit in layer.Units) { foreach (var w in unit.Weights) fileWeights.WriteLine(w); fileWeights.WriteLine("*"); } fileWeights.WriteLine("-"); } fileWeights.Close(); MessageBox.Show("Training Complete!", "Message"); } To classify the digit drawn on the picture box, we add the Classify button. The method triggered when the click event occurs is illustrated in Listing 12-6. In this method, we check for the existence of the weights.txt file, load the set of weights if the file exists, or output a warning message in any other case. If the weights have not been loaded then we run the ReadWeights() method and eventually execute the Predict() method of the NN and save the resulting classification in the classBox textbox.

473

Chapter 12

Handwritten Digit Recognition

Listing 12-6. Method Executed After the Classify Button Has Been Clicked private void ClassifyBtnClick(object sender, EventArgs e) { if (Directory.GetFiles(Directory. GetCurrentDirectory()).Any(file => file == Directory.GetCurrentDirectory() + "weights.txt")) { MessageBox.Show("Warning", "No weights file, you need to train your NN first"); return; } if (!_weightsLoaded) { ReadWeights(); _weightsLoaded = true; } var digitMatrix = GetImage(_bitmap); var prediction = _handwrittenDigitRecogNn. Predict(digitMatrix.Cast(). Select(c => c).ToArray()); classBox.Text = (prediction + 1).ToString(); } The ReadWeights() method, acting as an auxiliary mini-parser, is in charge of reading the file of weights and assigning them to every node in the NN (Listing 12-7). Weights are stored one per line in the file, and weights belonging to different units will be separated by a line containing a “*” symbol, which marks the end of the weights assignment to a given unit and the start of another one. The same thing occurs with the “-” symbol but at the layer level.

474

Chapter 12

Handwritten Digit Recognition

Listing 12-7. ReadWeights() Method private void ReadWeights() { _handwrittenDigitRecogNn = new HandwrittenDigitRe cognitionNn(new List(), NnInputs, NnHidden, NnOutputs, 0.002); var weightsFile = new StreamReader("weights.txt"); var currentLayer = _handwrittenDigitRecogNn. HiddenLayer; var weights = new List(); var j = 0; while (!weightsFile.EndOfStream) { var currentLine = weightsFile.ReadLine(); // End of weights for current unit. if (currentLine == "*") { currentLayer.Units[j].Weights = new List(weights); j++; weights.Clear(); continue; } // End of layer. if (currentLine == "-") { currentLayer = _handwrittenDigitRecogNn. OutPutLayer; j = 0;

475

Chapter 12

Handwritten Digit Recognition

weights.Clear(); continue; } weights.Add(double.Parse(currentLine)); } weightsFile.Close(); } Finally, let’s execute and take a look at our Handwritten Digit Recognition visual application (Figure12-4).

Figure 12-4. HDR visual application Now that we have completely developed the application, let’s see how it would perform after different drawings of digits 1, 2, and 3 are presented to the NN.

T esting Going back to Figure12-4, we can see the drawing space in our application is the picture box control with a black background; it is in this picture box that we will draw different digits to eventually obtain a classification by clicking the Classify button. Let’s examine some tests (Figure12-5).

476

Chapter 12

Handwritten Digit Recognition

Figure 12-5. Classification of handwritten digits In the same way as we can obtain a correct classification for many handwritten digits in this application, it could happen that for others we get an incorrect classification. The reason behind this inaccuracy, as the reader may expect at this point, is the very small training data set used while training the NN.To obtain higher accuracy we would need many more samples with different styles of handwriting.

477

Chapter 12

Handwritten Digit Recognition

Summary In this chapter we introduced the problem of handwritten digit recognition and developed a Windows Forms application that allows users to draw digits in it and eventually obtain a classification for the drawn digit. We considered only the set of digits {1, 2, 3} but the application can be easily extended to include all possible digits simply by adding new nodes to the output layer. We tested the results and, as mentioned before, due to the small number of training samples the application will probably misclassify some of the incoming data. Thus, adding new training data was a recommendation. The visual application presented in this chapter is an authentic representative of the power and possibilities of neural networks.

478

CHAPTER 13

Clustering & Multi-objective Clustering Thus far, we have discussed several methods related to supervised learning. In these methods, we approximated a function from a training data set containing labeled data. In this chapter, we will begin addressing unsupervised learning, a paradigm of machine learning where we deduce a function and the structure of data from an unlabeled data set. Unsupervised learning (UL) methods no longer have a “training” data set. Consequently, the training phase in UL disappears because data does not have an associated classification; the correct classification is considered unknown. Therefore, UL is far more subjective than supervised learning is, since there is no simple goal for the analysis such as prediction of a response. The general goal of UL methods, as imprecise as it may sound, is to find patterns that describe the structure of the data being analyzed. Because obtaining unlabeled data from a lab instrument or any measurement device is usually easier than obtaining labeled data, UL methods are being applied more and more to multiple problems that require learning the structure of data.

© Arnaldo Pérez Castaño 2018 A. Pérez Castaño, Practical Artificial Intelligence, https://doi.org/10.1007/978-1-4842-3357-3_13

479

Chapter 13

Clustering & Multi-objective Clustering

In this chapter, we will explore one of the most important learning tasks associated with UL, which is clustering, as well as a variant of it where we consider several objective functions to be minimized or maximized at the same time, which is called multi-objective clustering. A method of the broad family of clustering algorithms will be described and implemented throughout the chapter; namely, we will implement the k-means method. Moreover, some measures for determining object and cluster similarity will be also implemented.

Note Both supervised and unsupervised learning algorithms represent techniques of knowledge extraction frequently used in data-mining applications.

What Is Clustering? Clustering is a broad family of algorithms whose purpose is to partition a set of objects into groups or clusters, trying to ensure that objects in the same group have the highest similarity possible and objects in different groups have the highest dissimilarity possible. Similarity in this case is related to a property of the objects; it could be height, weight, color, bravery, or any other quality that our data set includes, typically in a numeric form. Figure13-1 illustrates clustering based on object color.

480

Chapter 13

Clustering & Multi-objective Clustering

Figure 13-1. Clustering a set of objects based on their color Clustering finds applications in various areas of science and business, such as astronomy, psychology, medicine, economics, fraud avoidance, architecture, demographic analysis, image segmentation, and more. A clustering algorithm is usually composed of three elements: •

Similarity Measure: function used to determine the similarity between two objects. In the example from Figure13-1, a similarity function could be Color(x, y) outputting an integer that determines the equivalence between objects x and y in regard to their colors. Typically, the larger the value outputted the greater the dissimilarity is between x and y; the smaller the value outputted the more similar x and y will be.

•

Criterion or Objective Function: function used to evaluate the quality of a clustering

•

Optimization or Clustering Algorithm: an algorithm that minimizes or maximizes the criterion function

481

Chapter 13

Clustering & Multi-objective Clustering

Some of the most popular similarity measures are the following: •

Euclidean Distance of n-dimensional vectors a, b: Euclidean ( a , b ) =

n

å(a i =1

i

- bi )

2

This is the ordinary distance between two points in space. •

Manhattan Distance of n-dimensional vectors a, b: n

Manhattan ( a , b ) = å a i - bi i =1

This is an approximation of the Euclidean Distance, and it’s cheaper to compute. •

Minkowski Distance of cells that belong to an n x m matrix T; p is a positive integer and is a generalization of the previously detailed distances: m

Minkowski p ( Tk , Th ) = p å ( Tki - Thi )

2

i =1

Among the criterion or objective functions used for determining or evaluating the quality of a clustering we can find the following: •

482

Intra-class Distance, also known as Compactness: as its name suggests with its “intra” (on the inside, within) prefix, it measures how close data points in a cluster (group) are to the cluster centroid. The cluster centroid is the average vector of all data points in a cluster. The Sum of Squared Errors is typically used as the mathematical function to measure this distance.

Chapter 13

•

Clustering & Multi-objective Clustering

Inter-class Distance, also known as Isolation or Separation: as its name suggests from the “inter” (between) prefix, it measures how far clusters are from each other.

The family of clustering algorithms can be divided into hierarchical, partitional, and Bayesian algorithms. Figure13-2 illustrates the relation between the different families of clustering algorithms.

Figure 13-2. Clustering algorithms family In this book, we will discuss hierarchical and partitional algorithms; Bayesian clustering algorithms try to generate a posteriori distribution over the set of all possible partitions of the data points. This family of algorithms is highly related to areas such as probability and statistics, so it will be left to the reader as supplementary research.

Note Clustering is a well-known NP-hard problem, meaning no polynomial time solution for the problem can be developed or designed on a deterministic machine. 483

Chapter 13

Clustering & Multi-objective Clustering

Hierarchical Clustering Hierarchical algorithms discover new clusters from previously discovered clusters; hence, new clusters become descendants of parent clusters after being nested within them, and the hierarchical relation is established that way. Hierarchical algorithms can be classified as agglomerative or divisive. An agglomerative (a.k.a bottom-up) hierarchical algorithm starts with each object as a separate cluster of size 1 and then begins merging the most similar clusters into consecutively larger clusters up to the point where it contains a single cluster with all objects in it. A divisive (a.k.a top down) hierarchical algorithm starts with the whole set in one cluster and in every step chooses a group to divide from the current set of clusters. It stops when each object is a separate cluster. Hierarchical algorithms can output a dendrogram, a binary tree–like diagram used to illustrate the arrangement of clusters. In a dendrogram, every level represents a different clustering. Figure13-3 shows an example of an agglomerative clustering being executed over a data set formed by points a, b, c, d, and e, along with the resulting clusters and the dendrogram obtained.

Figure 13-3. Agglomerative clustering example

484

Chapter 13

Clustering & Multi-objective Clustering

Because points a, b and c, d respectively are the nearest ones, they are clustered together. Afterward, clusters {a, b}, {c, d}, being the nearest ones, are grouped together and {e} is left as another cluster. Finally, all data points are merged into a cluster that contains all data points; in this case, we executed a bottom-up procedure. How do we determine clusters’ similarity or distance? The previously detailed measures or distances give us the similarity between two data points, but what about cluster similarity? The measures described in the next lines output the similarity between clusters: •

Average Linkage Clustering: determines the similarity between clusters C1 and C2 by finding the similarity or distance between all pairs (x, y) where x belongs to C1 and y to C2. These values are added and eventually divided by the total number of objects in both C1 and C2. Thus, ultimately, what we calculate is an average or mean of the distance between C1 and C2.

•

Centroid Linkage Clustering: determines the similarity between clusters C1 and C2 by finding the similarity or distance between any pair (x, y) where x is the centroid of C1 and y the centroid of C2.

•

Complete Linkage Clustering: determines the similarity between clusters C1 and C2 by outputting the maximum similarity or distance between any pair (x, y) where x is an object from C1 and y is an object from C2.

•

Single-Linkage Clustering: determines the similarity between clusters C1 and C2 by outputting the minimum similarity or distance between any pair (x, y) where x is an object from C1 and y is an object from C2.

485

Chapter 13

Clustering & Multi-objective Clustering

The pseudocode of an agglomerative hierarchical clustering demonstrates how easy it is in principle to implement this type of algorithm: AgglomerativeClustering (dataPoints) { Initialize each data point in dataPoints as a single cluster while (numberClusters> 1) find nearest clusters C1, C2 according to a cluster similarity measure merge(C1, C2) end } The agglomerative algorithm represents a more efficient approach than that of the divisive algorithm, but the latter often provides a more accurate solution. Notice that the divisive algorithm begins operating with the whole data set; thus, it’s able to find the best division or partition into two clusters at the original data set, and from that point on it’s able to find the best possible division within each cluster. The agglomerative method, on the other hand, at the moment of merging does not consider the global structure of data, so it’s restricted to analyzing merely pairwise structure.

Note In the 1850s during a cholera epidemic, London physician John Snow applied clustering techniques to plot the locations of cholera deaths on the map. The clustering indicated that death cases were located in the vicinity of polluted wells.

Partitional Clustering Partitional algorithms partition a set of n objects into k clusters or classes. In this case, k (number of clusters or classes) can be fixed a priori or be determined by the algorithm when optimizing the objective function. 486

Chapter 13

Clustering & Multi-objective Clustering

The most popular representative of the family of partitional clustering algorithms is k-means (MacQueen, 1967). K-means is one of the simplest unsupervised learning methods for finding a clustering of a set of objects. It follows a simple procedure to partition a given data set into k clusters, where k is a number fixed a priori. In its initialization phase it defines k centroids, one for each cluster. There are different approaches for defining centroids. We could choose k random objects from the data set as centroids (naïve approach) or choose them in a more sophisticated way by selecting them to be as far as possible from each other. The choice made can affect performance later, as the initial centroids will influence the final outcome. The main body of the k-means algorithm is formed by an outer loop that verifies whether a stopping condition has been reached; this outer loop contains an inner loop that passes through all data points. Within this inner loop—and while examining a data point P—we decide the cluster to which P should be added by comparing the distance of P to the centroid of every cluster, and ultimately we add it to the cluster with the nearest associated centroid. Once all data points have been examined for the first time—in other words, the inner loop ends for the first time—a primary phase of the algorithm has been completed and an early clustering has been obtained. At this point, we need to refine our clustering; therefore, we recalculate the k centroids obtained in the previous step (recall that centroids are the average vector of their respective clusters), which will give us new centroids. The inner loop is executed again if the stopping condition has not been met, adding every data point to the cluster with the nearest new associated centroid. This is the main process of k-means; notice that the k centroids change their location step-by-step until no more changes are made. In other words, a stopping condition for the algorithm is that the set

487

Chapter 13

Clustering & Multi-objective Clustering

of centroids does not change from one iteration to the next. A pseudocode of k-means can be seen in the following lines: K-Means(dataPoints, k) { cList = InitializeKCentroids() clusters = CreateClusters() while (!stoppingCondition) { foreach (pj in dataPoints) { dj = Calculate distance from pj to every centroid cList_j Assign pj to clusters_jwhose dj is minimum } UpdateCentroids() } } The objective function being optimized (minimized in this case) is the Sum of Squared Errors (SSE), also known as Intra-Class distance or Compactness: k

SSE = ååd ( x , centroid i )

2

i =1 xÎCi

Where k is the number of clusters, Ci is the ith cluster, centroidi represents the centroid associated with the ith cluster, and d(a, b) is a distance or similarity measure (usually Euclidean distance) between x and centroidi. Thus, another possible stopping condition for k-means is having reached a very small value for SSE. In Figure13-4 we can see the first step of the k-means algorithm— choosing k centroids. In this graphic, blue points denote data points and black points denote centroids. 488

Chapter 13

Clustering & Multi-objective Clustering

Figure 13-4. First step of k-means, choosing k = 3 random objects or data points as centroids Figure13-5 shows the k = 3 clusters that result from having selected the set of centroids from the first step.

Figure 13-5. Clustering obtained after choosing the set of centroids in the first step and considering a distance measure to determine similarity between data points 489

Chapter 13

Clustering & Multi-objective Clustering

The final step of the loop is to recalculate the centers of the clusters or centroids; this process is illustrated in Figure13-6.

Figure 13-6. Centroids being recalculated as the average vector of the cluster they represent The steps represented in the preceding figures are repeated until a stopping condition is met. To summarize, k-means is a simple, efficient algorithm that can end up at a local minimum if we use a very small value of SSE as the stopping condition. Its main disadvantage is its high sensitivity to outliers (isolated data points), which can be alleviated by removing data points that are much farther away from the set of centroids when compared to other data points.

Practical Problem: K-Means Algorithm In this section, we will be implementing what is probably the most popular clustering algorithm ever: k-means. To provide an object-oriented approach to our implementation, we will create Cluster and Element classes that will incorporate all actions and properties related to clusters and objects (data points); the Cluster class can be seen in Listing 13-1. 490

Chapter 13

Clustering & Multi-objective Clustering

Listing 13-1. Cluster Class public class Cluster { public List Objects { get; set; } public Element Centroid { get; set; } public intClusterNo { get; set; } public Cluster() { Objects = new List(); Centroid = new Element(); } public Cluster(IEnumerable centroid, intclusterNo) { Objects = new List(); Centroid = new Element(centroid); ClusterNo = clusterNo; } public void Add(Element e) { Objects.Add(e); e.Cluster = ClusterNo; } public void Remove(Element e) { Objects.Remove(e); }

491

Chapter 13

Clustering & Multi-objective Clustering

public void CalculateCentroid() { var result = new List(); vartoAvg = new List(Objects); var total = Total; if (Objects.Count == 0) { toAvg.Add(Centroid); total = 1; } var dimension = toAvg.First().Features.Count; for (vari = 0; i< dimension; i++) result.Add(toAvg.Select(o =>o.Features[i]).Sum() / total); Centroid.Features = new List(result); } public double AverageLinkageClustering(Cluster c) { var result = 0.0; foreach (var c1in c.Objects) result += Objects.Sum(c2 =>Distance. Euclidean(c1.Features, c2.Features)); return result / (Total + c.Total); } public int Total { get { return Objects.Count; } } } 492

Chapter 13

Clustering & Multi-objective Clustering

The Cluster class contains the following properties: •

Objects: set of objects included in the cluster

•

Centroid: centroid of the cluster

•

ClusterNo: ID of the cluster to differentiate it from the rest

•

Total: number of elements in the group or cluster

The class also contains the following methods: •

Add(): adds an element to the cluster

•

Remove(): removes an element from the cluster

•

CalculateCentroid(): calculates the centroid of a cluster

•

AverageLinkageClustering(): calculates the AverageLinkageClustering similarity measure between clusters, as previously detailed

The Element class representing an object to be clustered is shown on Listing 13-2.

Listing 13-2. Element Class public class Element { public List Features { get; set; } public int Cluster { get; set; } public Element(int cluster = -1) { Features = new List(); Cluster = cluster; } 493

Chapter 13

Clustering & Multi-objective Clustering

public Element(IEnumerable features) { Features = new List(features); Cluster = -1; } } The class contains a Cluster property that indicates the clusterID of the cluster to which the object belongs; code from both constructors is self-explanatory. The KMeans class, representing the algorithm of the same name, is illustrated in Listing 13-3.

Listing 13-3. KMeans and DataSet Classes public class KMeans { public int K { get; set; } public DataSetDataSet { get; set; } public List Clusters { get; set; } private static Random _random; private constintMaxIterations = 100; public KMeans(int k, DataSetdataSet) { K = k; DataSet = dataSet; Clusters = new List(); _random = new Random(); } public void Start() { InitializeCentroids(); vari = 0; 494

Chapter 13

Clustering & Multi-objective Clustering

while (i= 0) Clusters[oldCluster].Remove(obj); } UpdateCentroids(); i++; } } private void InitializeCentroids() { RandomCentroids(); } private void RandomCentroids() { var indices = Enumerable.Range(0, DataSet.Objects.Count). ToList(); Clusters.Clear(); for (vari = 0; i< K; i++) { varobjIndex = _random.Next(0, indices.Count); Clusters.Add(new Cluster(DataSet.Objects[objIndex].Features, i)); indices.RemoveAt(objIndex); } } 495

Chapter 13

Clustering & Multi-objective Clustering

private intMinDistCentroid(Element e) { var distances = new List(); for (vari = 0; i d == minDist); } private void UpdateCentroids() { foreach (var cluster in Clusters) cluster.CalculateCentroid(); } } public class DataSet { public List Objects { get; set; } public DataSet() { Objects = new List(); } public void Load(List objects) { Objects = new List(objects); } }

496

Chapter 13

Clustering & Multi-objective Clustering

The properties or fields are self-explanatory; in this case, we have decided to use a maximum number of iterations as the stopping condition. The methods of the class are described in the following points: •

InitializeCentroids(): method created considering the possibility of having different centroid initialization procedures.

•

RandomCentroids(): centroid initialization procedure where we assign k randomly selected objects as centroids of k clusters

•

MinDistCentroid(): returns the clusterID of the cluster to which the input object is closer; i.e., at minimum distance

•

UpdateCentroids(): updates the k centroids by calling the CalculateCentroid() method of the Cluster class

Now that we have all components in place, let’s test our clustering algorithm by creating a test application where we create a data set; Listing 13-4 illustrates this code.

Listing 13-4. Testing the K-Means Algorithm var elements = new List { new UnsupervisedLearning. Clustering.Element(new double[] {1, 2}), new UnsupervisedLearning. Clustering.Element(new double[] {1, 3}),

497

Chapter 13

Clustering & Multi-objective Clustering

new UnsupervisedLearning. Clustering.Element(new double[] {3, 3}), new UnsupervisedLearning. Clustering.Element(new double[] {3, 4}), new UnsupervisedLearning. Clustering.Element(new double[] {6, 6}), new UnsupervisedLearning. Clustering.Element(new double[] {6, 7}) }; vardataSet = new DataSet(); dataSet.Load(elements); varkMeans = new KMeans(3, dataSet); kMeans.Start(); foreach (var cluster in kMeans.Clusters) { Console.WriteLine("Cluster No {0}", cluster.ClusterNo); foreach (varobj in cluster.Objects) Console.WriteLine("({0}, {1}) in {2}", obj.Features[0], obj. Features[1], obj.Cluster); Console.WriteLine("--------------"); } The result obtained after executing the code from Listing 13-4 is shown in Figure13-7. Note that in this case we have three easily distinguished groups, as the figure illustrates.

498

Chapter 13

Clustering & Multi-objective Clustering

Figure 13-7. Execution of the k-means algorithm with k = 3 So far, we have examined single-clustering algorithms, or algorithms where we optimize a single objective function. In the case of k-means, it was the Sum of Squared Errors, also known as intra-class distance (minimizes the distance of objects within a group). Another function that we might try to optimize is the inter-class (maximize distance of objects from different groups) function. In the next section we will begin studying multi-objective clustering in which we do not consider only a single function to optimize but rather several functions, and we attempt to optimize them all at once.

Multi-objective Clustering Nowadays, many real-life problems force us to consider not only the best possible value for a given function but also the value of several functions all related to the problem at hand. For instance, zoning, a technique that belongs to the area of urban studies, appeared for the first time in the nineteenth century to separate residential areas from industrial ones. The main idea with this technique, the most popular in urbanization, is to produce a partition of hom*ogeneous regions according to several variables or criteria. These variables could be demographic—for instance, number of people who are older than twenty, number of people younger than ten, and so on. Finding such a partition is clearly a clustering problem involving 499

Chapter 13

Clustering & Multi-objective Clustering

the optimization of different functions. Therefore, we might try to find a clustering with the lowest intra-class distance (a.k.a compactness) and at the same time optimize the inter-class distance or any other function, which could very well be demographic in nature. A perfect clustering is that with the minimum intra-class distance and the maximum inter-class distance; hence, one could say that clustering is by nature a multi-objective optimization problem. We will begin this section by examining several relevant concepts or definitions related to multi-objective clustering. Many optimization problems often involve optimizing multiple objective functions at the same time; such problems are known as a multi- objective optimization problems (MOPs). They can be stated as follows: minimize F ( x ) = ( f1 ( x ) , f2 ( x ) ,¼, fn ( x ) ) x ÎA In this case, A represents the feasible space of the problem—the set of all feasible solutions, the ones fulfilling every constraint of the problem. A vector u = ( u1 , u 2 ,¼ u n ) is said to be dominated by another vector, v = ( v 1 , v 2 ,¼ v n ) , denoted u < v, if and only if for all of index i we can verify that u i £ v i . In any other case u is said to be a non-dominated vector. Notice that “domination” depends on whether we want to minimize or maximize the objective functions; recall that it’s always possible to transform a minimization problem into a maximization problem and the other way around. Having multiple objectives denotes a significant issue—the improvement of one objective function could lead to the deterioration of another. Thus, a solution that optimizes every objective rarely exists; instead of looking for that solution a trade-off is searched for. Pareto optimal solutions represent this trade-off.

500

Chapter 13

Clustering & Multi-objective Clustering

A feasible solution x is said to be Pareto optimal if there is no solution y such that F(x) 0) { var current = queue.Dequeue(); path.Add(current.State); foreach (var c in current.Children) queue.Enqueue(c); } return path; } } The DFS implemented in Listing 15-4 relies on a stack data structure that is used to simulate the intrinsically recursive nature of DFS; thus it helps us avoid having to use function recursion and allows us to reduce the coding to a simple loop. Remember: Stacks are LIFO (Last-In-First-Out) data structures, and therefore we stack children in reverse order, as the following code illustrates.

562

Chapter 15

Game Programming

Listing 15-4. Dfs Class public class Dfs :UninformedMethod { public Dfs(Tree tree):base(tree) { } public override ListExecute() { var path = new List(); var stack = new Stack(); stack.Push(Tree); while (stack.Count> 0) { var current = stack.Pop(); path.Add(current.State); for (vari = current.Children.Count- 1; i>= 0; i--) stack.Push(current.Children[i]); } return path; } } Any other uninformed search strategy is basically a variation of the previous ones—DFS and BFS.The depth-limited search class illustrated in Listing 15-5 is a direct descendant of DFS.In this class, we include two properties: •

DepthLimit: defines the maximum depth reached

•

Value: determines the value to be found in the tree of states 563

Chapter 15

Game Programming

In this case, we implement the recursive version of the DFS algorithm; it is easier for us to build the path from the root to the Value node if we use recursion. Notice we have three stopping conditions in the algorithm: the Value node has been found, we have reached the depth limit, or we have reached a leaf.

Listing 15-5. Dls Class public class Dls: UninformedMethod { public intDepthLimit{ get; set; } public T Value { get; set; } public Dls(Tree tree, intdepthLimit, T value) : base(tree) { DepthLimit = depthLimit; Value = value; } public override ListExecute() { var path = new List(); if (RecursiveDfs(Tree, 0, path)) return path; return null; } private bool RecursiveDfs(Tree tree, int depth, ICollection path) { if (tree.State.Equals(Value)) return true;

564

Chapter 15

Game Programming

if (depth == DepthLimit || tree.IsLeaf) return false; path.Add(tree.State); if (tree.Children.Any(child =>RecursiveDfs(child, depth + 1, path))) return true; path.Remove(tree.State); return false; } } Finally, iterative deepening search, as previously described, uses depth-limit search as a submethod to find the shallowest depth to a goal state (Listing 15-6).

Listing 15-6. Ids Class public class Ids :UninformedMethod { public DlsDls{ get; set; } public intMaxDepthSearch{ get; set; } public intDepthGoalReached{ get; set; } public T Value { get; set; } public Ids(Tree tree, intmaxDepthSearch, T value) : base(tree) { MaxDepthSearch = maxDepthSearch; Value = value; }

565

Chapter 15

Game Programming

public override ListExecute() { for (var depth = 1; depth = 0 && (string.IsNullOrEmpty (Path) || Path.Last() != (backwards ? 'U' : 'D'))) result.Add(up); if (down._blankPos.Item1 >= 0 && (string.IsNullOrEmpty (Path) || Path.Last() != (backwards ? 'D' : 'U'))) result.Add(down); if (lft._blankPos.Item1 >= 0 && (string.IsNullOrEmpty (Path) || Path.Last() != (backwards ? 'L' : 'R'))) result.Add(lft); if (rgt._blankPos.Item1 >= 0 && (string.IsNullOrEmpty (Path) || Path.Last() != (backwards ? 'R' : 'L'))) result.Add(rgt); return result; }

570

Chapter 15

Game Programming

public BoardMove(Move move, bool backwards = false) { varnewState = new T[_n, _n]; Array.Copy(State, newState, State.GetLength(0) * State. GetLength(1)); varnewBlankPos = new Tuple(-1, -1); var path = ""; switch (move) { case GameProgramming.Move.Up: if (_blankPos.Item1- 1 >= 0) { // Swap positions of blank tile and x tile var temp = newState[_blankPos.Item1- 1, _blankPos.Item2]; newState[_blankPos.Item1- 1, _blankPos.Item2] = Blank; newState[_blankPos.Item1, _blankPos.Item2] = temp; newBlankPos = new Tuple(_blankPos.Item1- 1, _blankPos.Item2); path = backwards ? "D" : "U"; } break; case GameProgramming.Move.Down: if (_blankPos.Item1 + 1 < _n) { var temp = newState[_blankPos.Item1 + 1, _blankPos.Item2]; newState[_blankPos.Item1 + 1, _blankPos.Item2] = Blank; newState[_blankPos.Item1, _blankPos.Item2] = temp; newBlankPos = new Tuple(_blankPos.Item1 + 1, _blankPos.Item2); path = backwards ? "U" : "D"; } break; 571

Chapter 15

Game Programming

case GameProgramming.Move.Left: if (_blankPos.Item2- 1 >= 0) { var temp = newState[_blankPos.Item1, _blankPos.Item2- 1]; newState[_blankPos.Item1, _blankPos.Item2- 1] = Blank; newState[_blankPos.Item1, _blankPos.Item2] = temp; newBlankPos = new Tuple(_blankPos.Item1, _blankPos.Item2- 1); path = backwards ? "R" : "L"; } break; case GameProgramming.Move.Right: if (_blankPos.Item2 + 1 < _n) { var temp = newState[_blankPos.Item1, _blankPos.Item2 + 1]; newState[_blankPos.Item1, _blankPos.Item2 + 1] = Blank; newState[_blankPos.Item1, _blankPos.Item2] = temp; newBlankPos = new Tuple(_blankPos.Item1, _blankPos.Item2 + 1); path = backwards ? "L" : "R"; } break; } return new Board(newState, Blank, newBlankPos, Path + path); } public bool Equals(Board x, Board y) { if (x.State.GetLength(0) != y.State.GetLength(0) || x.State.GetLength(1) != y.State.GetLength(1)) return false; 572

Chapter 15

Game Programming

for (vari = 0; i 0) { varcurrentForward = queueForward.Dequeue();

575

Chapter 15

Game Programming

varcurrentBackward = queueBackward.Dequeue(); varexpansionForward = currentForward.Expand(); varexpansionBackward = currentBackward.Expand(true); foreach (var c in expansionForward) { if (c.Path.Length == 1 &&c.Equals(c, Game.Goal)) return c.Path; queueForward.Enqueue(c); } foreach (var c in expansionBackward) queueBackward.Enqueue(c); var path = SolutionMet(queueForward, expansionBackward); if (path != null) return path; } return null; } private string SolutionMet(Queueexpansion Forward, ListexpansionBackward) { for (vari = 0; ib.Equals(b, expansionBackward[i]));

576

Chapter 15

Game Programming

return first.Path + new string(expansionBackward[i].Path. Reverse().ToArray()); } } return null; } } Our BS algorithm will perform two searches, each consisting of a BFS procedure that uses a queue to traverse the state tree through levels. We implement a BFS to search forward and another to search backward, and the point where these two searches meet is iteratively checked by the SolutionMet() method. The loop examining whether every expanded node with Path length 1 matches the goal state acts as a base case for the scenario where the goal state is a step away from the initial board. Figure15-8 graphically depicts the functioning of the bidirectional search algorithm.

Figure 15-8. The forward search (on the left) and the backward search (on the right).The point in the middle indicates the current node being processed in the BFS, and the circles around it represent different levels of the tree. Blue points indicate nodes that have been discovered and processed during the search, and gray ones indicate queued nodes. The green points indicate the node where both searches would meet. 577

Chapter 15

Game Programming

Both searches meet at the green point. To find this link or relationship between the forward and backward searches we checked the set of expanded nodes (gray points in the figure) using the SolutionMet() method. The purpose of this method is to check all enqueued points from the forward search against all expanded nodes (points in the nearest circle to the middle processed node) from the backward search and look for matches in their state or board. If a full match is found then we output the path that results from adding the subpaths of the node, forward and backward, where both searches met. In order to test our BS we will create the experiment shown in Listing 15-10.

Listing 15-10. Testing Our Bidirectional Search Algorithm on the Hardest 8-Puzzle Configuration var state = new[,] { {6, 4, 7}, {8, 5, 0}, {3, 2, 1} }; vargoalState = new[,] { {1, 2, 3}, {4, 5, 6}, {7, 8, 0} }; var board = new Board(state, 0, new Tuple(1, 2), ""); var goal = new Board(goalState, 0, new Tuple (2, 2), "");

578

Chapter 15

Game Programming

varslidingTilesPuzzle = new SlidingTilesPuzzle(board, goal); varbidirectionalSearch = new Bs(slidingTilesPuzzle); varstopWatch = new Stopwatch(); stopWatch.Start(); var path = bidirectionalSearch.BidirectionalBfs(); stopWatch.Stop(); foreach (var e in path) Console.Write(e + ", "); Console.WriteLine('\n' + "Total steps: " + path.Length); Console.WriteLine("Elapsed Time: " + stopWatch. ElapsedMilliseconds / 1000 + " segs"); In this experiment, we are using one of the hardest 8-puzzle configurations; it requires 31 steps to be solved in the optimal case. We are also using an object of type Stopwatch to measure the time consumed by the algorithm while finding a solution. The result after executing the previous code can be seen in Figure15-9.

Figure 15-9. Solution obtained in 11 seconds To verify the correctness of the solution we can simply loop through the path or list of moves obtained and execute the equivalent moves from the initial board, checking that the last board obtained matches the goal state.

579

Chapter 15

Game Programming

Note Before outputting the sequence of moves of the BS algorithm we must reverse the path string obtained in the backward search. Remember that this path was built by adding moves to the end, not the beginning, of the string; therefore, we must reverse it in order to get the correct path to the goal node.

I nformed Search In an informed search we use knowledge of the problem apart from its own definition with the intention of using it in solving a problem as efficiently as possible. Thus, in an informed search algorithm we try to be smart about what paths to explore. The general approach for informed search methods is represented by a family of algorithms known as Best First Search. A Best First Search type of method always relies on an evaluation function F(n) that associates a value with every node n of the state tree. This value is supposed to represent how close the given node is to reaching a goal node; hence, a Best First Search method usually chooses a node n with the lowest value F(n) to continue the search procedure (Figure15-10). Even though we refer to this family of algorithms as “Best First,” in reality there’s no certain way to determine the lowest-cost path to a goal node. If that were possible then we would always be able to obtain an optimal solution without the need to put in any extra effort (heuristics and so forth).

580

Chapter 15

Game Programming

Figure 15-10. In a Best First Search method we always pick a node n with the lowest possible F(n) value to continue the search. In this case, F = 3, so the search continues from that node. Because informed search strategies search the most promising branches of the state space first, they are capable of •

finding a solution more quickly;

•

finding solutions even when there is limited time available; and

•

finding a better solution, since the more profitable parts of the state space can be examined while ignoring the unprofitable parts.

Best First Search is a search strategy and, as mentioned before, a family of algorithms whose main representatives are Greedy Best First Search and the A* search. A Greedy Best First Search is basically a Best First Search in which the evaluation function F(n) is a heuristic function; i.e., F(n) = H(n). Examples of heuristic functions for different problems include straight distance on a map between two points, number of misplaced elements, and so on. They represent an approach for embedding additional knowledge in the solution process of a problem. When H(n) = 0 it implies we have reached a goal node. Greedy Best First Search expands the node that appears to be closest to goal but is neither optimal nor complete (can fall into infinite 581

Chapter 15

Game Programming

loops). An obvious problem with the method is that it doesn’t take into account the cost up to the current node, so as mentioned before it isn’t optimal and can wander into deadends, like DFS.In methods where we use heuristics we could obtain a drastic reduction of complexity if we use a smart heuristic that would lead us in the right direction in a few steps.

Note When the state space is too big, an uninformed blind search can simply take too long to be practical, or can significantly limit how deep we’re able to look into the space. Thus, we must look for methods that reduce the area of the state space by making smart decisions along the way; i.e., we must look for informed methods. A* search (Hart, Nilsson, and Raphael, 1968) is a very popular method and is the best-known member of the Best First Search family of algorithms. The main idea behind this method is to avoid expanding paths that are already expensive (considering the cost of traversing through the root to the current node) and always expanding the most promising first. The evaluation function in this method is the sum of two functions; i.e., F(n) = G(n) + H(n), where •

G(n) is the cost (so far) of reaching node n; and

•

H(n) is a heuristic to estimate the cost of reaching a goal state from node n.

Because we’re actually looking for the optimal path between the initial state and some goal state, a better measure of how promising a state is would be the sum of the cost-so-far and our best estimate of the cost from that node to the nearest goal state (Figure15-11).

582

Chapter 15

Game Programming

Figure 15-11. Diagram showing the relation between G(s) and H(s) To guide the search through the immense space state, we use heuristics. The information provided by the heuristic is supposed to help us find a feasible, short path to the goal state or configuration. When developing a heuristic it’s important to make sure that it holds the admissibility criteria. A heuristic is considered admissible if it doesn’t overestimate the minimum cost of reaching the goal state from the current state, and if admissible then the A* search algorithm will always find an optimal solution.

A* fortheSliding Tiles Puzzle The tree structure representing the state space for the Sliding Tiles Puzzle will be the same as was developed for the bidirectional search. The neighborhood of the current node will consist of boards that have their blank tile swapped into all possible positions. The most common heuristic for the Sliding Tiles Puzzle is Misplaced Tiles, and it is probably also the simplest heuristic for this puzzle. The Misplaced Tiles heuristic, as the name suggests, returns the number of tiles that are misplaced—whose position in the current board does not match their position in the goal state or board. It’s admissible since the number returned does not overestimate the minimum number of moves required to get to the goal state. At the very least you have to move every misplaced tile once to swap them to their goal position; hence, it is admissible. 583

Chapter 15

Game Programming

It’s important to point out that when calculating any heuristic for the Sliding Tiles Puzzle we should never take into account the blank tile. If we consider the blank tile in the heuristic calculation then we could be overestimating the real cost of the shortest path to the goal state, which makes the heuristic non-admissible. Consider what would happen if we took into account the blank tile in a board that is just a step away from reaching the goal state, as shown in Figure15-12.

Figure 15-12. If we consider the blank tile, our path to a goal state would be 2, but in reality it is 1; thus, we are overestimating the real cost of a shortest path toward a goal state The A* algorithm with the Misplaced Tiles heuristic takes about 2.5 seconds to find the goal state. In reality, we can do much better than that, so let’s try to find a more clever heuristic that will lower the timeframe and the number of nodes visited.

Note For a full code in C# of this problem, refer to the following article by the author: https://visualstudiomagazine.com/ Articles/2015/10/30/Sliding-Tiles-C-Sharp-AI.aspx.

584

Chapter 15

Game Programming

The Manhattan Distance, or Block Distance, heuristic between points A=(x1, y1) and B=(x2, y2) is defined as the sum of the absolute difference of their corresponding coordinates: MD = x 1 - x 2 + y 1 - y 2 Manhattan Distance is admissible because for each tile it returns the minimum number of steps required to move that tile to its goal position. Manhattan Distance is a more accurate heuristic than Misplaced Tiles; therefore, the reduction in time complexity and nodes visited will be substantial. We are providing better information to guide the search and so the goal is found much more quickly. Using this heuristic, we get an optimal solution in 172 milliseconds (refer to the previously detailed article for the complete code in C#). The Linear Conflict heuristic provides information on necessary moves that are not counted by the Manhattan Distance. Two tiles tj and tk are said to be in a linear conflict if tj and tk are in the same line, the goal positions of tj and tk are both in that line, tj is to the right of tk, and the goal position of tj is to the left of the goal position of tk.

Figure 15-13. Tiles 3 and 1 are in the correct row but in the wrong column To get them to their goal positions we must move one of them down and then up again; these moves are not considered in the Manhattan Distance. A tile cannot appear related in more than one conflict, as solving 585

Chapter 15

Game Programming

a determined conflict might imply the resolution of other conflicts in the same row or column. Hence, if tile 1 is related to tile 3in a conflict then it cannot be related to a conflict with tile 2, as this may become an overestimation of the shortest path to a goal state and could turn our heuristic into a non-admissible one. To test the Linear Conflict + Manhattan Distance heuristic combination, we’ll use the 4 × 4 board seen in Figure15-14; this board requires 55 moves to reach the goal state. The value of a node n will be given by F(n) = Depth(n) + MD(n) + LC(n). It’s possible to combine these heuristics as the moves they represent do not intersect, and consequently we will not be overestimating the cost of the shortest path to a goal state.

Figure 15-14. 4 × 4 board for testing Manhattan Distance + Linear Conflict heuristic. A 15-tile problem has a much broader state space than the 8-tile problem. After completing an execution that traversed over a million nodes and consumed a time of 124199 milliseconds (little over 2 mins), the algorithm provided us with a solution. The pattern database heuristic is defined by a database containing different states of the game. Each state is associated with the minimum number of moves required to take a pattern (subset of tiles) to its goal position. In this case, we built a small pattern database by making a BFS backward, starting at the 8-tile goal state. The results were saved in a

586

Chapter 15

Game Programming

.txt file of merely 60,000 entries. The pattern chosen for the database is typically known as the fringe, and in this case it contains tiles from the top row and the leftmost column.

Figure 15-15. Pattern used in 3 × 3 board The pattern database heuristic function is computed by a table look-up function. In this case, it’s a dictionary lookup that has 60,000 stored patterns. It philosophically resembles those of the Divide and Conquer and Dynamic Programming techniques. Using the pattern database technique, we can obtain a time of 50 milliseconds for solving the hardest 8-tile problem or configuration. The more entries we add to the database the lower the time consumed by the algorithm in finding a goal state. In this case, the trade-off between memory and time favors the former and helps us obtain a good running time. This is how it usually works; you use more memory in order to reduce the execution time of your algorithms. The pattern database heuristic represents the definitive alternative when you want to solve 4 x 4 puzzles or m x n puzzles where n and m are greater than 3. A final suggestion to the reader would be to combine the A* search and heuristics presented in this section with a bidirectional search and compare results.

587

Chapter 15

Game Programming

Summary In this chapter we introduced game programming and, more specifically, searching in games. We analyzed the fundamental methods for searching in state space, including those that classify as uninformed search—BFS, DFS, DLS, IDS, and BS—and those that classify as informed search: Best- First Search and A*. We implemented a bidirectional search tailored to the Sliding Tiles Puzzle and using BFS as a sub-procedure. Ultimately, we showed how to develop an A* search for the Sliding Tiles Puzzle using different heuristics, combining some of those heuristics, and assessing their performance in regards to time complexity through the use of the C# Stopwatch class.

588

CHAPTER 16

Game Theory: Adversarial Search & Othello Game The most relevant figure associated with game theory is, without any doubt, John von Neumann, the Hungarian-American mathematician— one of the greatest of the twentieth century. Although others preceded him in formulating concepts connected to game theory (notably Emile Borel), it was von Neumann who in 1928 published the paper that laid the foundation for the theory of two-person zero-sum games. His work culminated in an essential book on game theory written in collaboration with Oskar Morgenstern and titled Theory of Games and Economic Behavior (1944). The theory developed by von Neumann and Morgenstern is highly associated with a class of games called two-person zero-sum games, or games where there are only two players and in which one player wins what the other player loses. Their mathematical framework initially made the theory applicable only under special and limited conditions. Over the past six decades this situation has dramatically changed, and the framework has been strengthened and generalized. Since the late 1970s it has been possible to assert that game theory is one of the most important and useful

© Arnaldo Pérez Castaño 2018 A. Pérez Castaño, Practical Artificial Intelligence, https://doi.org/10.1007/978-1-4842-3357-3_16

589

Chapter 16

Game Theory: Adversarial Search & Othello Game

tools in many fields of science, particularly in economics. In the 1950s and 1960s, game theory was broadened theoretically and applied to problems of war and politics. Additionally, it has found applications in sociology and psychology and established links with evolution and biology. Game theory received special attention in 1994 with the awarding of the Nobel Prize in Economics to John Nash, John Harsanyi, and Reinhard Selten. John Nash, the subject of the 2001 Oscar-winning movie A Beautiful Mind, transformed game theory into a more general tool that enabled the analysis of win-win and lose-lose scenarios, as well as win-lose situations. Nash enabled game theory to address a central question: should we compete or cooperate? In this chapter, we will discuss various concepts and ideas drawn from game theory. We will address a sub-branch of game theory known as adversarial search, and we will describe the Minimax algorithm, which is typically applied in two-player zero-sum games of perfect information in a deterministic environment.

Note In 1950, John Nash demonstrated that finite games always have an equilibrium point at which all players choose actions that are best for them given their opponents' choices. The Nash equilibrium, also called strategic equilibrium, is a list of strategies, one for each player, that has the property that no player can unilaterally change his strategy and get a better payoff.

What Is Game Theory? A game is a structured set of tasks defined in an entertaining environment and manner so as to attract players (1 or more) to comply with logical rules that if properly fulfilled result in the game’s being completed.

590

Chapter 16

Game Theory: Adversarial Search & Othello Game

Game theory is the mathematical theory of how to analyze games and how to play them optimally; it’s also a way of looking at multiple human behaviors as if they were part of a game. Some of the most popular games that can be analyzed in game theory are Othello, blackjack, poker, chess, tic-tac-toe, backgammon, and so on. In reality, not only games as we know them or think about them are the topic of analysis in game theory. Rather, there are many other situations that can be formulated as games. Whenever rational people must make decisions within a framework of strict and known rules, and when each player gets a payoff based on the decisions of other players, we have a game. Examples include auctions, negotiations, military tactics, and more. The theory was initiated by mathematicians in the first half of the last century, but since then much research in game theory has been done outside of the mathematics area. The key aspects of game theory revolve around the identification of process participants and their various quantifiable options (choices), as well as the consideration of their preferences and subsequent reactions. If all these factors are carefully thought of, then the task of modeling the problem by game theory—along with the identification of all possible situations—becomes easier. One of the classic examples presented in the scientific literature to describe how games are analyzed in game theory is the Prisoner’s Dilemma (PD). The name of the game derives from the following situation, typically used to exemplify it. Suppose the police have arrested two people they know have committed an armed robbery together. Unfortunately, they lack enough admissible evidence to get a jury to convict them. They do, however, have enough evidence to send each prisoner away for two years for theft of the getaway car. The police chief now makes the following offer to each prisoner: If you will confess to the robbery, implicating your partner, and he does not also confess, then you’ll go free and he’ll get ten years. If you both confess, you’ll each get five years. If neither of you confesses, then you’ll each get two years for the auto theft. Table16-1 illustrates the payoff or benefit matrix in this problem. 591

Chapter 16

Game Theory: Adversarial Search & Othello Game

Table 16-1. Prisoner’s Dilemma Payoff Matrix Prisoner B, stays silent

Prisoner B, betrays

Prisoner A, stays silent

2, 2

0, 10

Prisoner A, betrays

10, 0

5, 5

The cells of the matrix define payoffs for both players and for each combination of actions. In every pair (a, b), player A’s payoff equals a and player B’s payoff equals b. •

If both players stay silent then they each get a payoff of 2. This appears in the upper-left cell.

•

If neither of them stays silent, they each get a payoff of 5; this appears as the lower-right cell.

•

If player A betrays and player B remains silent then player A gets a payoff of 10 (going free) and player B gets a payoff of 0 (ten years in prison); this appears in the lower-left cell.

•

If player B betrays and player A stays silent then player B gets a payoff of 10 and player A gets 0; this appears in the upper-right cell.

Each player evaluates his or her two possible actions here by comparing their personal payoffs in each column, since this shows which of their actions is preferable, just to themselves, for each possible action by their partner. Therefore, if player B betrays then player A gets a payoff of 5 by also betraying and a payoff of 0 by staying silent. If player B stays silent then player A gets a payoff of 2 by also staying silent or a payoff of 10 by betraying player B.Consequently, player A is better off betraying regardless of what player B does. Player B, on the other hand, evaluates his actions by comparing his payoffs down each row, and he comes to exactly

592

Chapter 16

Game Theory: Adversarial Search & Othello Game

the same conclusion that player A does. Whenever an action for a player is superior when compared to each possible action by an opponent we say that the first action strictly dominates the second one (recall terms such as Pareto set and Pareto optimality from Chapter 13). In the PD, confessing strictly dominates refusing for both players. Both players know this about each other, entirely eliminating any temptation to depart from the strictly dominated path. Hence, both players will betray, and both will go to prison for five years. These days, AIs capable of defeating human champions for games such as chess, checkers, and backgammon have been created. Most recently (March 2016), the Google DeepMind’s AlphaGo program, using a self-learning algorithm (we’ll look into this in Chapter 17, “Reinforcement Learning”), was able to defeat the world champion of Go, Lee Sedol (Figure16-1).

A dversarial Search In this book, we will focus on a sub-branch of game theory known as adversarial search, which is usually applied to board games. In adversarial search, we examine problems that arise when we try to plan ahead or look into the future of a world where other agents are planning against us. Thus, adversarial search becomes necessary in competitive environments where there are conflicting goals and more than one agent. Board-game analysis is one of the oldest branches of AI (Shannon, Turing, Wiener, and Shanon 1950). Such games present a very abstract and pure form of competition between two opponents and clearly require a form of “intelligence.” The states of a game are easy to represent, and the possible actions of the players are well defined. The world states are fully accessible even though it’s a contingency problem, because the characteristics of the opponent are not known in advance. Board games are not only difficult because of their contingency, but also because the search trees can become astronomically large. 593

Chapter 16

Game Theory: Adversarial Search & Othello Game

Figure 16-1. Lee Sedol vs AlphaGo, March 2016 Concepts from the area of game theory for which we will need to find a common ground of understanding are presented in the following points:

594

•

Deterministic Game Environment: A game is said to be deterministic if it does not involve any random process like the throwing of a dice; i.e., a player’s actions lead to completely predictable outcomes. Games such as checkers, chess, and Othello are deterministic.

•

Stochastic Game Environment: A game is said to be stochastic if it involves some random process like the throwing of a dice. Games such as backgammon and dominoes are stochastic.

•

Utility Function: is a mapping from states of the world to real numbers. These numbers are interpreted as measures of an agent’s level of happiness in the given states.

Chapter 16

Game Theory: Adversarial Search & Othello Game

•

Constant-Sum Game: A two-player game is constant- sum if there exists a constant c such that for each strategy s ∈ A1 × A2 it is the case that u1(s) + u2(s) = c being A1 is the set of actions of one of the players and A2 the set of actions of the other player.

•

Zero-Sum Game: a constant-sum game where c = 0; i.e., utility values at the end of the game are always equal in absolute value and opposite in sign.

•

Imperfect Information Game: a game where the players do not have all information regarding the state of other players. Games such as poker, Scrabble, and bridge are imperfect in their information.

•

Perfect Information Game: a game whose environment is fully observable by all players; i.e., every player is aware of other players’ state. Games such as Othello, checkers, and chess are of perfect Information.

Considering previously detailed concepts, we can create Table16-2, which details by row and column headers what method would be required to solve a game that depends on conditions defined.

Table 16-2. Methods for Solving Different Types of Games Zero-Sum

Non-Zero Sum

Perfect Information

Minimax, Alpha-Beta

Backward induction, retrograde analysis

Imperfect Information

Probabilistic Minimax

Nash equilibrium

595

Chapter 16

Game Theory: Adversarial Search & Othello Game

In this book, we will focus on two-player zero-sum games—games where the value achieved by a player is lost, in the same quantity, by the other. Thus, from the next section onward, we’ll be discussing the most relevant algorithm that is applied to this type of game.

Note An international program known as “Prism” run by the US Secret Service agencies uses a software model based on game theory to determine the predictability of terrorist activities, identities, and possible locations.

Minimax Search Algorithm Minimax search is an algorithm applied in two-player, zero-sum, deterministic, perfect information games to determine the optimal strategy for a player (MAX) at a given stage of the game and assuming the other player will also make optimal plays (MIN). It’s applied in games such as chess, Othello, tic-tac-toe, and more. When executing this algorithm, we traverse the state space tree and represent each move in terms of losses or gains for one of the players. Therefore, this method can only be used to make decisions in zero-sum games, where one player’s loss is the other player’s gain. Theoretically, this search algorithm is based on Von Neumann’s Minimax theorem, which states that in these types of games (zero-sum, deterministic, perfect information) there is always a set of strategies that leads to both players’ gaining the same value, and that seeing as this is the best possible value one can expect to gain, one should employ this set of strategies.

596

Chapter 16

Game Theory: Adversarial Search & Othello Game

Note A Minimax player (MAX) is a player that plays optimally, assuming its opponent (MIN) is also playing optimally but in a different direction; i.e., one maximizes and the other minimizes results. Hence, in the Minimax algorithm we assume there are two players; namely, MAX and MIN.A search tree is generated in a depth-first style, starting with the current game position and going all the way up to an end-game position. An end-game position could be reached when we get to either a leaf node (node representing an actual end of the game) or a node at MaxDepth, the maximum depth the search will go to. Because most games possess a gigantic state search, we typically cannot make it to a leaf node. Thus, it is usually the node at MaxDepth where the DFS stops and starts backtracking. Before backtracking, the procedure gets a utility value from the end-game position node. This value is obtained from a heuristic that tells us how close we are to winning from that point onward. Afterward, the utility value is backtracked, and, depending on whether the parent node N belongs to a tree level or a depth corresponding to a MAX player or a MIN player, the utility value of N is obtained from its children c1, c2, … ,cm as Max(c1, c2, …, cm), where Max() is a function returning the maximum value of its arguments, or as Min(c1, c2, …, cm), where Min() is a function returning the minimum value of its arguments. Figure16-2 illustrates the functioning of the algorithm.

597

Chapter 16

Game Theory: Adversarial Search & Othello Game

Figure 16-2. Execution of a Minimax algorithm where MaxDepth = 2. The method first calculates the values of nodes at MaxDepth and then moves those values up according to whether a node is a Max node or a Min node. Nodes denoted in orange are the ones selected to have their values elevated in the tree. A pseudocode of the algorithm would be the following: Minimax(Node n): output Real-Value { if (IsLeaf(n)) then return Evaluate(n); if (MaxDepth) then return Heuristics(n); if (n is a MAX node) { v = NegativeInfinity foreach (child of n) { v' = Minimax (child) if (v' > v) v= v' } return v }

598

Chapter 16

Game Theory: Adversarial Search & Othello Game

if (n is a MINnode){ v = PositiveInfinity foreach (child of n) { v' = Minimax (child) if (v' < v)v= v' } return v } } Notice in the pseudocode that we distinguish two methods for evaluating end-game nodes (leaf or MaxDepth reached). If we reached a leaf node, the evaluation procedure would output H or L depending on whether the root player is MAX or MIN.These values correspond to the range [L; H] of possible values a node can take. H indicates a win for MAX and L a win for MIN; because this is a zero-sum game we know that L + H = 0; i.e., L = -H. If we reach a node at MaxDepth then we output a value in the range [L; H] indicating how good that path would be from that point onward.

Note Every single-agent problem can be considered as a special case of a two-player game by making the environment one of the players, with a constant utility function; e.g., always 0.

A lpha-Beta Pruning A Minimax algorithm can potentially explore many nodes of the generated tree whose paths would eventually be dismissed by the algorithm as they would be overtaken (in terms of higher or lower values) by the value of other nodes. Let’s consider this scenario in the Minimax tree shown in Figure16-3. 599

Chapter 16

Game Theory: Adversarial Search & Othello Game

Figure 16-3. Pruning child nodes of MIN node with utility value 1 In this Minimax tree we have a subtree that can be pruned. Remember: Minimax executes a DFS for traversing the tree; therefore, at some point it will backtrack to the MIN node colored green—let it be G from now on. Once at G, it would have already discovered and updated values for MIN nodes 2 and 3. All discovered nodes whose values would have been updated at the moment of updating G are colored orange. Because when updating G the algorithm would already be aware of sibling nodes and their corresponding utility values 2 and 3, and considering that it already knows that because G is a MIN node its value will be always lower than the value it already discovered (1), then by simple logic facts, it must be that the final value of the root at MAX node must be 3. Thus, any further exploration of children of G would be in vain, and those branches can be dismissed, pruned in the search. For determining which branches or subtrees can be pruned, the Minimax algorithm suffers a slight modification where two values are added; namely, Alpha and Beta. The first will continuously update the highest value found on a level of the tree, while the latter will continuously update the lowest value. Using these values as reference, we will be able to decide whether a subtree should be pruned. A pseudocode of the algorithm can be seen in the next lines: 600

Chapter 16

Game Theory: Adversarial Search & Othello Game

MinimaxAlphaBetaPruning(Node n, Real beta, Real alpha): output Real-Value { if (IsLeaf(n)) then return Evaluate(n); if (MaxDepth) then return Heuristics(n); if (n is a max node) { v = beta foreach (child of n) { v' = minimax (child,v, alpha) if (v' > v) v = v' if (v >alpha) return alpha } return v } if (n is a min node) { v = alpha foreach (child of n) { v' = minimax (child,beta, v) if (v' < v) v = v' if (v Prob(S) where Prob(S) is a probability distribution—or deterministic, where T : S x A -> S.

Note Both planning and MDPs are considered search problems, with the difference being that in the first we deal with explicit actions and subgoals and in the latter we deal with uncertainty and utilities. In MDPs, a horizon determines whether our decision-making process will have an infinite time, a finite time, or an indefinite time (until some criteria is met). MDPs with infinite horizons are easier to solve as they do not have a deadline; furthermore, because in many cases it’s not clear how long a process will execute, it’s popular to consider infinite-horizon models of optimality.

639

Chapter 17

Reinforcement Learning

Figure 17-3. MRP modeling the working day of an android An infinite-horizon return vt is the total discounted reward from time step t up to infinity: ¥

v t = rt +1 + g * rt + 2 +¼ = å g k * rt + k +1 k =0

Notice again the convenience of the discount factor. If we were to add up all the rewards out into infinity, the sums would be infinite in general. To keep the math nice, and to put some pressure on the agent to get rewards sooner rather than later, we use a discount factor.

Value/Action–Value Functions & Policies Having rewards in MRPs and MDPs permits us to define values for states depending on the associated rewards. These tabular values are part of the value function, state–value function, or simply value of a state in an MRP. It’s the expected return starting from state s: V (s ) = R (s ) + g * 640

N (s)

å T [ s , s ¢] * V ( s ¢ )

s ¢ÎN ( s )

Chapter 17

Reinforcement Learning

In the preceding formula, we compute the expected long-term value of the next state by summing over all possible next states or neighbor states, s′, the product of the probability of making a transition from s to s′, and the infinite horizon expected discounted reward; i.e., value of s’. This formulation is based on Bellman’s Equation (1957), a.k.a. the Dynamic Programming Equation, and its Principle of Optimality, which states that an optimal policy has the property that whatever the initial state and initial decision are, the remaining decisions must constitute an optimal policy with regard to the state resulting from the first decision. In this case, the value function can be decomposed into an immediate reward R and a discounted value of a successor, neighbor state s′; i.e., γ * V(s′).

Note In computer science, a problem that can be divided into subproblems that produce an overall optimal solution (such as using Bellman’s Principle) is said to have optimal substructure. To see how to calculate this equation, let’s assume a discount factor g = 0.9 and the MDP shown on a prior figure; we can calculate the value of the leftmost state (Wash Face) as follows: V('Wash Face') = 1 + 0.9 * (0.7 * V('Have Breakfast') + 0.3 * V('Get Dressed')) Notice that if we were to set g = 0 then the values associated with each state would match its reward. To fully compute V(s), for all s, we would need to solve n equations in n unknowns, considering n is the number of states in the MRP. In classical planning, we created a plan that was either an ordered list of actions or a partially ordered set of actions (we discussed it in prior chapters) meant to be executed without reference to the state of the environment. In an MDP, the assumption is that you could potentially go from any state to any other state in one step. And so, to be prepared, it is typical to compute a whole policy rather than a simple plan. 641

Chapter 17

Reinforcement Learning

A policy is a mapping from states to actions that defines a course of action or sequence of actions that the agent will follow in the environment. It’s usually denoted by the Greek letter pi: π(s). Because of the Markov property, we’ll find that the choice of action only needs to be dependent on the current state (and possibly the current time) and not on any of the previous states. We’ll try to find the policy that maximizes, for each state, the expected reward of executing the policy in that state. We will call such a policy an optimal policy and denote it as π*(s). A policy can be deterministic and output a single action for each state or stochastic and output an action dependent on various probabilities.

Note Since a policy is a sequence of actions, when you take an MDP and fix a policy then all actions have been chosen and what you have left is a Markov chain. The state–value function V. at follows policy π in an MDP is the expected return starting from state s and then following policy π: Vp ( s ) = R p ( s ) + g *

N (s)

å T [ s , s ¢] * V ( s ¢ )

s ¢ÎN ( s )

p

p

An optimal state–value function is the maximum value function over all policies, as follows: Vp* ( s ) = max Vp ( s ) p

The action–value function Q(s,a), or simply Q-function, is the expected return starting from state s, taking action a, and then following policy π, as follows: Qp ( s , a ) = R ( s, a ) + g * 642

N (s)

å T [ s , s ¢] * V ( s ¢ )

s ¢ÎN ( s )

a p

p

Chapter 17

Reinforcement Learning

Note that Q(s, a) can be expressed in terms of V(s) and that it considers not only states but also actions leading to states.

Note The Q-function represents the quality of a certain action given a state. An optimal action–value function is the maximum action–value function over all policies, as follows: Q*p ( s, a ) = max Qp ( s, a ) p

What would be the goal of an RL agent? Its goal should be to learn an optimal policy by optimizing either V(s) or Q(s, a); it has been proven that all optimal policies achieve the optimal state–value and action–value functions, as follows: Vp* ( s ) = Q*p ( s, p ( s ) ) = V* = Q* where V*,Q* represent the optimal values of V(s) and Q(s, a) respectively. Thus, it would seem logical to try to optimize one of these functions to obtain an optimal policy for the agent. Remember that this is our main goal in MDP and specifically in RL. If the reward and transition values are known to the agent, then he can use a model-based algorithm known as value iteration to calculate V* and obtain an optimal policy. Another approach for obtaining an optimal policy and solving MDPs is the policy iteration algorithm. This is also a model-based method that manipulates the policy directly rather than finding it indirectly via the optimal value function. As occurs with the value iteration method, it assumes the agent is aware of the reward and transition functions.

643

Chapter 17

Reinforcement Learning

Later, we will discuss Q-learning, a model-free learning method that can be used in situations where the agent initially knows only that certain states and actions are possible but is unaware of the transition and reward probability functions. In Q-learning the agent improves its behavior by learning from the history of interactions with the environment. It only discovers that there is a reward for going from one state to another via a given action when it does so and receives a reward. Similarly, it only figures out what transitions are available from a given state by ending up in that state and looking at its options. If state transitions are stochastic, it learns the probability of transitioning between states by observing how frequently different transitions occur.

Note In a model-based method, the agent has a built-in model (reward and transition functions) of the environment and therefore can simulate it so as to find the right decision. In a model-free method, the agent knows how to act, but doesn’t explicitly know anything about the environment.

Value Iteration Algorithm In value iteration we will compute V*(s) for all states s by applying an iterative procedure in which our current approximation for V*(s) gets closer to the optimal value over time. We start by initializing V(s) to 0 for all states. We could actually initialize to any values we want, but it’s

644

Chapter 17

Reinforcement Learning

easiest to just start at 0. This algorithm uses the updating rule for V(s); a pseudocode of the method is shown in the following lines:

A common stopping condition for this problem is having a change in value from step t to step t + 1 less than or equal to a predefined epsilon multiplied by a discount factor variable, as shown in the previous pseudocode. In this case, δ represents the maximum change of V(s) in some iteration. V and V′ represent utility vectors and ε the maximum error allowed in the utility of a state. This algorithm converges to the correct utilities over time.

645

Chapter 17

Reinforcement Learning

Policy Iteration Algorithm In the policy iteration algorithm we search for optimal policy and utility values at the same time; thus, we manipulate the policy directly rather than finding it indirectly via the optimal value function. A pseudocode of the algorithm is shown in the following lines:

646

Chapter 17

Reinforcement Learning

where V is the utility vector and π. presents the policy outputted by the algorithm, initialized with random values. The PolicyEvaluation() subroutine solves the following: system of linear equations: R ( si ) + g * maxa

N ( si )

å T éë s , p ( s ) , s¢ùû * V ¢ ( s¢)

s ¢ÎN ( si )

i

i

PI picks an initial policy, usually just by taking rewards on states as their utilities and computing a policy according to the maximum expected utility principle. Then, it iteratively performs two steps: value determination, which calculates the utility of each state given the current policy, and policy improvement, which updates the current policy if any improvement is possible. The algorithm terminates when the policy stabilizes. Policy iteration often converges in a few iterations, but each iteration is expensive; recall the method has to solve large systems of linear equations.

Q-Learning & Temporal Difference The value iteration and policy iteration algorithms work perfectly for determining an optimal policy, but they assume our agent has a great deal of problem-specific knowledge. Specifically, they assume the agent accurately knows the transition function and the reward for all states in the environment. This is actually quite a bit of information; in many cases, our agent may not have access to this. Fortunately, there is a way to learn this information. In essence, we can trade learning time for a priori knowledge. One way to do this is through a form of reinforcement learning known as Q-learning. Q-learning is a form of model-free learning, meaning that an agent does not need to have any model of the environment; it only needs to know what states exist and 647

Chapter 17

Reinforcement Learning

what actions are possible in each state. The way this works is as follows: we assign each state an estimated value, called a Q value. When we visit a state and receive a reward, we use this to update our estimate of the value of that state. (Since our rewards might be stochastic, we may need to visit a state many times.) Considering that V * (s) = max Q(s,a¢) , we can rewrite the previously a¢ detailed formula for Q(s, a) only in terms of the Q function. Qp ( s, a ) = R ( s, a ) + g *

N (s)

å T [ s, s¢] * Q ( s¢, a )

s ¢ÎN ( s )

a p

p

The previous formula is the update rule used in the Q-learning algorithm, described in the following lines:

For Q-learning to converge we must guarantee that every state is visited infinitely often; one cannot learn from that which it does not experience, and therefore it must infinitely visit every state in order to guarantee convergence and find an optimal policy. 648

Chapter 17

Reinforcement Learning

Q-learning belongs to a class of methods known as temporal difference algorithms. In a temporal difference algorithm (TDA) we learn by reducing the difference between estimates at different time frames (t, t′). Q-learning is a particular case of TDA where we reduce the estimate of Q for a state and its consecutive states, also known as neighbors or successors. We could just as well design an algorithm that reduces discrepancies between this state and more distant descendants or ancestors. The most popular TD algorithm is probably TD(λ) (Sutton 1988), a general version of TDA that relies on the idea that we can calculate Q as follows: Q n ( s t , a t ) = rt + g * rt +1 + ¼+ g n -1 * rt + n -1 + g n * max Q ( st + n ,a ) a

Notice in the previous formulation that we do not only include a onestep lookahead as we did in Q-learning, but rather we are considering n steps into the future. TD(λ) mixes various lookahead distances using a 0 £ l £ 1 parameter in the following manner: Ql ( s t , a t ) = (1 - l ) * éëQ1 ( s t , a t ) + l * Q 2 ( s t , a t ) + l 2 * Q3 ( s t , a t ) + ¼ùû When considering l = 0 we end up with the Q-learning rule, the one where we simply look one step ahead. As we increase λ, he algorithm places more emphasis on discrepancies based on moredistant lookaheads. When we reach the value l = 1, only the observed rt + i values are considered, with no contribution from the current Q estimate value. The motivation for the TD(λ) method is that in some settings training will be more efficient if more-distant lookaheads are considered.

649

Chapter 17

Reinforcement Learning

ractical Problem: Solving aMaze Using P Q-Learning In this practical problem we will demonstrate the application of the Q-learning method through a very simple and intuitive situation: solving a maze. In the maze, the agent starts at cell (0, 0) and must find a way out at cell (n- 1, m- 1) where n represents the number of rows and m the number of columns in a zero index–based matrix. Figure17-4 illustrates the maze to be solved in this chapter.

Figure 17-4. Maze to be solved Notice how in the previous maze there are several policies the agent can follow to reach the exit cell, but there’s only one optimal policy (Figure17-5). Because learning will occur over time (as occurs in real life) we must guarantee a continuous visit of every state (cell) in each episode; this is the necessary condition for Q-learning to converge. An episode is how we’ll refer to an agent’s completing the maze, and whenever the maze is completed we’ll say that the agent will move from episode E to episode E + 1. 650

Chapter 17

Reinforcement Learning

Figure 17-5. Optimal policy followed by the agent to solve the maze The Q-learning agent, which we will call Qagent, is represented by the class shown in Listing 17-1.

Listing 17-1. Properties, Fields, and Constructor of the QAgent Class public class QAgent { public int X { get; set; } public int Y { get; set; } public Dictionary QTable { get; set; } public double Randomness { get; set; } public double[,] Reward { get; set; } private readonly bool[,] _map; private readonly int _n; private readonly int _m; 651

Chapter 17

Reinforcement Learning

private readonly double _discountFactor; private static readonly Random Random = new Random(); private readonly Dictionary _freq; public QAgent(int x, int y, double discountFactor, int n, int m, double [,] reward, bool [,] map, double randomness) { X = x; Y = y; Randomness = randomness; InitQTable(n, m); _n = n; _m = m; Reward = reward; _map = map; _discountFactor = discountFactor; _freq = new Dictionary {{new Tuple(0, 0), 1}}; } } This class contains the following properties or fields:

652

•

X: represents the row of the agent’s position on the board

•

Y: represents the column of the agent’s position on the board

Chapter 17

Reinforcement Learning

•

QTable: matrix representing the Q function in tabular form, i.e., the Q(s, a) function where rows indicate states and columns indicate actions. It’s coded as a dictionary of Tuple (states) and a list of four (actions up, down, left, right) double values for each tuple.

•

Randomness: Because from time to time we need to wander around to try to get the agent to visit every state, we use the Randomness variable to indicate a value in the range [0; 1] corresponding to the chance of generating a random action.

•

Reward: represents the reward matrix for every state

•

_ map: variable that represents the map of the environment (maze)

•

_n: number of rows in the environment

•

_m: number of columns in the environment

•

_discountFactor: discount factor as previously detailed and used in the Q-learning update rule

•

_freq: dictionary detailing the frequency of visit of every state; it will be used in the strategy applied to guarantee the agent visits every state infinitely often and seeking to obtain an optimal policy

The InitQTable() method (Listing 17-2) included in the class constructor was created with the purpose of initializing the QTable; i.e., the dictionary of (state, {actionUp, actionDown, actionLeft, actionRight}) entries. At the beginning it will be that Q(s, a) = 0 for every possible action a.

653

Chapter 17

Reinforcement Learning

Listing 17-2. InitQTable() Method private void InitQTable(int n, int m) { QTable = new Dictionary(); for (var i = 0; i < n; i++) { for (var j = 0; j < m; j++) QTable.Add(new Tuple(i, j), new List { 0, 0, 0, 0}); } } The Q-learning process occurs in the following method (Listing 17-3); the actionByFreq parameter will determine if we use the strategy of visiting states by frequency + randomness or if we will rely only on Q values to complete the maze. Since every learning process requires some time, we will need to rely merely on the frequency + randomness strategy to try to “learn”—i.e., visit every state frequently enough to learn from these experiences and be able to learn in the end an optimal policy that would lead us to the exit of the maze in the shortest time and in the shortest number of steps.

Listing 17-3. InitQTable() Method public void QLearning(bool actionByFreq = false) { var currentState = new Tuple(X, Y); var action = SelectAction(actionByFreq); if (!_freq.ContainsKey(ActionToTuple(action))) _freq.Add(ActionToTuple(action), 1);

654

Chapter 17

Reinforcement Learning

else _freq[ActionToTuple(action)]++; ActionToTuple(action, true); var reward = Reward[currentState.Item1, currentState.Item2]; QTable[currentState][(int) action] = reward + _discountFactor * QTable[new Tuple(X, Y)].Max(); } The very important action-selection strategy that will lead the agent into learning an optimal policy is coded in the SelectAction() method shown in Listing 17-4. In case the actionByFreq variable has been activated (set to True), the agent will perform an action according to a frequency + randomness strategy; otherwise, it will always choose the Q(s', a) with the highest value.

Listing 17-4. SelectAction() Method private { var var var

QAgentAction SelectAction(bool actionByFreq) bestValue = double.MinValue; bestAction = QAgentAction.None; availableActions = AvailableActions();

if (actionByFreq) return FreqStrategy(availableActions); for (var i = 0; i < 4; i++) { if (!availableActions.Contains(Action Selector(i))) continue;

655

Chapter 17

Reinforcement Learning

var value = QTable[new Tuple(X, Y)][i]; if (value > bestValue) { bestAction = ActionSelector(i); bestValue = value; } } return bestAction; } The previous method uses the FreqStrategy() method seen in Listing 17-5. In this method, we apply a random action with probability 0.5 or a frequency-based visit; i.e., visit the adjacent state least visited according to the _freq dictionary.

Listing 17-5. FreqStrategy() Method private QAgentAction FreqStrategy(List availableActions) { var newPos = availableActions.Select(availableAction => ActionToTuple(availableAction)).ToList(); var lowest = double.MaxValue; var i = 0; var bestIndex = 0; if (Random.NextDouble() = 0 && _map[X, Y- 1]) result.Add(QAgentAction.Left); 657

Chapter 17

Reinforcement Learning

if (Y + 1 < _m && _map[X, Y + 1]) result.Add(QAgentAction.Right); return result; } We adopted the convention of matching actions in the order {up, down, left, right} with integers starting from 0; hence, up = 0, down = 1, left = 2, right = 3. The ActionSelector() method shown in Listing 17-7 mutates an integer into its equivalent action (we’ll soon see the QAgentAction enum). In Listing 17-7 we can also see the ActionToTuple() method, which converts a QAgentAction into a Tuple representing the resulting state after executing that action.

Listing 17-7. ActionSelector() and ActionToTuple() Methods public QAgentAction ActionSelector(int action) { switch (action) { case 0: return QAgentAction.Up; case 1: return QAgentAction.Down; case 2: return QAgentAction.Left; case 3: return QAgentAction.Right; default: return QAgentAction.None; } }

658

Chapter 17

Reinforcement Learning

public Tuple ActionToTuple(QAgentAction action, bool execute = false) { switch (action) { case QAgentAction.Up: if (execute) X--; return new Tuple(X- 1, Y); case QAgentAction.Down: if (execute) X++; return new Tuple(X + 1, Y); case QAgentAction.Left: if (execute) Y--; return new Tuple(X, Y- 1); case QAgentAction.Right: if (execute) Y++; return new Tuple(X, Y + 1); default: return new Tuple(-1, -1); } } To conclude the QAgent class, we add the Reset() method (Listing 17-8), which resets or prepares the agent for a new episode by setting it to the start position and cleaning the _frequency dictionary. The QAgentAction enum describing possible agent actions is shown in Listing 17-8.

Listing 17-8. Reset() Method and QAgentAction Enum public void Reset() { X = 0; Y = 0; 659

Chapter 17

Reinforcement Learning

_freq.Clear(); } public enum QAgentAction { Up, Down, Left, Right, None } We already presented the machine learning code of the program, but we are missing a component: the GUI on Windows Forms. The inheritor of the Form class that will visually represent the maze is MazeGui, illustrated in Listing 17-9. Remember that we are coding a Windows Forms application.

Listing 17-9. Fields and Constructor from MazeGui Class public partial class MazeGui : Form { private readonly int _n; private readonly int _m; private readonly bool[,] _map; private readonly QAgent _agent; private Stopwatch _stopWatch; private int _episode; public MazeGui(int n, int m, bool [,] map, double [,] reward) { InitializeComponent(); timer.Interval = 100; _n = n; _m = m; _map = map; _agent = new QAgent(0, 0, 0.9, _n, _m, reward, map, .5); 660

Chapter 17

Reinforcement Learning

_stopWatch = new Stopwatch(); } } The class contains the following properties or fields: •

_n: number of rows in the maze

•

_m: number of columns in the maze

•

_map: matrix with Boolean values indicating whether a cell is a wall or not

•

_agent: instance of the QAgent class

•

_stopWatch: stopwatch used to measure the time taken in every episode of the Q-learning process

•

_episode: number of episodes carried out so far in the Q-learning process

To draw all elements on the maze, we implement the Paint event for the drawing control (Picture Box) as shown in Listing 17-10.

Listing 17-10. Paint Event of the Picture Box Representing the Maze private void MazeBoardPaint(object sender, PaintEventArgs e) { var pen = new Pen(Color.Wheat); var cellWidth = mazeBoard.Width / _n; var cellHeight = mazeBoard.Height / _m; for (var i = 0; i < _n; i++) e.Graphics.DrawLine(pen, new Point(i * cellWidth, 0), new Point(i * cellWidth, i * cellWidth + mazeBoard.Height));

661

Chapter 17

Reinforcement Learning

for (var i = 0; i < _m; i++) e.Graphics.DrawLine(pen, new Point(0, i * cell Height), new Point(i * cellHeight + mazeBoard.Width, i * cellHeight)); for (var i = 0; i < _map.GetLength(0); i++) { for (var j = 0; j < _map.GetLength(1); j++) { if (!_map[i, j]) e.Graphics.FillRectangle(new Solid Brush(Color.LightGray), j * cellWidth, i * cellHeight, cellWidth, cellHeight); } } for (var i = 0; i < _map.GetLength(0); i++) { for (var j = 0; j < _map.GetLength(1); j++) { if (_map[i, j]) e.Graphics.DrawString(String.Format("{0:0.00}", _agent.QTable[new Tuple(i, j)][0]. ToString(CultureInfo.GetCultureInfo ("en-US"))) + "," + String.Format("{0:0.00}", _agent.QTable[new Tuple(i, j)][1].ToString(CultureInfo. GetCultureInfo("en-US"))) + "," + String.Format("{0:0.00}", _agent.QTable[new Tuple(i, j)][2].ToString (CultureInfo.GetCultureInfo("en-US"))) + "," +

662

Chapter 17

Reinforcement Learning

String.Format("{0:0.00}", _agent.QTable[new Tuple(i, j)][3].ToString(CultureInfo. GetCultureInfo("en-US"))) ,new Font("Arial", 8, FontStyle.Bold), new SolidBrush(Color.White), j * cellWidth, i * cellHeight); } } e.Graphics.FillEllipse(new SolidBrush(Color. Tomato), _agent.Y * cellWidth, _agent.X * cellHeight, cellWidth, cellHeight); e.Graphics.DrawString("Exit", new Font("Arial", 12, FontStyle.Bold), new SolidBrush(Color.Yellow), (_m- 1) * cellWidth + 15, (_n- 1) * cellHeight + 15); } We will draw the agent as an ellipse and the walls as gray cells; we will also draw four values on each walkable cell: the values Q(s, a) for state s and all possible actions. To get and execute an action from the agent we included a timer that triggers every second and calls upon the QLearning() method of the agent using the frequency + randomness strategy while the current episode is less than 20. It’s also in the method that handles the tick event (Listing 17-11) that we reset the stopWatch and the agent’s state and write the episode elapsed time in a file.

Note When in a goal state s, we do not apply the Q-learning rule to update Q(s, a); on the contrary, we take the reward value of the goal state and assign it directly to Q(s, a).

663

Chapter 17

Reinforcement Learning

Finally, we refresh the mazeBoard to show the new set of changes to the GUI.

Listing 17-11. Method Handling the Tick Event private void TimerTick(object sender, EventArgs e) { if (!_stopWatch.IsRunning) _stopWatch.Start(); if (_agent.X != _n- 1 || _agent.Y != _m- 1) _agent.QLearning(_episode < 20); else { _agent.QTable[new Tuple (_n- 1, _m- 1)] = new List { _agent.Reward [_n- 1, _m_agent.Reward [_n- 1, _m_agent.Reward [_n- 1, _m_agent.Reward [_n- 1, _m}; _stopWatch.Stop(); _agent.Reset();

1], 1], 1], 1]

var file = new StreamWriter("E:/time_difference.txt", true); file.WriteLine(_stopWatch.ElapsedMilliseconds); file.Close();

664

Chapter 17

Reinforcement Learning

_stopWatch.Reset(); _episode++; } mazeBoard.Refresh(); } } Now that we have all components in place, let’s try to test the application and run it, as we have done throughout this book, in a console application, creating the necessary map and reward matrixes (Listing 17-12).

Listing 17-12. Testing the MazeGui Application var map = new [,] { {true, {true, {true, {true, {true, };

false, true, false, true}, true, true, false, true}, false, true, false, true}, false, true, true, true}, true, true, false, true}

var reward = new [,] { {-0.01, -0.01, -0.01, -0.01, -0.01}, {-0.01, -0.01, -0.01, -0.01, -0.01}, {-0.01, -0.01, -0.01, -0.01, -0.01}, {-0.01, -0.01, -0.01, -0.01, -0.01}, {-0.01, -0.01, -0.01, -0.01, 1}, }; Application.EnableVisualStyles(); Application.SetCompatibleTextRenderingDefault(false); Application.Run(new MazeGui(5, 5, map, reward)); 665

Chapter 17

Reinforcement Learning

The result obtained after executing the code from Listing 17-12 would be an instance of the Windows Forms application developed throughout this chapter (Figure17-6). The reward function contains a reward of 1 for the goal state and -0.01 for any other state. Once the agent has completed the first episode the goal state (Exit) will contain reward 1 for every action; i.e., Q('Exit', {up, down, left, right}) = 1.

Figure 17-6. Episode 2, the QAgent is learning and updating Q values, which are shown in the upper-left corner of every cell Using the exploration strategy previously described (the one where we mix frequency of visited cells and randomness for executing actions), we continuously visit each state in each episode. After 20 episodes have been completed, the agent starts taking actions that rely only on the Q values learned and always executing the action that corresponds to Q(s', a) with the highest value. In this case, we were able to find the optimal policy, which was detailed in Figure17-5.

666

Chapter 17

Reinforcement Learning

Figure17-7 illustrates the values ultimately calculated for Q(s, a) and after 20 episodes have passed. The reader can check that a path starting at cell (0, 0) and choosing always the action (remember they appeared in the order up, down, left, right) with the highest Q value will lead it to the optimal policy—the one leading to the Exit (goal state) in the least number of steps.

Figure 17-7. Optimal policy found and executed by the agent Recall that our goal in Q-learning is to actually learn the Q function, Q(s, a). In this case, we learn the function in its tabular form, which has states as rows and actions as columns of a table or matrix. In some scenarios it might be intractable to do it this way, given the fact that we may have a large state space. In such a scenario, we can rely on a function approximator such as neural networks to approximate the Q function. This is actually the approach used by Tesauro in its popular backgammon agent, capable of defeating the backgammon world champion of its time.

667

Chapter 17

Reinforcement Learning

Summary In this chapter, we described the interesting topic of reinforcement learning (RL), one of the most important machine learning paradigms along supervised and unsupervised Learning. We began by defining Markov decision processes (MDPs), the mathematical framework used in RL to model problems of the real world. We described the value function (V) and the action–value function (Q) and demonstrated the relationship between these and their importance in obtaining an optimal policy. The concept of policy was also included in the chapter. We provided several methods for solving MDPs. Namely, we detailed the value iteration and policy iteration algorithms. In the end, we discussed Q-learning and implemented a practical problem where we used it to get an agent to learn how to exit a maze in the shortest number of steps.

668

Index A A Beautiful Mind, 590 Activision-Blizzard (Call of Duty), 549 Adversarial search agents, 593 board-games analysis, 593 constant-sum game, 595 deterministic game environment, 594 imperfect information game, 595 Lee Sedol vs. AlphaGo, 594 methods, game types, 595 Minimax search algorithm (see Minimax search algorithm) perfect information game, 595 stochastic game environment, 594 utility function, 594 zero-sum game, 595 Agent architectures deliberative architecture alternatives generation, 122 BDI architecture (see Beliefs, Desires, and Intentions (BDI) architecture) © Arnaldo Pérez Castaño 2018 A. Pérez Castaño, Practical Artificial Intelligence, https://doi.org/10.1007/978-1-4842-3357-3

diagram, 120 filtering, 122 goal-based behavior, 119 logical reasoning, 119 means-end reasoning, 121 planning component, 121–122 practical reasoning, 121 problems, 119 hybrid architecture goal-based component, 127 horizontal and vertical layering, 128–130 mediator function, 130 reactive and deliberative components, 128 InteRRaP, 133–134 properties, 113 reactive architecture (subsumption) behavior-based, 115 Brooks’ architecture, 115 characteristics, 116 cleaning agent, 115–118 diagram, 114 principle, 116 reactive agent, 114 touring machines, 131–132 669

Index

Agent Communication Language (ACL), 251 FIPA (see Foundation for Intelligent Physical Agents (FIPA)) KQML, 204–207 Agents actuators, 92 autonomy, 95 bots, 92 cleaning robot advantages, 110 AgentAction(List percepts), 108–109 cleaning agent, 110–113 constructor and fields, 103 List, 103 loop, agent function, 104 methods Clean(), IsDirty(), MoveAvailable(int x, int y), and Print() methods, 105 Percepts enum and Perceived() method, 106 UpdateState() method, 107 definition, 92–93 environments, 94 accessible, 101 continuous, 101 decision-making process, 99 deterministic, 100 discrete, 101 dynamic, 101 episodic, 102 inaccessible, 101 670

non-deterministics, 100–101 static, 101 fundamental AI entities, 91 intelligent, 93–94 proactive, 97 proactiveness, 95 properties of, 98–99 purely reactive, 95 rationality, 95 reactive agent advantages, 96 cleaning robot, 96 decision-making process, 95–96 disadvantages, 97 reactivity, 95 sensors, 92 social ability, 95 state-based, 102–103 Agglomerative clustering, 484, 486 Airport simulation Airplane class, 298–299 AirplaneEvtArrival class, 302–303 AirplaneEvtBreakdown class, 305 AirplaneEvtProcess Cargo class, 303–304 AirportEvent abstract class, 299 methods, 302 properties, 301 arrivals, time and lambda parameter, 297

Index

console application, 312 constructor, fields and properties of Simulation class, 305–307 Execute() method, 307–309 initialize and test simulation, 311 passengers, cargo, 297–298 RunwayAvailable() and TryToLand() methods, 310 Alpha-Beta pruning branches/subtrees, 600 Minimax tree, 599–600 optimal child, 602 pseudocode, 600 Application programming interface (API), 221 Artificial immune systems (AISs), 522 Atomic propositions, 4 Automated theorem proving (ATP), 40 applications, 42 automation, 43 binary decision tree, 45 Boolean values, 44 classical application, 43 description, 42 flow diagram, 42 hardware verification, 43 logical language, 43 proof assistant, 43 proof checking, 44 proof generation, 45

SATisfiability (SAT), 44 software verification, 43 string-matching algorithms, 42 Average linkage clustering, 485

B Backpropagation algorithm ActivationValue and ErrorTerm properties, 454 chain rule, 442 classification-related methods, 455 classification vector, 456 CreateLayers() method, 448 flow backward, 445 FunctionDerivative() method, 451, 452 gradient descent search method, 440, 459 handwritten digit recognition, 459 hyperbolic tangent and ReLU units, 444, 453–454 List of SigmoidUnit, 448 MultiLayerNetwork and Layer classes, 446–448 Predict() method, 455 PredictSet() method, 448 ReLU function, 445 ReturnIndexByHalf() and ReturnIndexByMax() methods, 455 SigmoidUnit class, 449 671

Index

Backpropagation algorithm (cont.) stochastic approximation, 441–442 training data set, 441 Training() method, 450, 451 TrainingSample class, 457–458 UpdateWeight() method, 454–455 weight-update formulas, 443 Basic geostatical area (BGA), 542 BDT, see Binary decision t ree (BDT) Bee colony (BC), 522 Beliefs, Desires, and Intentions (BDI) architecture agent’s action function, 127 beliefs, 124 bold agent, 125 cautious agent, 125 components, 126–127 desires, 124 diagram, 123 intentions, 124–125 practical reasoning, 126 Best First Search, 580–581 Bidirectional search (BS) simultaneous searches, 559 Sliding Tiles Puzzle (see Sliding Tiles Puzzle) Binary comparer circuit, 19, 21 Binary decision tree (BDT) advantages, 30 AI data structure, 27 conditions, 26 672

constructors and properties, 27–28 decision-making processes, 30 leaf and non-leaf nodes, 27 recursive structure, 28 static methods, 29 varIndex variable, 30 visual representation, 30 Breadth-first search (BFS), 142, 157–158 Bfs class, 561–562 derivations, 559 graph-based search algorithms, 556 procedure, 557 time and space complexity, 556

C C4.5 algorithm binary decision node, 397–398 error reduction, 394–396 Gain() method, 405–406 GainContinuous() method, 403–404 gain ratio, 397 GainRatio() and SplitInformation(), 404–405 handling continuous attributes, 393, 397 handling missing values, 393, 398

Index

HighestGainAttribute() method, 401, 403 implementation, 399 main body, 399–401 overfitting, 393–394 pruning process, cross-validation, 394 rule pruning, 394, 396 SubsetEntropy() method, 406, 407 testing, console application, 408–410 validation set, training data, 395 Canonical hyperplane, 323 Centroid linkage clustering, 485 Clause class, 46–47 methods, 48 Cleaning agents, 249, 288 CleaningAgentPlatform class, 254–255 CleaningTask class, 251–252 fields/properties, 253 methods, 253 Contract Net, 256–261 static methods, 261 FipaAcl class, 262–265 methods, 266–267 GUI, Room class, 280–282 MasCleaningAgent class Action() method, 278–279 Bid() method, 274–276 fields, properties, and constructor, 267–268 methods, 280

propertiesand fields, 269–270 ReactionTimeOnTick() method, 272–274 Run() method, 271 SetSocialLaw() method, 276–278 program structure, 250–251 running application agents exchange messages, Contract Net, 284–285 console application, 283–284 InitCommunicationService() method, 284 “Task Finished” message, 286–287 Cleaning robot CleaningRobot C# class, 83–87 creation, 82 features, 82 grid, 82 predicates and functions, 83 print() method, 87 Start() method, 87 terrain, 88 Clustering algorithms family, 483 applications, 481 compactness, 482 criterion/objective function, 481 definition, 480 Euclidean distance, 482 673

Index

Clustering (cont.) hierarchical (see Hierarchical clustering) isolation/separation, 483 Manhattan distance, 482 Minkowski distance, 482 object color, 480–481 optimization, 481 partitional algorithms (see K-means algorithm) similarity measure, 481 Cnf class, 48–50 DPLL algorithm, 52 Formula hierarchy, 52, 53 Literals() method, 53, 54 methods, 51 RemoveParenthesis(And and) method, 52 Common Language Runtime (CLR), 222 Communication ACL (see Agent Communication Language (ACL)) blackboard systems, 200–201 classification, agents, 199–200 message passing, 201 Speech Act Theory, 201 WCF (see Windows Communication Foundation (WCF)) Complete linkage clustering, 485 Compound propositions, 4 Conjunction logical connective, 8

674

Conjunctive normal form (CNF), 5, 17 And class with ToCnf() method override, 38 DISTRIBUTE-CNF, 36 function, 36 Or class with ToCnf() method override, 38–39 ToCnf() and DistributeCnf() methods, 37 ToCnf() method override, Not and Variable classes, 39 transformation algorithm, 36 variables, 36 Contract Net announcement, 215 awarding, 215 bidding, 215 contractors, 215, 217 expediting, 215 FIPA-ACL specification, 218 manager, 215, 217 process, 215, 216 Coordination and cooperation approaches, 212 basic strategy, 213 benevolent, 212 coherence, 211 Contract Net (see Contract Net)) decision making, 211 description, 211–212 designing, 212

Index

interests of individuals, organizations, companies, 212 possibilities, 213 results sharing, 214 social norms and societies, 218–219 Subscribe/Notify pattern, 214 task sharing problem decomposition, 213 solution synthesis, 214 subproblem solution, 214 Crytek (Far Cry), 549

D Data contract, 224 Data mining, 367 Davis-Putnam-LogemannLoveland (DPLL) algorithm auxiliary methods Dpll(Cnfcnf ) method, 57–59 OneLiteral(Cnfcnf ) method, 59–62 PureLiteralRule() method, 62–65 Split() method, 65, 67 binary decision tree, 55 CNF formula, 55 definition, 55 heuristics and metaheuristics, 67 OneLiteral, 56–57 pseudocode, 55–56

PureLiteral, 56–57 SAT problem, 55 Split, 56–57 tree construction, 67 Decision tree (DT) attributes and values, 369 data classification, 371 DecisionTree class, 380–382 methods, 383 properties, 382 definition, 368 generation Hunt’s algorithm, 372 ID3 algorithm, 373 training data set size, 373–374 leaf nodes, 369 multiple internal nodes, 369 partition, 368 root node, 369 SVM/neural networks, 369 training data set, 370–371 Depth-first search (DFS) backtracks, 557 derivations, 559 Dfs class, 562–563 DLS, 559 graph-based search algorithm, 557 infinite paths, 558 procedure, 558 time and space complexity, 558 visited nodes, 557 675

Index

Depth-limited search (DLS) DFS, 559 Dls class, 563–565 Digital information flow, 19 Dirichlet’s Box Principle, 67 Discreteet-event simulation (DES), 290–291, 313 events, 293 knowledge, 292 objects, 292 probability and statistics, 293 properties, 292 queues, 293 resources, 293 time, 292–293 Disjunction logical connective, 9 Disjunctive normal form (DNF), 17 DPLL algorithm, see DavisPutnam-LogemannLoveland (DPLL) algorithm)

E Entropy definition, 375 function, 375 ID3 algorithm, 376 Equivalence logical connective, 11–12 Estimation of Distribution algorithms (EDAs), 522 Evolutionary algorithms (EAs), 522 Extension methods, 387–391 676

F Fault contract, 224 First-order logic (FOL) components, 77 evaluation, 79 formula, 77 interpretation, 79–80 predicates, 75 Dog class, 80 filter and get objects, 81 property, 76 propositional logic, 76 quantifiers, 78 requirement, 76 rules of interpretation, 78 syntax of, 77 FOL, see First-order logic (FOL) Foundation for Intelligent Physical Agents (FIPA) components, 207 inform performative, 210 parameters, 208 performatives, 208–209 request performative, 210 structure of, 207

G Game programming AI methods, 550 development, 549 disciplines specific, 550 economic impact, 549

Index

informed search A* search, 582–583 Best First Search, 580–581 Greedy Best First Search, 581 search procedures features, 555 information usage, 555 Sliding Tiles Puzzle, 553–555 systematicity, 555 uninformed search (see Uninformed search algorithms) video game AI game development, 552 companies, 549 design phase, 552 game engine, 552 Halo Series, 551 software development, 551 Game theory A Beautiful Mind, 590 adversarial search (see Adversarial search) applications in sociology and psychology, 590 definition, 591 identification of process participants, 591 mathematical framework, 589 Nobel Prize in Economics, 590 Othello (see Othello game) popular games, 591 Prisoner’s Dilemma (PD), 591–592

two-person zero-sum, 589 Gaussian kernel, 348 Genetic algorithms (GA), 523 Gradient descent search (GDS), 428–431, 459 Greedy Best First Search, 581–582

H Handwritten digit recognition (HDR) classification, handwritten digits, 476–477 Classify button, 473–474 extract features from image, 471–472 handwriting, 463 HandwrittenDigit RecognitionNn class, 467 HandwrittenRecognition Gui class, 468–469 low-resolution images, 462 Mouse-Event methods, 470–471 multi-layer NN hidden layers, 465 image pixels, 465 initialization of weights, 467 structure of, 466 training data, 466 OCR applications, 463 physiological/behavioural characteristics, 462 ReadWeights() method, 474–475 677

Index

Handwritten digit recognition (HDR) (cont.) testing, 476 training data set, 464 universe of characters, 463 visual application, 476 weights, training data set, 472–473 Windows Forms application, 467 Hessian matrix, 333, 336, 341 Heuristics features, 511 Mars Rover, 509 NP-Hard problems, 509 problem-independent iterative process, 510 Sliding Tiles Puzzle, 511 speed-up process, 510 Hierarchical clustering agglomerative, 484, 486 divisive, 484 measures, 485 Hill climbing method diversification, 523 intensification, 523 Execute() Method, 519, 521 GA, 523 genetic manner, 523 InitialSolution(), Neighborhood(), and NSpherePoints() methods, 518–519 local optimum, 513–514 678

Local Search (LS), 515, 522 MathParserNuget package, 516 mutation operator, 524 n-sphere surrounding, 515–516 optimization methods, 525 parabolic function, 521 properties/fields, 517 pseudocode of algorithm, 514 public property, 516 selection, mutation and crossover methods, 525 testing, 521 TSP (see Traveling Salesman Problem (TSP) types, 512–513 Hunt’s algorithm, 372–373

I, J Implication logical connective, 10–11 Incremental gradient descent, 431–432 Information gain calculation, 377 definition, 377 formula, 376 Inheritance and C# operators abstract Formula class, 22 AND class, 23 BinaryGate class, 22 creating and evaluating formula, 25

Index

Or, Not, and Variable classes, 23–25 result, executing code, 26 structural recursion, 21 Integration of Rational Reactive behavior and Planning (InteRRaP), 133–134 Intelligent agent, 93–94 Interactive Dichotomizer 3 (ID3) algorithm, 373, 383–386 attributes and training data set, 377–378 fields and properties, 379 attribute-splitting test, 373 DecisionTree class, 380–382 console application, 391–393 methods, 383 properties, 382–383 entropy, 373 extension methods, 387–390 implementation, 377 information gain, 373 tree splitting, 377 Iterative deepening search (IDS), 559, 565–566, 568

properties, 493 data points and centroids, 487–489 description, 486–487 Element class, 493 Euclidean distance, 488 execution of, 499 initialization phase, 487 inner and outer loops, 487 isolated data points, 490 KMeans and DataSet classes, 494–496 properties and fields, 497 pseudocode, 488 SSE, 488, 499 testing, 497 unsupervised learning method, 487 Knowledge Interchange Format (KIF), 204 Knowledge Query and Manipulation Language (KQML), 204–207

K

L

Karush-Kuhn-Tucker (KKT) conditions, 349–350 K-means algorithm centroids, 487, 490 Cluster class, 490–492 methods, 493

Lagrange multipliers, 325–326 Lagrangian method, 325 Laws of Propositional Logic, 12, 14–16 Least Mean Square (LMS), 427 Linear regression, 316 679

Index

Linear SVM classifying hyperplane, 343 console application, 341 GetIndicesFromValues() method, 336 Predict() method, 336–337 properties and fields, 328–330 SetInitValue() method, extension class, 335–336 SvmGui Windows Forms class, 337–341 Training() method, dualoptimization problem, 330–333 UpdateWeightVector() and UpdateBias() methods, 334 Local Search (LS) algorithms, 515, 522 Logic circuits (see Logic circuits) computational, 2 definition, 2 DPLL, 1 fundamental, 1 philosophers, 1 Logical connectives conjunction, 8 disjunction, 9 equivalence, 11–12 implication, 10–11 negation, 7 symbols, 6 unary/binary functions, 6 Logical gate, 18–19 680

Logic circuits binary comparer, 19, 21 bivalent functions, 17 computer, 18 conjunction component (AND), 20 conjunction gates, 19 disjunction component (OR), 20 electronic component, 19 information flows, 18 logical gate, 18–19 negation component (NOT), 19

M Manhattan Distance, 585 Markov decision processes (MDPs), 634, 668 decision-making process, 639 discount factor, 638 discrete state–time transition system, 636 horizons, 639 infinite horizons, 639, 640 Markov chain, 639 MRP, 639 optimization problem, 638 probability distribution, 639 reward types, 638 robot mouse, 636 sum of rewards, 638 transition probabilities, 637 working day of android, 640

Index

Markov Property, 637 Markov reward process (MRP), 639 Mars Rover architecture BDI layer, 141 beliefs, 142 BFS, 142 deliberation process, 142 heuristics, 143 hybrid, 140 layers, 141 path-finding algorithms, 142 planning layer, 142 reactive layer, 141 relative frequency, 142 Total Relative Frequency, 142–143 BDI, 137, 140 classic rovers, 139 coding Action() method, 162–165 BDI classes, 148–152 beliefs, desires, percepts, plans and actions, 152 BFS algorithm, 157–158 Brf() method, 169–170 ExistsPlan() and ExecuteAction() methods, 175–176 fields, variables, and constructor, 143–146 Filter() and ChoosePlan() methods, 174

FulFill() method, 158 GetCurrentTerrain() method, 160–161 GetPercepts() method, 158–159 InjectBelief() method, 165 InjectBelief(), SetRelativeFreq(), and RelativeFreq() methods, 167–169 Manhattan Distance, 173 Mars class, 147–148 MoveAvailable() and LookAround() methods, 159 Options() method, 172 Percept and Plan classes, 153–157 RandomMove() method, 166 UpdateBelief() method, 171–172 UpdatePerceivedCellsDicc() and CheckTerrain() methods, 161–162 definition, 138 diagram, 139 Earth, 139 movement, 139 obstacles, 140 space agencies, 138 space exploration, 137 spirit and opportunity, 138 visual application (Visual application, Mars Rover) 681

Index

MasCleaningAgent class, 267–268 Action() method, 278–279 Bid() method, 274–276 methods, 280 properties and fields, 269–270 ReactionTimeOnTick() method, 272–274 Run() method, 271 SetSocialLaw() method, 276–278 Message contract, 224 Minimax search algorithm Alpha-Beta pruning (see AlphaBeta pruning) backtracking, 597 description, 596 end-game position, 597 evaluation procedure, 599 execution, 598 game types, 596 pseudocode, 598 search tree, 597 utility value, 597 zero-sum games, 596 Misplaced Tiles, 583–584 Multi-agent organizations description, 196 flat/democracy, 198 hierarchical, 197 hybrids, 198 modular, 198 subsumption, 198 Multi-agent systems (MAS) agent architecture, 196 682

air traffic control scenario, 194–195 autonomous, 196 cleaning agent, 193 coalition, 195 communication (see Communication) definition, 194 discrete, 196 distributed artificial intelligence, 193 efficiency, 196 flexibility, 197 modularity, 196 multi-agent organization, 196–198 platform, 195 problem solving, 196 real-world applications, 193 reliability, 197 reusability, 196 strategy, 195 Multi-layer networks deep learning, 438 deep neural networks, 438 hidden layers, 439 layers, 435–436 layers and power, 438 sigmoid function, 437 underfitting, 439 XOR function, 437 Multi-objective clustering inter-class distance, 500 intra-class distance, 500

Index

MOPs, 500–501 non-dominated vector, 500 objective function, 500 Pareto Frontier Builder (see Pareto Frontier Builder) Pareto optimal, 500–501 zoning, 499 Multi-objective optimization problems (MOPs), 500–501

N Negation logical connective, 7 Negation normal form (NNF), 16 function, 31 Nnf() override Not class, 33–34 Variable class, 35 ToNnf() abstract method Formula abstract class, 32 And, Or classes, 32–33 transformation algorithm, 31 Nerve cells, 18 Neural networks (NNs) activation function, 413 Adaline and GDS, 427–430 Adaline class, 432–435 artificial, 461 artificial intelligence, 411 biological neuron, 413 electrochemical signals, 412 face recognition, 461 graph, 414

HDR (see Handwritten digit recognition (HDR)) iterative processes, 411 learning process, 461 multi-layer, 412, 435–439 neuron, 412 Perceptron algorithm (see Perceptron algorithm) single-neuron networks, 414 stochastic approximation, 431–432 training data set, 461 Neurons, 18 Normal forms, 16–17

O Offline character recognition, 463 One-vs-All classification (OVA), 364 Online character recognition, 463 Operation contract, 224 Optical character recognition (OCR), 461, 463 Osuna’s theorem, 349–351 Othello game 8 x 8 board, 602–603 creator, Goro Hasegawa, 602 heuristics corner closeness, 606 corner occupancy, 605 mobility, 606 piece difference, 605 range of values, 606 utility value, 606–607 683

Index

Othello game (cont.) imaginary arrangement, 604 initial configuration, 603 Minimax class GetOptimalMove() and Execute() methods, 630–631 properties and fields, 629 white pieces, 604–605 Windows Forms AvailableMoves AroundPiece() and SetPieceCreatedBoard() methods, 612–617 CheckUpDown(), CheckLftRgt(), CheckDiagonal(), UpdateFlips(), SetPiece(), FlipPieces(), and UpdatePiecePos() methods, 617–621 development, 629 EmptyCell(), Expand(), AvailableMoves() and IsLegalMove() methods, 610–611 handling paint and mouseclick events, 625–626 HeuristicUtility(), 623 OthelloBoard class, 607–609 OthelloGui class, 623–624 UpdateBoardGui() and AiPlayTimerTick() methods, 626–628 UtilityValue, 622 684

P Pareto Frontier Builder bi-objective optimization, 501 description, 501 functions, 501 iterations, step values, 505–506 linkage mechanism, 503 stages, 502–503 strategy, 502–503 variations, 503–504 Pareto Frontier Linkage, 503 Pareto optimal, 500–501 Particle swarm optimization (PSO), 522 Perceptron algorithm activation function, 415 class constructor, 422–423 console application, 425–426 data set, 426 dot product, 419 equation of line, 416 fields/properties, 422 hyperplanes, 416, 423 learning process, 417 learning rate, 420 learning rule, 419 linear classifier, 416 pseudocode, 418–419 setting random values, 417 SingleNeuralNetwork abstract class, 420–422

Index

fields and properties, 422 methods, 422–423 Perceptron class, 423–424 training data set, 417 training/learning process, 417 weight vector and bias, 426 Pigeonhole Principle, 67–68, 74 Policy iteration algorithm, 646–647 Polynomial kernel, 347 Prism, 596 Prisoner’s Dilemma (PD), 591–592 Proof assistant, 43 Proof checking, 44 Proof generation, 45 Propositional logic ATP, 75 compound, 4 CNF, 5 contradiction/unsatisfiable, 5 definition, 3 examples, 3 formula (p ˅ q) ˄ (p ˅ ¬q) ˄ (¬p ˅ q) ˄ (¬p ˅ ¬r), 71–72 formula (p ˅ q ˅ ¬r) ˄ (p ˅ q ˅ r) ˄ (p ˅ ¬q) ˄ ¬p, 72 formula (p ˅ q ˅ r) ˄ (p ˅ q ˅ ¬r) ˄ (p ˅ ¬q ˅ r) ˄ (p ˅ ¬q ˅ ¬r) ˄ (¬p ˅ q ˅ r) ˄ (¬p ˅ q ˅ ¬r) ˄ (¬p ˅ ¬q ˅ r), 73 interpretation, 5 Name property to Variable class and ToString() overrides for Variable, Not, And, Or, and Cnf classes, 69–70

Pigeonhole Principle, 67–68, 74 simple/atomic, 4 syntax of, 5 tautology/logic law, 5 Propositional variables, 4 Pruning process error reduction pseudocode, 395 subtree replacement, 396 rule pruning attribute tests, 397 steps, 396

Q Q-learning, 634 agent, behavior, 644 model-free learning, 644, 647 optimal policy, 648 problem-specific knowledge, 647 Q value, 648 solving maze ActionSelector() and ActionToTuple() methods, 658 AvailableActions() method, 657 fields and constructor, MazeGui class, 660–661 FreqStrategy() method, 656–657 InitQTable() method, 653–654 685

Index

optimal policy, 643 optimal state–value function, 642 optimal values, 643 policy, 642 policy iteration, 646–647 Principle of Optimality, 641 Q-learning (see Q-learning) Reward Hypothesis, 635 state–value function, 642 tabular values, 640 TDA, 649 trial-and-error, 634 value function, 641 value iteration, 644–645

Q-learning (cont.) method handling, tick event, 663–665 neural networks, 667 optimal policy, 650–651, 667 Paint event of Picture Box, 661–663 QAgent and Q values, 666 QAgent class, 651–653 Reset() method and QAgentAction enum, 659 SelectAction() method, 655–656 testing, MazeGui application, 665 state transitions, 644

R Reinforcement learning (RL) action–value function, 642 basic flow, 635 Bellman’s Equation, 641 classical planning, 641 components, 635 definition, 634 development, 635 Dynamic Programming Equation, 641 machine learning paradigm, 634 MDPs (see Markov decision processes (MDPs)) optimal action–value function, 643 686

S SATisfiability (SAT), 1, 44, 55 Scatter search (SS), 522 Sequential minimal optimization (SMO) All vs. All, 365 bias, 355 classifying hyperplane, 364 clipped value, 353 description, 348 ExamineExample() and TakeStep() methods, 357–358 KKT conditions, 349–350 Lagrange multipliers, constraints, 351–352 learning rate, 355

Index

learning/update rule, 353–355 LFunctionValue() and Kernel. Polynomial() methods, 361–362 LFunctionValue() method, bias and weight vector, 362–363 linear constraint, 352–354 multi-class, 364–365 Osuna’s theorem, 349–351 OVA, 364 TakeStep() method, 358–361 TrainingBySmo() method, LinearSvmClassifier class, 356–357 training data, 351 Service contract, 224 Service-oriented application (SOA), 222–223 Simple Object Access Protocol (SOAP), 223 Simple propositions, 4 Simulation airport (see Airport simulation) analytic approach, 290–291 definition, 290 DES (see Discreteet-event simulation (DES)) flexibility, 290 modeling classification, 289 definition, 289 features, 289 probabilistic distributions discrete random variable, 294

exponential distribution, 295 normal distribution, 296 parameter μ, 294 parameter σ2, 294 Poisson process, 294–295 Single-linkage clustering, 485 Sliding Tiles Puzzle, 553–555 A* search algorithm cost of shortest path, 584 Linear Conflict, 585 Linear Conflict + Manhattan Distance heuristic, 586 Manhattan Distance, 585 Misplaced Tiles, 583–584 pattern database, 586–587 tree structure, 583 AI search methods, board game, 553, 568 Board classes, 569–573 Bs class, 575–577 Expand() method, 574 forward and backward searches, 577–578 hardest 8-puzzle configuration, 578–579 IEqualityComparer interface, 575 Move() method, 574 node generation, 574 path variable, 574 reverse of swap operation, 568 SolutionMet() method, 577–578 states and trees, 553–555 687

Index

S-metaheuristics algorithms adaptive memory, 539 BGA, 542 clustering-related problem, 547 diversification, 539 HC-related components, 540 hom*ogeneity, 542, 545 iteration, 539 iterative methods, 538 k data-set elements, 544 medium-and long-term memories, 540–541 multi-objective optimization problem, 542, 543 Pareto Frontier Builder, 543, 545–546 Tabu List data structure, 544 Toluca Valley, 544 zoning problem, 543–544 Social commitment, 218 Social norms/laws, 218–219 Speech Act Theory, 201 Stochastic gradient descent (SGD), 431–432 Subtree replacement, 396 Sum of Squared Errors (SSE), 488, 499 Supervised learning classifier, 316–317 dataset, 633 image, properties, 633 linear regression, 316 phases, 315 prediction, 315–316 regressor, 316–317 688

tabular data, 633 training data, 315–316 Support vector machines (SVMs) classifiers and regressors, 318 duality, 325 generalized Lagrangian, 326 hyperplane bias/intercept, 321 canonical, 323 classes, 321 classification, 319–320, 323, 325 constraints, 323–324 normalization, 322 optimization problem, 324 support vectors, 321–323 weight vector, 321–322 Lagrange multipliers, 325–326 Lagrangian method, 325, 345 linear (see Linear SVM) non-linear case data mapping from 2D to 3D space, 346 feature mapping, 346 Gaussian kernel, 348 hyperplane, 346 kernel function, 347, 348 polynomial feature mapping, 347 polynomial kernel, 347 quadratic problem, 347 training data, 345 optimal classifying hyperplane, 343–344 optimization problem, 318, 327

Index

reformulation, training data, 344 SMO (see Sequential minimal optimization (SMO)) soft-margin and hard-margin, 345 text-classification tasks, 319

T Tabu Search (TS), see S-metaheuristics algorithms) Take-Two Interactive (NBA2K series), 549 Temporal difference algorithm (TDA), 649 Temporal difference (TD), 634 Theory of Games and Economic Behavior, 589 Touring machines, 131–132 Traveling Salesman Problem (TSP) biological process, 526 Canonic property, 531, 533 chromosome/solution encoding, 528 crossover operator, 531 GeneticAlgorithmTsp class, 533–537 InitialPopulation() method, 534 NewPopulation() method, 534 NP-Hard problem, 527 OffSprings() method, 534 problem-specific issues, 528

Solution class, 529–531, 533 US map, 526–527 Two-person zero-sum games, 589

U Ubisoft (Assassin’s Creed), 549 Uninformed search algorithms BFS, 556, 558–559 blind search, 556 BS, 559 DFS, 557–559 DLS, 559 Execute() lines, results, 568 Graph Theory toolbox, 556 IDS, 559 testing, console application, 566–567 Tree class, 560–561 UninformedMethod abstract class, 561 Unsupervised learning data structure, 633 methods, 479–480

V Value iteration algorithm, 644–645 Visual application, Mars Rover, 137 actual water location, 189–190 diversification, 187–188 explore–exploit tradeoff, 188 intensification phase, 188 lower-left corner, water location, 189 689

Index

Visual application, Mars Rover (cont.) new water-location belief, 191 plan (sequence of actions), 185 SenseRadius parameter, 184 set up, Mars Rover and world, 182–183 WanderThreshold, 191 water-location belief, 185–186, 191 water-location belief and obstacle-location belief, 183–184, 186–187 Windows Form, 176–181, 184

W, X, Y, Z Windows Communication Foundation (WCF) agents adding, WCF service, 232 AgentCommunication Service class, 237 Callback Contract implementation, 239–240 class and interface, 232 client application, 243–245 Client UI in Windows Forms, 245 console application, 241–242 create WCF service, 232 exchanging messages, 247 executing service and clients, 246

690

implementation process, 231 lock statement, 237 Proxy implementation, 238 Publisher/Subscriber pattern, 231 Send() method, 237 service and callback contracts, 232–233 service implementation, 234–237 Subscriber() method, 237 synchronization context, 240 UpdatedListEventArgs, 240–241 API, 221 bindings, 227–228 CLR types, 222 contracts description, 224 Duplex pattern, 225 IHelloWorld service, 227 One-Way pattern, 225 Request–Response pattern, 225 service implementation, 226–227 types, 224 endpoints, 229 .NET Framework, 221 network application, 222 Publisher/Subscriber pattern, 221, 230–231 services, 222–223