Index

As this ebook edition doesn't have fixed pagination, the page numbers below are hyperlinked for reference only, based on the printed edition of this book.

Symbols

_get_exch_socket 77

__init__ constructor 85

_parse_message function 96

_process_lmodel_upload function 79

_process_polling function 80

_push_cluster_models function 82, 83

_push_local_models function 82

_send_cluster_models_to_all function 81, 82

A

Act on the Protection of Personal Information (APPI) 8

add_agent function 89, 90

Advanced Driver Assistance Systems (ADASs) 244

adversarial agents 172

aggregation, using coordinate-wise median 173

aggregation, using geometric median 172, 173

aggregation, using Krum algorithm 173, 174

protecting, against 166, 167

agent-side local retraining FL cycle and process 54

aggregate_local_models function 91, 92

aggregation 56

coordinate-wise median, using 173

criteria, checking 87

geometric median, using 172, 173

Krum algorithm, using 173, 174

reviewing 158, 159

aggregation code (aggregation.py) 71

aggregation, for non-ideal cases

adversarial agents 172

heterogeneous computational power, handling 167, 168

modifying 167

non-IID datasets 174, 175

aggregator

class, defining and initializing 91

libraries, importing 90

messages, handling from 95, 96

software components 70

aggregator-side codes 71

aggregation code (aggregation.py) 71

FL server code (server_th.py) 71

FL state manager (state_manager.py) 71

aggregator-side FL cycle and process 50

local ML models, accepting 50, 51

local ML models, aggregating 52, 53

local ML models, caching 50, 51

aggregator, software components

aggregator-side codes 71

configuration 72

lib/util codes 71

AI bias 11

algorithm bias 11

Amazon Web Services (AWS) 2, 131

anti-money laundering (AML) 32, 235, 236

demo of FL 239

proposed solution, to existing approach 236-238

application programming interfaces (APIs) 41, 106

asynchronous FL 49

B

Bank Secrecy Act (BSA) 235

big data

aspects 4

at present 3

definition 2

nature 1, 2

training 11

training, challenges 10

training cost 10, 11

training, impact 9

big data, aspects

abundance of observations 4

acceptance of messiness 4

ambivalence of causality 5

big data, characteristics

variety 2

velocity 2

volume 2

big data ML system

versus FL system 14

black box modeling 20

Blockchain Based Aggregator Free Federated Learning (BAFFLE) 258

reference link 258

blockchain technology

reference link 34

buffer_local_models 87, 88

business intelligence 10

C

California Consumer Privacy Act (CCPA) 7, 255

Centers for Disease Control and Prevention (CDC) 12

Central Data Repositories (CDRs) 234

centralized FL 256

centralized ML

limitation 254

chief information officers (CIOs) 1

Chord Distributed Hash Table (Chord DHT) 61

CIFAR-10

Flower, integrating 221, 222

IBM FL, integrating 220, 221

OpenFL, integrating 218-220

STADLE, integrating 222, 223

CIFAR-10 dataset

preparing 150, 151

skewing 217, 218

classes, for defining types of ML models and messages

AgentMsgType class 267

AggMsgType class 267

DBMsgType class 266

ModelType class 266

clear-box model 20

clear_saved_models function 88, 89

client libraries

integrating, into image classification (IC) 126

cluster aggregator 38, 39

FL server module 39

FL state manager 40

model aggregation module 40

collective intelligence (CI) 258

intelligence-centric era with 259

Internet of Intelligence 260

reference link 259

Colorado Privacy Act (CPA) 8

communication handler 269

libraries, importing for 269

communication handler, functions 270

init_client_server function 271

init_db_server function 270

init_fl_server function 270

receive function 272

send function 271

send_websocket function 272

computational power distributions 165, 166

computer vision (CV) model 11

concept drift 12

Consumer Data Protection Act (CDPA) 8

Continuous Delivery for Machine Learning (CD4ML) 13

continuous FL cycle and process 47, 48

continuous integration/continuous delivery (CI/CD) 13

convert_LDict_to_Dict function 273

Convolutional Neural Networks (CNNs) 103

coordinate-wise median

using, for aggregation 173

cross-device FL 32 257

cross-silo FL 32 257

crowdsourced learning 14, 260

with FL 260, 261

D

data accessibility, challenges

solutions 228

data accessibility, problem

components 226

database server 41

agent information 42

aggregator information 41

base model information 42

cluster global models 42

configuration 93

database 41

data, pushing to 96

defining 93

implementing 93

libraries, importing for pseudo database 94

message, parsing 96, 97

messages, handling from aggregator 95, 96

performance data 42

PseudoDB class, defining 94

PseudoDB, initializing 94, 95

running 93, 99

data-centric platform 259

data drift 12

DataInterface

creating 198-201

implementing 197, 198

data isolation problem 235

data lakes 3

data loader

creating 191

data minimalism 9

data privacy 5

as bottleneck 5, 6

increased data protection regulations 6

risks, in handling 6

versus data security 5

Data Protection Impact Assessment (DPIA) 8

data protection officer (DPO) 8

data security

versus data privacy 6

dataset distributions 161, 162

data-sharing approach 178

data structure handler 272

libraries, importing for 272

decentralized FL 257

decentralized FL system 34, 35

deep learning (DL) 2, 22, 23, 227

deep neural network (DNN) 22

development-operations (DevOps) 13

Devron

URL 255

differential privacy (DP) 33

distributed agent 38-40

FL client module 40

local ML engine and data pipelines 41

distributed agent-side code 104

client.py file 105

lib/util code 105

distributed computing 23, 24

benefits 25

e-commerce example 24, 25

distributed learning

for privacy 255

for training efficiency 255

distributed ML 26

double descent 23

drug discovery 230

E

edge AI 26

edge computing 26, 241

FL, applying to 240

with IoT over 5G 240, 241

Edge FL example, object detection

results, examining 243

technical settings 242

working 242, 243

edge inference 26-28

edge training 28

Electronic Health Records (EHRs) 226, 233

representation learning 235

Electronic Medical Records (EMRs) 233

exch_socket

obtaining, to push global model back to agents 77

F

feature-based FL 32, 256

FedAvg, dataset distributions

IID case 162, 163

non-IID Case 164, 165

FedAvg function 92

FedCurv 175

implementing 175-177

federated Ai ecosystem (FATE) 32

federated averaging (FedAvg) 29, 56, 57, 71, 159-161

applying 59

example 58, 59

computational power distributions 165, 166

dataset distributions 161, 162

protecting, against adversarial agents 166, 167

federated learning (FL) 3, 9, 29, 103, 183 225

applying, to distributed learning for big data 250, 251

applying, to edge computing 240

applying, to financial sector 235

applying, to healthcare sector 226

applying, to robotics 245-247

as solutions, for data problems 14, 15

autonomous driving 243-245

benefits 254

benefits, for risk detection systems 240

considerations 32

crowdsourced learning with 260, 261

defining 29

demo, in AML space 239

developments projects 256

enhanced distributed learning frameworks with 257

in Web 3.0 249

research 256

types and approaches, exploring 256

federated learning (FL), system

configuring 130

configuring, with JSON files 131

data and database folders 136-138

database, with SQLite 138

environment, installing 130, 131

image classification (IC), integrating into 123-126

internal libraries 264, 265

local ML engine, integrating into 119

running 130

security 32-34

versus big data ML system 14

versus traditional big data ML system 15

Federated Machine Learning

reference link 256

Federated Stochastic Gradient Descent (FedSGD) 159

federated training, of NLP model

data loader, creating 191

Flower for SST-2, integrating 208, 209

FL training approach, adopting 192

IBM FL for SST-2, integrating 203

model, training 192

OpenFL, for SST-2 195, 196

sentiment analysis model, defining 189, 190

STADLE for SST-2, integrating 212-214

TensorFlow Federated, integrating for SST-2 193-195

federated transfer learning (FTL) 31

FedML

URL 255

FedProx 168, 169

implementing 169-171

fine-tuned aggregation 100

FL client

minimal example, running 134-136

FL client, libraries

client state, manipulating 116

designing 114

FL client core threads, registering 114

global models, saving 115

global models, waiting from aggregator 118

local models, sending to aggregator 116, 117

FL client module 40

communication handler 40

FL participation handler 40

FL client-side components

configuring 105

distributed agent-side code 104

overview 104

FL client-side functionalities

agent participation 109-111

Client class, defining 107

client process, initializing 107-109

implementing 106

libraries, importing 106

model exchange synchronization 111, 112

polling method, implementing 112

push method, implementing 112

FL frameworks 185

Flower 185

IBM FL 187

OpenFL 186

PySyft 188, 189

STADLE 187, 188

TensorFlow Federated (TFF) 185

Flower 185

integrating, for CIFAR-10 221, 222

URL 185

Flower, for SST-2

client, implementing 210

example, running 212

integrating 208, 209

server, creating 211

FL process 29-31

horizontal FL 32

personalization 31

transfer learning (TL) 31

vertical FL 32

FL server

aggregator, running 133

database, running 133

potential enhancements 100

running 92, 93

FL server code (server_th.py) 71

FL server module 39

communication handler 39

model synthesis routine 40

system configuration handler 39

FL server, potential enhancements

database, redesigning 100

fine-tuned aggregation 100

performance metrics, for local and global models 100

registry, automating of initial model 100

FL server-side

functionalities, implementing 73

FL server-side, functionalities

class, defining 74

cluster models, pushing to database 82, 83

global model synthesis routine 80, 81

initializing 74, 75

libraries, importing 73

local models, pushing to database 82

ML models, pushing to database 83

register function, of agents 75, 76

used, for handling messages from local agents 78, 79

used, for sending global models to agents 81

FL server-side, register agents

agent participation, confirming with updated global model 78

exch_socket, obtaining to push global model back to 77

process, initializing 77

FL state manager 40

FL state manager (state_manager.py) 71

FL system architecture 38

advantages 39

cluster aggregator (or aggregator) 38, 39

database server (or database) 41

distributed agent (or agent) 38-40

intermediate servers, for low computational agent devices 42, 43

FL system flow 43

agent-side local retraining cycle and process 54, 55

aggregator-side FL cycle and process 50

asynchronous FL 49

continuous FL cycle and process 47, 48

initialization processes 44, 45

initial model upload process, by initial agent 46

model interpretation 55

synchronous FL 49

FL training

approach, adopting 192

G

General Data Protection Regulation (GDPR) 7, 241, 255

geometric median

using, for aggregation 172, 173

glass-box model 20

global models 29

_get_exch_socket, used for pushing back to agent 77

initializing 86, 87

performance metrics 100

global model synthesis routine 80, 81

Google Cloud Platform (GCP) 2, 23

H

healthcare providers (HCPs) 233

healthcare sector

challenges 226

FL, applying to 226

Health Insurance Portability and Accountability Act (HIPAA) 255

helper libraries

importing for 274

helper library, functions 274

compatible_data_dict_read function 276

create_data_dict_from_models function 276

create_meta_data_dict function 276

generate_id function 275

generate_model_id function 275

get_ip function 279

init_loop function 279

load_model_file function 278

read_config function 274

read_state function 278

save_model_file function 277

set_config_file function 274, 275

write_state function 279

heterogeneous computational power

automatic adjustment 168, 169

FedProx, implementing 169-171

handling 167, 168

manual adjustment 168

heterogeneous FL 256

homogeneous FL 256

homomorphic encryption (HE) 34

horizontal FL 256

horizontal FL (homogeneous FL) 32

I

IBM FL 185-187

integrating, for CIFAR-10 220, 221

reference link 187

IBM FL, for SST-2

configuration files, defining 204-207

DataHandler, creating 203, 204

example, running 208

integrating 203

party, creating 207, 208

image classification example, with CNN

running 152

running, evaluation 152, 153

image classification (IC)

client libraries, integrating into 126

five agents, running 153-155

integrating, into FL system 123-126

ML model, using for FL 151

results, analyzing 150

running 150

image classification model

federated training, on non-IID data 216, 217

increment_round function 90

Independently and Identically Distributed (IID) 31, 159, 162, 163

Information and Communications Technology (ICT) 4

information privacy 5

initialization processes, FL system flow 44

agent initialization and registration 45, 46

aggregator initialization and registration 45

database server initialization 45

initialize_model_info function 86

initial model upload process, FL system flow 46

input and output (I/O) variables 11

intelligence-centric platform 259

Intelligence from Data (IfD) 228

potential 232, 233

intermediate aggregation 178

intermediate servers

for low computational agent devices 42, 43

internal libraries, FL system 264

communication_handler.py 264

data_struc.py 264

helpers.py 265

messengers.py 265

states.py 264

Internet of Everything 260

Internet of Federated Things (IoFT) 249

Internet of Intelligence 247, 248, 260

Internet of Things (IoT) devices 2, 257 225

InterPlanetary File System (IPFS) 61

J

JSON files, for FL system configuration

config_agent.json file 132, 133

config_aggregator.json file 131, 132

config_db.json file 131

K

Krum algorithm 173

using, for aggregation 173, 174

Kubernetes framework

URL 60

L

Lei Geral de Proteção de Dados Pessoais (LGPD) 8

lib/util codes 71

LimitedDict class 273

functions 273

local agents

model upload, processing by 79

polling, processing by 80

local ML engine

client libraries, integrating into 122, 123

functions, testing 120, 121

integrating, into FL system 119

libraries, importing 119

models, defining 120, 121

models, training 120, 121

local ML engine and data pipelines 41

local models 29

aggregating 90

performance metrics 100

M

machine learning (ML) 38, 103, 183, 227

current state 19

model creation process, automating 21, 22

medical imaging 227

messengers

communication payloads, generating 280

libraries, importing for 280

messengers, functions

generate_ack_message function 285

generate_agent_participation_confirm_message function 284

generate_agent_participation_message function 282, 283

generate_cluster_model_dist_message function 282

generate_db_push_message function 280, 281

generate_lmodel_update_message function 281

generate_polling_message function 285

minimum example

one agent, running 139-144

two agent, running 144-149

ML bias 11

ML, in hospitals

example use case 228, 229

ML model

performance, degradation 11

pushing, to database 83

ML operations (MLOps 13, 251

model 20

training 192

stopping, scenarios 12, 13

model aggregation

basics 56

Federated Averaging (FedAvg) 56-59

performing 56

model aggregation module 40

model bias

impact 9

model-centric platform 259

model decay 11

model drift 11, 12

monitoring 13, 14

types 12

model operations (ModelOps) 12, 251

models for aggregation, maintaining with state manager 84

agents, adding 89, 90

aggregation criteria, checking 87

class, defining 84

FL round, incrementing 90

global model, initializing 86, 87

__init__ constructor 85

libraries, importing 84

local models, buffering 87, 88

saved models, clearing 88, 89

Model Zoo 260

N

National Commission for Data Protection (CNDP) 7

natural language processing (NLP) 235

NLP model

federated training 189

non-IID Case 164, 165

non-IID data

federated training, of image classification model on 216, 217

non-IID datasets 174

data-sharing approach 178

FedCurv, implementing 175-177

personalization, through fine-tuning 178

O

OpenFL 184-186

integrating, for CIFAR-10 218-220

URL 186

OpenFL, for SST-2

DataInterface, implementing 197, 198

example, running 202

FLExperiment, creating 198-200

integrating 195, 196

ShardDescriptor, implementing 196, 197

open source software 2

ordinary least squares (OLS) 21

P

parallelized SGD (pSGD) 160

parameter freezing 31

Performance-Based Neighbor Selection (PENS) algorithm 178

Personal Health Records (PHRs) 233

personalization 31, 178

personally identifiable information (PII) 6

polling method

from agent, to aggregator 113

implementing 112

precision medicine 230-232

private set intersection (PSI) 32

pseudo database (PseudoDB)

class, defining 94

initializing 94, 95

libraries, importing 94

push method

from aggregator, to agent 113

implementing 112

PySyft 188, 189

reference link 188

R

ready_for_local_aggregation 87

receive_msg_from_agent function 78, 79

Reinforcement Learning (RL) 245

Remote Procedure Calls (RPCs) 185

Rivest-Shamir-Adleman (RSA) 32

robotics

FL, applying to 245-247

S

sample-based FL 256

scalability, with horizontal design 60

asynchronous agent participation 62

distributed database 61

semi-global model, creating 60, 61

semi-global model synthesis 63, 64

secure multi-party computation (MPC) 34

Secure Sockets Layer (SSL) 112

semi-global model synthesis 63, 64

sentiment analysis model 189, 190

ShardDescriptor

implementing 196, 197

Smart Contracts (SCs) 34, 258

SplitFed 258

split learning 257, 258

SQLite database

cluster model 139

defining 97

entry, inserting into 98, 99

initializing 98

libraries, importing 97

local models 138

SQLiteDBHandler class

defining 97

initializing 97

Squaring the Net 7

SST-2

TensorFlow Federated (TFF), integrating 193, 194

STADLE 187, 188

integrating, for CIFAR-10 222, 223

URL 187, 255

STADLE client-side library 185

STADLE, for SST-2

example, running 215, 216

integrating 212-214

Ops project, creating 214, 215

state classes, for defining message location

DBPushMsgLocation class 268

GMDistributionMsgLocation class 268

ModelUpMSGLocation class 269

ParticipateConfirmationMSGLocation class 268

ParticipateMSGLocation class 267

PollingMSGLocation class 269

state manager

class, defining 84

initializing 85

libraries, implementing 84

states as enumeration classes, for FL system implementation

client state classes 266

IDPrefix, for defining FL system components 265

libraries, importing to define list of states 265

message location 267

ML models and messages 266

suspicious activity reports (SARs) 235

swarm learning 35, 258

synchronous FL 49

T

TensorFlow Federated (TFF) 184, 185

integrating, for SST-2 193, 194

reference link 185

traditional big data ML system

versus FL system 15

transfer learning (TL) 31

Transport Layer Security (TSL) 112

U

updated global model

agent participation, confirming with 78

V

vertical FL

reference link 256

vertical FL (heterogeneous FL) 32

W

white-box models 20

Y

year-over-year (YoY) 1