J. Ren, H. Wang, T. Hou, et al., Federated learning-based computation offloading optimization in edge computing-supported Internet of things, IEEE Access, 7:69194–69201, June 2019. https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8728285 DOI: 10.1109/access.2019.2919736. 128
C. Nadiger, A. Kumar, and S. Abdelhak, Federated reinforcement learning for fast personalization, In Proc. of IEEE 2nd International Conference on Artificial Intelligence and Knowledge Engineering (AIKE), August 2019. DOI: 10.1109/aike.2019.00031. 128
Carnegie Mellon University, LEAF: A benchmark for federated settings, July 2019. https://leaf.cmu.edu/ and https://github.com/TalwalkarLab/leaf 13
S. Caldas, P. Wu, T. Li, et al., LEAF: A benchmark for federated settings, January 2019. https://arxiv.org/abs/1812.01097 13
T. Li, A. K. Sahu, M. Zaheer, et al., Federated optimization for heterogeneous networks, July 2019. https://arxiv.org/abs/1812.06127 10, 33, 55, 56
D. Song, Decentralized federated learning, June 2019. https://drive.google.com/file/d/1Bk3ldYJcYo405uwATsqC8ZD1_UcLGlRL/view 65
N. Hynes, R. Cheng, and D. Song, Efficient deep learning on multi-source private data, July 2018. https://arxiv.org/abs/1807.06689 65
N. Agarwal, A. T. Suresh, F. Yu, et al., cpSGD: Communication-efficient and differentially-private distributed SGD, May 2018. https://arxiv.org/abs/1805.10559 64
K. Pillutla, S. M. Kakade, and Z. Harchaoui, Robust aggregation for federated learning, May 2019. https://krishnap25.github.io/papers/2019_rfa.pdf 64
L. Melis, C. Song, E. D. Cristofaro, et al., Exploiting unintended feature leakage in collaborative learning, November 2018. https://arxiv.org/abs/1805.04049 DOI: 10.1109/sp.2019.00029. 10, 64
M. Mohri, G. Sivek, and A. T. Suresh, Agnostic federated learning, February 2019. https://arxiv.org/abs/1902.00146 64
Y. Ma, X. Zhu, and J. Hsu, Data poisoning against differentially-private learners: Attacks and defenses, March 2019. https://arxiv.org/abs/1903.09860 DOI: 10.24963/ijcai.2019/657. 64
T. D. Nguyen, S. Marchal, M. Miettinen, et al., DÏoT: A federated self-learning anomaly detection system for IoT, May 2019. https://arxiv.org/abs/1804.07474 65
C. Xie, S. Koyejo, and I. Gupta, Asynchronous federated optimization, May 2019. https://arxiv.org/abs/1903.03934 10, 65
J. Wang and G. Joshi, Adaptive communication strategies to achieve the best error-runtime trade-off in local-update SGD, March 2019. https://arxiv.org/abs/1810.08313 64
V. Zantedeschi, A. Bellet, and M. Tommasi, Fully decentralized joint learning of personalized models and collaboration graphs, June 2019. https://arxiv.org/abs/1901.08460 53, 65
Google Workshop on Federated Learning and Analytics, June 2019. https://sites.google.com/view/federated-learning-2019/home 64, 65
N. Srivastava, G. Hinton, A. Krizhevsky, et al., Dropout: A simple way to prevent neural networks from overfitting, Journal of Machine Learning Research, 15:1929–1958, June 2014. 60
O. Gupta and R. Raskar, Distributed learning of deep neural network over multiple agents, Journal of Network and Computer Applications, 116:1–8, August 2018. https://arxiv.org/abs/1810.06060v1 DOI: 10.1016/j.jnca.2018.05.003. 7
A. W. Trask, Grokking Deep Learning, Manning Publications, February 2019. 1, 33
P. Vepakomma, O. Gupta, T. Swedish, et al., Split learning for health: Distributed deep learning without sharing raw patient data, In ICLR Workshop on AI for Social Good, May 2019. https://splitlearning.github.io/ 7, 8
T.-Y. Liu, W. Chen, and T. Wang, Distributed machine learning: Foundations, trends, and practices, In Proc. of the 26th International Conference on World Wide Web Companion (WWW Companion), April 2017. DOI: 10.1145/3041021.3051099. 33
E. Hesamifard, H. Takabi, and M. Ghasemi, CryptoDL: Deep neural networks over encrypted data, ArXiv Preprint ArXiv:1711.05189, November 2017. https://arxiv.org/abs/1711.05189 29, 90
R. Thibaux and M. I. Jordan, Hierarchical beta processes and the Indian buffet process, In Proc. of 11th International Workshop on Artificial Intelligence and Statistics, pp. 564–571, 2007. 110
M. Yurochkin, M. Agarwal, S. Ghosh, et al., Bayesian nonparametric federated learning of neural networks, ArXiv Preprint ArXiv:1905.12022, May 2019. https://arxiv.org/abs/1905.12022 110
A. G. Roy, S. Siddiqui, S. Pölsterl, et al., Braintorrent: A peer-to-peer environment for decentralized federated learning, ArXiv Preprint ArXiv:1905.06731, May 2019. https://arxiv.org/abs/1905.06731 110
M. J. Sheller, G. A. Reina, B. Edwards, et al., Multi-institutional deep learning modeling without sharing patient data: A feasibility study on brain tumor segmentation, In International MICCAI Brainlesion Workshop, pp. 92–104, Springer, 2018. DOI: 10.1007/978-3-030-11723-8_9. 11, 110
S. Silva, B. Gutman, E. Romero, et al., Federated learning in distributed medical databases: Meta-analysis of large-scale subcortical brain data, ArXiv Preprint ArXiv:1801.08553, October 2018. https://arxiv.org/abs/1810.08553 DOI: 10.1109/isbi.2019.8759317. 110
I. Augenstein, S. Ruder, and A. Søgaard, Multi-task learning of pairwise sequence classification tasks over disparate label spaces, ArXiv Preprint ArXiv:1802.09913, February 2018. https://arxiv.org/abs/1802.09913 DOI: 10.18653/v1/n18-1172. 114
X. Chen and C. Cardie, Multinomial adversarial networks for multi-domain text classification, ArXiv Preprint ArXiv:1802.05694, February 2018. https://arxiv.org/abs/1802.05694 DOI: 10.18653/v1/n18-1111. 114
K. Cho, B. van Merriënboer, C. Gulcehre, et al., Learning phrase representations using RNN encoder-decoder for statistical machine translation, ArXiv Preprint ArXiv:1406.1078, June 2014. https://arxiv.org/abs/1406.1078 DOI: 10.3115/v1/d14-1179. 112
S. Hochreiter and J. Schmidhuber, Long short-term memory, Neural Computation, 9(8):1735–1780, November 1997. DOI: 10.1162/neco.1997.9.8.1735. 112
S. Ji, S. Pan, G. Long, et al., Learning private neural language modeling with attentive aggregation, ArXiv Preprint ArXiv:1812.07108, December 2018. https://arxiv.org/abs/1812.07108 DOI: 10.1109/ijcnn.2019.8852464. 114
D. Leroy, A. Coucke, T. Lavril, et al., Federated learning for keyword spotting, In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6341–6345, 2019. DOI: 10.1109/icassp.2019.8683546. 113
H. B. McMahan, D. Ramage, K. Talwar, et al., Learning differentially private recurrent language models, ArXiv Preprint ArXiv:1710.06963, October 2017. https://arxiv.org/abs/1710.06963 32
S. Ruder and B. Plank, Strong baselines for neural semi-supervised learning under domain shift, ArXiv Preprint ArXiv:1804.09530, April 2018. https://arxiv.org/abs/1804.09530 DOI: 10.18653/v1/p18-1096.
S. Ruder, I. Vulic, and A. Søgaard, A survey of cross-lingual word embedding models, ArXiv Preprint ArXiv:1706.04902, June 2017. https://arxiv.org/abs/1706.04902 DOI: 10.1613/jair.1.11640. 114
S. Zhang, L. Yao, A. Sun, et al., Deep learning based recommender system: A survey and new perspectives, ACM Computing Surveys, 52(1):5:1–5:38, 2019. DOI: 10.1145/3285029. 115
G. Adomavicius and A. Tuzhilin, Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions, IEEE Transactions on Knowledge and Data Engineering, 17(6):734–749, 2005. DOI: 10.1109/tkde.2005.99. 116
Y. Zhou, D. M. Wilkinson, R. Schreiber, et al., Large-scale parallel collaborative filtering for the netflix prize, In Proc. of 4th International Conference Algorithmic Aspects in Information and Management (AAIM), pp. 337–348, June 2018. DOI: 10.1007/978-3-540-68880-8_32. 116
E. Kharitonov, Federated online learning to rank with eution strategies, In Proc. of the 12th ACM International Conference on Web Search and Data Mining, (9):249–257, February 2019. DOI: 10.1145/3289600.3290968. 118
M. Ammad-ud-din, E. Ivannikova, S. A. Khan, et al., Federated collaborative filtering for privacy-preserving personalized recommendation system, ArXiv Preprint ArXiv:1901.09888, January 2019. http://arxiv.org/abs/1901.09888 11, 117
J. Trienes, A. T. Cano and D. Hiemstra, Recommending users: Whom to follow on federated social networks, ArXiv Preprint ArXiv:1811.09292, November 2018. http://arxiv.org/abs/1811.09292 118
H. Mao, Z. Zhang, Z. Xiao, et al., Modelling the dynamic joint policy of teammates with attention multi-agent DDPG, In Proc. of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pp. 1108–1116, July 2019. 130
J. Foerster, I. A. Assael, N. de Freitas, et al., Learning to communicate with deep multi-agent reinforcement learning, In Advances in Neural Information Processing Systems, pp. 2137–2145, 2016. 130
G. Barth-Maron, M. W. Hoffman, D. Budden, et al., Distributed distributional deterministic policy gradients, ArXiv Preprint ArXiv:1804.08617, April 2018. http://arxiv.org/abs/1804.08617
L. Espeholt, H. Soyer, R. Munos, et al., Impala: Scalable distributed deep-RL with importance weighted actor-learner architectures, ArXiv Preprint ArXiv:1802.01561, February 2018. http://arxiv.org/abs/1802.01561
R. M. Kretchmar, Parallel reinforcement learning, In The 6th World Conference on Systemics, Cybernetics, and Informatics, 2002. 127
M. Grounds and D. Kudenko, Parallel reinforcement learning with linear function approximation, In Proc. of the 5th, 6th, and 7th European Conference on Adaptive and Learning Agents and Multi-agent Systems: Adaptation and Multi-agent Learning, pp. 60–74, Springer-Verlag, 2008. DOI: 10.1007/978-3-540-77949-0_5. 127
V. Mnih, K. Kavukcuoglu, D. Silver, et al., Human-level control through deep reinforcement learning, Nature, 518:529–533, February 2015. DOI: 10.1038/nature14236.
B. Liu, L. Wang, M. Liu, et al., Lifelong federated reinforcement learning: A learning architecture for navigation in cloud robotic systems, ArXiv Preprint ArXiv:1901.06455, January 2019. http://arxiv.org/abs/1901.06455 DOI: 10.1109/lra.2019.2931179. 128
H. Zhu and Y. Jin, Multi-objective eutionary federated learning, ArXiv Preprint ArXiv:1812.07478v2, December 2018. http://arxiv.org/abs/1812.07478v2 47
R. Cramer, I. Damgård, and J. B. Nielsen, Multiparty computation from threshold homomorphic encryption, B. Pfitzmann, Ed., EUROCRYPT 2001 (LNCS), 2045:280–299, Springer, Heidelberg, 2001. DOI: 10.1007/3-540-44987-6_18. 22
M. Keller, E. Orsini, and P. Scholl, Mascot: Faster malicious arithmetic secure computation with oblivious transfer, In Proc. of the ACM SIGSAC Conference on Computer and Communications Security (CSS), pp. 830–842, October 2016. DOI: 10.1145/2976749.2978357. 22, 23, 25, 48
M. Keller, V. Pastro, and D. Rotaru, Overdrive: Making SPDZ great again, In J.B. Nielsen and V. Rijmen, Eds., Advances in Cryptology—EUROCRYPT, pp. 158–189, Cham, Springer International Publishing, 2018. DOI: 10.1007/978-3-319-78372-7. 25, 48
I. Damgård, D. Escudero, T. Frederiksen, et al., New primitives for actively-secure MPC over rings with applications to private machine learning, IACR Cryptology ePrint Archive, 2019. DOI: 10.1109/sp.2019.00078. 26
M. Abadi, A. Chu, I. Goodfellow, et al., Deep learning with differential privacy, In Proc. of the ACM SIGSAC Conference on Computer and Communications Security, pp. 308–318, 2016. DOI: 10.1145/2976749.2978318. 32, 43, 51
Y. Aono, T. Hayashi, L. Trieu Phong, et al., Scalable and secure logistic regression via homomorphic encryption, In Proc. of the 6th ACM Conference on Data and Application Security and Privacy, pp. 142–144, 2016. DOI: 10.1145/2857705.2857731. 43, 88
H. Bae, J. Jang, D. Jung, et al., Security and privacy issues in deep learning, December 2018. https://arxiv.org/abs/1807.11655
K. Bonawitz, V. Ivanov, B. Kreuter, et al., Practical secure aggregation for federated learning on user-held data, ArXiv Preprint ArXiv:1611.04482, November 2016. http://arxiv.org/abs/1611.04482 44, 48, 89
K. Chaudhuri and C. Monteleoni, Privacy-preserving logistic regression, In Advances in neural information processing systems, pp. 289–296, 2009. 42
W. Du and Z. Zhan, Building decision tree classifier on private data, In Proc. of the IEEE International Conference on Privacy, Security and Data Mining-Ume 14, pp. 1–8, Australian Computer Society, Inc., 2002. 41
C. Dwork, Differential privacy: A survey of results, In Theory and Applications of Models of Computation, 5th International Conference, TAMC, Proceedings, pp. 1–19, Xi’an, China, April 2008. DOI: 10.1007/978-3-540-79228-4_1. 42
C. Dwork, A firm foundation for private data analysis, Communications of the ACM, 54(1):86–95, 2011. DOI: 10.1145/1866739.1866758. 43
C. Dwork and A. Roth, The algorithmic foundations of differential privacy, Foundations and Trends® in Theoretical Computer Science, 9(3–4):211–407, 2014. DOI: 10.1561/0400000042. 31
W. Fang and B. Yang, Privacy preserving decision tree learning over vertically partitioned data, In IEEE International Conference on Computer Science and Software Engineering, 3:1049–1052, 2008. DOI: 10.1109/csse.2008.731. 10, 41
S. E. Fienberg, W. J. Fulp, A. B. Slavkovic, et al., Secure log-linear and logistic regression analysis of distributed databases, In Proc. of International Conference on Privacy in Statistical Databases, pp. 277–290, Springer, 2006. DOI: 10.1007/11930242_24. 43
G. Jagannathan, K. Pillaipakkamnatt, and R. N. Wright, A practical differentially private random decision tree classifier, In Proc. of IEEE International Conference on Data Mining Workshops, pp. 114–121, 2009. DOI: 10.1109/icdmw.2009.93. 42
X. Lin, C. Clifton, and M. Zhu, Privacy-preserving clustering with distributed EM mixture modeling, Knowledge and Information Systems, 8(1):68–81, 2005. DOI: 10.1007/s10115-004-0148-7. 44
Y. Lindell and B. Pinkas, Privacy preserving data mining, Journal of Cryptology, 15(3), 2002. DOI: 10.1007/s00145-001-0019-2. 25, 41
M. Liu, H. Jiang, J. Chen, et al., A collaborative privacy-preserving deep learning system in distributed mobile environment, In International Conference on Computational Science and Computational Intelligence (CSCI), pp. 192–197, 2016. DOI: 10.1109/csci.2016.0043. 10, 44
O. L. Mangasarian, E. W. Wild, and G. M. Fung, Privacy-preserving classification of vertically partitioned data via random kernels, ACM Transactions on Knowledge Discovery from Data (TKDD), 2(3):12, 2008. DOI: 10.1145/1409620.1409622. 10, 43
R. Mendes and J. P. Vilela, Privacy-preserving data mining: Methods, metrics, and applications, IEEE Access, 5:10562–10582, 2017. DOI: 10.1109/access.2017.2706947. 10, 17, 20, 44
P. Mohassel and Y. Zhang, SecureML: A system for scalable privacy-preserving machine learning, In Proc. of Symposium on Security and Privacy (SP), pp. 19–38, 2017. DOI: 10.1109/sp.2017.12. 10, 23, 25, 44, 48, 74
N. Papernot, M. Abadi, U. Erlingsson, et al., Semi-supervised knowledge transfer for deep learning from private training data, ArXiv Preprint ArXiv:1610.05755, October 2016. http://arxiv.org/abs/1610.05755 31
N. Papernot, S. Song, I. Mironov, et al., Scalable private learning with pate, ArXiv Preprint ArXiv:1802.08908, February 2018. http://arxiv.org/abs/1802.08908 31
M. Park, J. Foulds, K. Chaudhuri, et al., DP-EM: Differentially private expectation maximization, ArXiv Preprint ArXiv:1605.06995, May 2016. http://arxiv.org/abs/1605.06995 43
J. Quinlan, Induction of decision trees, Machine Learning, pp. 81–106, March 1986. DOI: 10.1007/bf00116251. 41
R. L. Rivest, L. Adleman, and M. L. Dertouzos, On data banks and privacy homomorphisms, Foundations of Secure Computation, 4(11):169–180, 1978. 19, 26
A. Shamir, How to share a secret, Communications of the ACM, 22(11):612–613, 1979. DOI: 10.1145/359168.359176. 22, 24
A. B. Slavkovic, Y. Nardi, and M. M. Tibbits, Secure logistic regression of horizontally and vertically partitioned distributed databases, In 7th International Conference on Data Mining Workshops (ICDMW), pp. 723–728, 2007. DOI: 10.1109/icdmw.2007.114. 44
S. Song, K. Chaudhuri, and A. D. Sarwate, Stochastic gradient descent with differentially private updates, In Global Conference on Signal and Information Processing, pp. 245–248, 2013. DOI: 10.1109/globalsip.2013.6736861. 43
J. Vaidya and C. Clifton, Privacy preserving naive Bayes classifier for vertically partitioned data, In Proc. of the SIAM International Conference on Data Mining, pp. 522–526, 2004. DOI: 10.1137/1.9781611972740.59. 10, 44
P. Vepakomma, T. Swedish, R. Raskar, et al., No peek: A survey of private distributed deep learning, ArXiv Preprint ArXiv:1812.03288, December 2018. vailable: http://arxiv.org/abs/1812.03288 7, 10, 40, 44
K. Wang, Y. Xu, R. She, et al., Classification spanning private databases, In Proc. of the National Conference on Artificial Intelligence, 21:293, Menlo Park, CA, Cambridge, MA, London, AAAI Press, MIT Press, 1999, 2006. 41
E. Wild and O. Mangasarian, Privacy-preserving classification of horizontally partitioned data via random kernels, Technical Report, 2007. 10, 43
K. Xu, H. Yue, L. Guo, et al., Privacy-preserving machine learning algorithms for big data systems, In Proc. of 35th International Conference on Distributed Computing Systems, pp. 318–327, 2015. DOI: 10.1109/icdcs.2015.40. 10, 44
S. Yakoubov, A gentle introduction to Yao’s garbled circuits, 2017. http://web.mit.edu/sonka89/www/papers/2017ygc.pdf 23
A. C. Yao, Protocols for secure computations, In FOCS, 82:160–164, 1982. DOI: 10.1109/sfcs.1982.38. 19, 28
A. C.-C. Yao, How to generate and exchange secrets, In Proc. of 27th Annual Symposium on Foundations of Computer Science, pp. 162–167, 1986. DOI: 10.1109/sfcs.1986.25. 21, 23
H. Yu, X. Jiang, and J. Vaidya, Privacy-preserving SVM using nonlinear kernels on horizontally partitioned data, In Proc. of the ACM Symposium on Applied Computing, pp. 603–610, 2006. DOI: 10.1145/1141277.1141415. 44
J. Zhan and S. Matwin, Privacy-preserving support vector machine classification, International Journal of Intelligent Information and Database Systems, 1(3–4):356–385, 2007. DOI: 10.1504/ijiids.2007.016686. 44
D. Zhang, X. Chen, D. Wang, et al., A survey on collaborative deep learning and privacy-preserving, In 3rd International Conference on Data Science in Cyberspace (DSC), pp. 652–658, 2018. DOI: 10.1109/dsc.2018.00104. 10, 43, 44
L. Song, J. Mao, Y. Zhuo, et al., HyPar: Towards hybrid parallelism for deep learning accelerator array, In Proc. of 25th International Symposium on High-Performance Computer Architecture, February 2019. https://arxiv.org/abs/1901.02067 DOI: 10.1109/hpca.2019.00027. 40
A. Krizhevsky, One weird trick for parallelizing conutional neural networks, ArXiv Preprint ArXiv:1404.5997, April 2014. https://arxiv.org/abs/1404.5997 40
M. Wang, C.-C. Huang, and J. Li, Unifying data, model and hybrid parallelism in deep learning via tensor tiling, ArXiv Preprint ArXiv:1805.04170, May 2018. https://arxiv.org/abs/1805.04170 40
N. Pansare, M. Dusenberry, N. Jindal, et al., Deep learning with Apache SystemML, ArXiv Preprint ArXiv:1802.04647, February 2018. https://arxiv.org/abs/1802.04647 39
D. Shrivastava, S. Chaudhury, and Dr. Jayadeva, A data and model-parallel, distributed and scalable framework for training of deep networks in Apache Spark, ArXiv Preprint ArXiv:1708.05840, August 2017. https://arxiv.org/abs/1708.05840 40
M. Boehm, S. Tatikonda, B. Reinwald, et al., Hybrid parallelization strategies for large-scale machine learning in SystemML, In Proc. of the VLDB Endowment, pp. 553–564, March 2016. DOI: 10.14778/2732286.2732292. 39
Apache Hadoop MapReduce, June 2019. https://hadoop.apache.org/docs/r2.8.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html 37, 39
Apache Hadoop YARN, June 2019. https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html 39
Apache Storm, June 2019. https://storm.apache.org/ 39
Y. Feunteun, Parallel and distributed deep learning: A survey, April 2019. https://towardsdatascience.com/parallel-and-distributed-deep-learning-a-survey-97137ff94e4c 33, 36
X. Tian, B. Xie, and J. Zhan, Cymbalo: An efficient graph processing framework for machine learning, In Proc. of IEEE International Conference on Parallel and Distributed Processing, December 2018. DOI: 10.1109/bdcloud.2018.00090. 39
Z. Zhang, P. Cui, and W. Zhu, Deep learning on graphs: A survey, ArXiv Preprint ArXiv:1812.04202, December 2018. https://arxiv.org/abs/1812.04202 39
W. Xiao, J. Xue, Y. Miao, et al., Tux2: Distributed graph computation for machine learning, In Proc. of the 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI), March 2017. 39
Apache DeepSpark, June 2019. http://deepspark.snu.ac.kr/ 35, 38
H. Kim, J. Park, J. Jang, et al., Deepspark: Spark-based deep learning supporting asynchronous updates and Caffe compatibility, ArXiv Preprint ArXiv:1602.08191, October 2016. https://arxiv.org/abs/1602.08191 38
Z. Jia, S. Lin, C. R. Qi, et al., Exploring hidden dimensions in accelerating conutional neural networks, In Proc. of the 35th International Conference on Machine Learning (ICML), July 2018. https://cs.stanford.edu/zhihao/papers/icml18full.pdf 38
Z. Jia, M. Zaharia, and A. Aiken, Beyond data and model parallelism for deep neural networks, In Proc. of the Conference on Systems and Machine Learning (SysML), April 2019. 37, 38
A. L. Gaunt, M. A. Johnson, A. Lawrence, et al., AMPNet: Asynchronous model-parallel training for dynamic neural networks, In Proc. of the 6th International Conference on Learning Representations, May 2018. 38
T. Chilimbi, Y. Suzue, J. Apacible, et al., Project Adam: Building an efficient and scalable deep learning training system, In Proc. of the 11th USENIX Conference on Operating Systems Design and Implementation (OSDI), pp. 571–582, October 2014. 38
J. Dean, G. Corrado, R. Monga, et al., Large scale distributed deep networks, In Proc. of the 25th International Conference on Neural Information Processing Systems (NIPS), pp. 1223–1231, December 2012. 10, 38, 40
K. Fukuda, Technologies behind distributed deep learning: AllReduce, July 2018. https://preferredresearch.jp/2018/07/10/technologies-behind-distributed-deep-learning-allreduce/ 37
M. Li, D. G. Andersen, J. W. Park, et al., Scaling distributed machine learning with the parameter server, In Proc. of the 11th USENIX Conference on Operating Systems Design and Implementation (OSDI), pp. 583–598, October 2014. DOI: 10.1145/2640087.2644155. 10, 33, 37, 52
A. Das, Distributed training of deep learning models with PyTorch, April 2019. https://medium.com/intel-student-ambassadors/distributed-training-of-deep-learning-models-with-pytorch-1123fa538848 37, 38
S. Wang, Distributed machine learning, January 2016. https://www.slideshare.net/stanleywanguni/distributed-machine-learning?from_action=save 10, 37, 38, 39
Google Inc., Distributed training in TensorFlow, June 2019. https://www.tensorflow.org/guide/distribute_strategy 35
S. Arnold, Writing distributed applications with PyTorch, June 2019. https://pytorch.org/tutorials/intermediate/dist_tuto.html 36
Microsoft, Distributed machine learning Toolkit (DMTK), June 2019. http://www.dmtk.io/ 35
Turi-Create, June 2019. https://turi.com/ 35
G. Malewicz, M. H. Austern, A. J. C. Bik, et al., Pregel: A system for large-scale graph processing, In Proc. of the ACM SIGMOD International Conference on Management of Data (SIGMOD), June 2010. DOI: 10.1145/1807167.1807184. 35
Y. Low, J. Gonzalez, A. Kyrola, et al., GraphLab: A new framework for parallel machine learning, ArXiv Preprint ArXiv:1006.4990, June 2010. https://arxiv.org/abs/1006.4990 35, 39
Apache Spark MLlib, June 2019. https://spark.apache.org/mllib/ 35
Apache Spark GraphX, June 2019. https://spark.apache.org/docs/latest/graphx-programming-guide.html 35
T. Ben-Nun and T. Hoefler, Demystifying parallel and distributed deep learning: An in-depth concurrency analysis, ArXiv Preprint ArXiv:1802.09941, September 2018. https://arxiv.org/abs/1802.09941 DOI: 10.1145/3320060. 10, 33, 36
J. Devlin, M. W. Chang, K. Lee, et al., BERT: Pre-training of deep bidirectional transformers for language understanding, ArXiv Preprint ArXiv:1810.04805, May 2019. https://arxiv.org/abs/1810.04805 34, 38
A. Galakatos, A. Crotty, and T. Kraska, Distributed machine learning, In Encyclopedia of Database Systems, December 2018. DOI: 10.1007/978-1-4614-8265-9_80647. 33, 36
R. Bekkerman, M. Bilenko, and J. Langford, Scaling up machine learning: Parallel and distributed approaches, Cambridge University Press, February 2012. DOI: 10.1017/cbo9781139042918. 33, 36
Y. Liu, J. Liu, and T. Basar, Differentially private gossip gradient descent, In IEEE Conference on Decision and Control (CDC), pp. 2777–2782, December 2018. DOI: 10.1109/cdc.2018.8619437. 54
J. Daily, A. Vishnu, C. Siegel, et al., GossipGraD: Scalable deep learning using gossip communication based asynchronous gradient descent, ArXiv Preprint ArXiv:1803.05880, March 2018. http://arxiv.org/abs/1803.05880 54
C. Hardy, E. Le Merrer, and B. Sericola, Gossiping GANs: Position paper, In Proc. of the 2nd Workshop on Distributed Infrastructures for Deep Learning, pp. 25–28, December 2018. DOI: 10.1145/3286490.3286563. 54
I. Hegedüs, G. Danner, and M. Jelasity, Gossip learning as a decentralized alternative to federated learning, In Proc. of the 14th International Federated Conference on Distributed Computing Techniques, pp. 74–90, June 2019. DOI: 10.1007/978-3-030-22496-7_5. 54
D. Liu, T. Miller, R. Sayeed, et al., FADL: Federated-autonomous deep learning for distributed electronic health record, ArXiv Preprint ArXiv:1811.11400, November 2018. https://arxiv.org/abs/1811.11400 56
T. Nishio and R. Yonetani, Client selection for federated learning with heterogeneous resources in mobile edge, ArXiv Preprint ArXiv:1804.08333, October 2018. https://arxiv.org/abs/1804.08333 DOI: 10.1109/icc.2019.8761315. 64
I. J. Goodfellow, O. Vinyals, and A. M. Saxe, Qualitatively characterizing neural network optimization problems, ArXiv Preprint ArXiv:1412.6544, May 2015. https://arxiv.org/abs/1412.6544 60, 65
S. Ioffe and C. Szegedy, Batch normalization: Accelerating deep network training by reducing internal covariate shift, In Proc. of the 32nd International Conference on Machine Learning (ICML), July 2015. 57
H. Tang, C. Yu, C. Renggli, et al., Distributed learning over unreliable networks, ArXiv Preprint ArXiv:1810.07766, May 2019. https://arxiv.org/abs/1810.07766 51, 52, 56, 60
Q. Ho, J. Cipar, H. Cui, et al., More effective distributed machine learning via a stale synchronous parallel parameter server, In Proc. of the 26th International Conference on Neural Information Processing Systems (NIPS), pp. 1223–1231, December 2013. 52
H. Su and H. Chen, Experiments on parallel training of deep neural network using model averaging, ArXiv Preprint ArXiv:1507.01239, July 2018. https://arxiv.org/abs/1507.01239 51, 52, 60
X. Shu, G.-J. Qi, J. Tang, et al., Weakly-shared deep transfer networks for heterogeneous-domain knowledge propagation, In Proc. of the 23rd ACM International Conference on Multimedia (MM), pp. 35–44, 2015. DOI: 10.1145/2733373.2806216. 87
F. Seide, G. Li, and D. Yu, Conversational speech transcription using context-dependent deep neural networks, In 12th Annual Conference of the International Speech Communication Association, pp. 437–440, 2011.
M. Kamp, L. Adilova, J. Sicking, et al., Efficient decentralized deep learning by dynamic model averaging, In Proc. of Machine Learning and Knowledge Discovery in Databases (KDD), pp. 393–409, September 2018. DOI: 10.1007/978-3-030-10925-7_24. 63
S. Han, H. Mao, and W. J. Dally, Deep compression: Compressing deep neural networks with pruning, trained quantization and Huffman coding, ArXiv Preprint ArXiv:1510.00149, February 2016. https://arxiv.org/abs/1510.00149 63
Y. Lin, S. Han, H. Mao, et al., Deep gradient compression: Reducing the communication bandwidth for distributed training, In International Conference on Learning Representations (ICLR), April 2018. 50
E. Zhong, W. Fan, Yang, et al., Cross validation framework to choose amongst models and datasets for transfer learning, In J. L. Balcázar, F. Bonchi, A. Gionis, and M. Sebag, Eds., Machine Learning and Knowledge Discovery in Databases, pp. 547–562, Springer, Heidelberg, 2010. DOI: 10.1007/978-3-642-15883-4.
I. Kuzborskij and F. Orabona, Stability and hypothesis transfer learning, In Proc. of the 30th International Conference on Machine Learning (ICML), 28(3):942–950, 2013.
B. Hitaj, G. Ateniese, and F. Pérez-Cruz, Deep models under the GAN: Information leakage from collaborative deep learning, In Proc. of the ACM SIGSAC Conference on Computer and Communications Security, pp. 603–618, October 2017. DOI: 10.1145/3133956.3134012. 10, 50, 52, 89
F. McSherry, Deep learning and differential privacy, https://github.com/frankmcsherry/blog/blob/master/posts/2017-10-27.md 89
Z. Li, Y. Zhang, Y. Wei, et al., End-to-end adversarial memory network for cross-domain sentiment classification, In Proc. of the 26th International Joint Conference on Artificial Intelligence (IJCAI), pp. 2237–2243, August 2017. DOI: 10.24963/ijcai.2017/311. 84
S. J. Pan, X. Ni, J.-T. Sun, et al., Cross-domain sentiment classification via spectral feature alignment, In Proc. of the 19th International Conference on World Wide Web (WWW), pp. 751–760, April 2010. DOI: 10.1145/1772690.1772767. 84
Y. Zhu, Y. Chen, Z. Lu, et al., Heterogeneous transfer learning for image classification, In Proc. of the 25th AAAI Conference on Artificial Intelligence (AAAI), pp. 1304–1309, August 2011. 84
M. Oquab, L. Bottou, I. Laptev, et al., Learning and transferring mid-level image representations using conutional neural networks, In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition CVPR, pp. 1717–1724, June 2014. DOI: 10.1109/cvpr.2014.222. 86
R. Bahmani, M. Barbosa, F. Brasser, et al., Secure multiparty computation from SGX, In Proc. of International Conference on Financial Cryptography and Data Security Financial Cryptography and Data Security (FC), pp. 477–497, December 2017. DOI: 10.1007/978-3-319-70972-7_27. 71
T. Chen and C. Guestrin, XGBoost: A scalable tree boosting system, In Proc. of the 22nd International Conference on Knowledge Discovery and Data Mining (KDD), pp. 785–794, August 2016. DOI: 10.1145/2939672.2939785. 77, 78
K. Chang, N. Balachandar, C. K. Lam, et al., Institutionally distributed deep learning networks, ArXiv Preprint ArXiv:1709.05929, September 2017. https://arxiv.org/abs/1709.05929 53
K. Chang, N. Balachandar, C. Lam, et al., Distributed deep learning networks among institutions for medical imaging, Journal of the American Medical Informatics Association, 25(8):945–954, August 2018. DOI: 10.1093/jamia/ocy017. 53
L. T. Phong, Privacy-preserving stochastic gradient descent with multiple distributed trainers, In Proc. of the 11th International Conference on Network and System Security (NSS), pp. 510–518, July 2017. DOI: 10.1007/978-3-319-64701-2_38. 10
L. T. Phong and T. T. Phuong, Privacy-preserving deep learning via weight transmission, IEEE Transactions on Information Forensics and Security, April 2019. https://arxiv.org/abs/1809.03272 DOI: 10.1109/tifs.2019.2911169. 33, 52, 53
L. T. Phong, Y. Aono, T. Hayashi, et al., Privacy-preserving deep learning via additively homomorphic encryption, IEEE Transactions on Information Forensics and Security, 13(5):1333–1345, May 2018. DOI: 10.1109/tifs.2017.2787987. 10, 49, 50, 51, 52, 60, 62, 88, 89
L. Su and J. Xu, Securing distributed gradient descent in high dimensional statistical learning, In Proc. of the ACM on Measurement and Analysis of Computing Systems, 3(1), March 2019. DOI: 10.1145/3309697.3331499.
S. Tutdere and O. Uzunko, Construction of arithmetic secret sharing schemes by using torsion limits, ArXiv Preprint ArXiv:1506.06807, June 2015. https://arxiv.org/abs/1506.06807 DOI: 10.15672/hujms.460348. 24
A. Beimel, Secret-sharing schemes: A Survey, IWCC, LNCS 6639, pp. 11–46, Springer-Verlag, 2011. DOI: 10.1007/978-3-642-20901-7_2. 24
A. Acar, H. Aksu, A. S. Uluagac, et al., A survey on homomorphic encryption schemes: Theory and implementation, ACM Computing Surveys (CSUR), 51(4):79:1–79:35, 2018. DOI: 10.1145/3214303. 27, 28, 62, 88
Y. Aono, T. Hayashi, L. Wang, et al., Privacy-preserving deep learning via additively homomorphic encryption, IEEE Transactions on Information Forensics and Security, 13(5):1333–1345, 2018. DOI: 10.1109/TIFS.2017.2787987. 19, 46
F. Armknecht, C. Boyd, C. Carr, et al., A guide to fully homomorphic encryption, IACR Cryptology ePrint Archive, 2015. https://eprint.iacr.org/2015/1192.pdf 27
H. Bae, D. Jung, and S. Yoon, Anomigan: Generative adversarial networks for anonymizing private medical data, ArXiv Preprint ArXiv:1901.11313, January 2019. https://arxiv.org/abs/1901.11313
E. Bagdasaryan, A. Veit, Y. Hua, et al., How to backdoor federated learning, ArXiv Preprint ArXiv:1807.00459, August 2019. https://arxiv.org/abs/1807.00459 20, 140
M. Barreno, B. Nelson, R. Sears, et al., Can machine learning be secure? In Proc. of the ACM Symposium on Information, Computer and Communications Security, pp. 16–25, 2006. DOI: 10.1145/1128817.1128824. 17, 18
D. Beaver, Efficient multiparty protocols using circuit randomization, In Proc. of the Annual International Cryptology Conference, pp. 420–432, Springer, 1991. DOI: 10.1007/3-540-46766-1_34. 24, 48
D. Beaver, Correlated pseudorandomness and the complexity of private computations, In Proc. STOC Proceedings of the 28th Annual ACM Symposium on Theory of Computing, pp. 479–488, May 1996. DOI: 10.1145/237814.237996. 23
M. Bellare and S. Micali, Non-interactive oblivious transfer and applications, In G. Brassard, Ed., Advances in Cryptology—CRYPTO Proceedings, pp. 547–557, Springer, New York, 1990. DOI: 10.1007/0-387-34805-0. 22
M. Ben-or, S. Goldwasser, and A. Wigderson, Completeness theorems for non-cryptographic fault-tolerant distributed computation (extended abstract), In Proc. of the 20th Annual ACM Symposium on Theory of Computing, pp. 1–10, January 1988. DOI: 10.1145/62212.62213.
D. Bogdanov, L. Kamm, S. Laur, et al., Privacy-preserving statistical data analysis on federated databases, In Annual Privacy Forum, pp. 30–55, Springer, 2014. DOI: 10.1007/978-3-319-06749-0_3. 10, 18
D. Boneh, R. Gennaro, S. Goldfeder, et al., Threshold cryptosystems from threshold fully homomorphic encryption, In H. Shacham and A. Boldyreva, Eds., Advances in Cryptology—CRYPTO, pp. 565–596, Springer International Publishing, 2018. DOI: 10.1007/978-3-319-96881-0.
D. Boneh, E.-J. Goh, and K. Nissim, Evaluating 2-DNF formulas on ciphertexts, In Theory of Cryptography Conference, pp. 325–341, Springer, 2005. DOI: 10.1007/978-3-540-30576-7_18. 26, 28
K. Bonawitz, V. Ivanov, B. Kreuter, et al., Practical secure aggregation for privacy-preserving machine learning, In Proc. of the ACM SIGSAC Conference on Computer and Communications Security (CCS), pp. 1175–1191, November 2017. DOI: 10.1145/3133956.3133982. 25, 49, 51, 52
K. Bonawitz, H. Eichner, W. Grieskamp, et al., Towards federated learning at scale: System design, ArXiv Preprint ArXiv:1902.01046, March 2019. https://arxiv.org/abs/1902.01046 11, 56, 65, 66, 98, 113, 141, 143
Z. Brakerski, C. Gentry, and V. Vaikuntanathan, Fully homomorphic encryption without bootstrapping, IACR Cryptology ePrint Archive, 2011. 28
Z. Brakerski and V. Vaikuntanathan, Fully homomorphic encryption from ring-LWE and security for key dependent messages, In P. Rogaway, Ed., Advances in Cryptology—CRYPTO, pp. 505–524, Springer, 2011. DOI: 10.1007/978-3-642-22792-9.
R. Canetti, Universally composable security: A new paradigm for cryptographic protocols, In Proc. IEEE International Conference on Cluster Computing, pp. 136–145, October 2001. DOI: 10.1109/sfcs.2001.959888.
K. Chaudhuri, C. Monteleoni, and A. D. Sarwate, Differentially private empirical risk minimization, Journal of Machine Learning Research, pp. 1069–1109, March 2011.
D. Cozzo and N. P. Smart, Using TopGear in overdrive: A more efficient ZKPoK for SPDZ, Cryptology ePrint Archive, Report 2019/035, 2019. https://eprint.iacr.org/2019/035
I. Damård, V. Pastro, N. P. Smart, et al., Multiparty computation from somewhat homomorphic encryption, Cryptology ePrint Archive, Report 2011/535, 2011. https://eprint.iacr.org/2011/535 DOI: 10.1007/978-3-642-32009-5_38. 24, 25, 48
I. Damård, M. Keller, E. Larraia, et al., Practical covertly secure MPC for dishonest majority—or: Breaking the SPDZ limits, Cryptology ePrint Archive, Report 2012/642, 2012. https://eprint.iacr.org/2012/642 DOI: 10.1007/978-3-642-40203-6_1. 25
I. Damgård and J. B. Nielsen, Universally composable efficient multiparty computation from threshold homomorphic encryption, In D. Boneh, Ed., Advances in Cryptology—CRYPTO, pp. 247–264, Springer, 2003. DOI: 10.1007/978-3-540-45146-4_15. 22
D. Demmler, T. Schneider, and M. Zohner, Aby-a framework for efficient mixed-protocol secure two-party computation, In NDSS, February 2015. DOI: 10.14722/ndss.2015.23113. 23, 25
W. Diffie and M. E. Hellman, New directions in cryptography, IEEE Transactions on Information Theory, 22(6):644–654, November 1976. DOI: 10.1109/tit.1976.1055638. 23
N. Dowlin, R. Gilad-Bachrach, K. Laine, et al., CryptoNets: Applying neural networks to encrypted data with high throughput and accuracy, In International Conference on Machine Learning, pp. 201–210, June 2016.
W. Du, Y. S. Han, and S. Chen, Privacy-preserving multivariate statistical analysis: Linear regression and classification, In Proc. of the SIAM International Conference on Data Mining, pp. 222–233, Society for Industrial and Applied Mathematics, April 2004. DOI: 10.1137/1.9781611972740.21. 46, 75, 85
C. Dwork, Differential privacy, Encyclopedia of Cryptography and Security, pp. 338–340, 2011. DOI: 10.1007/978-1-4419-5906-5_752.
C. Dwork, K. Kenthapadi, F. McSherry, et al., Our data, ourselves: Privacy via distributed noise generation, In Annual International Conference on the Theory and Applications of Cryptographic Techniques, pp. 486–503, Springer, 2006. DOI: 10.1007/11761679_29. 30
C. Dwork, V. Feldman, M. Hardt, et al., Preserving statistical validity in adaptive data analysis, ArXiv Preprint ArXiv:1411.2664, March 2016. https://arxiv.org/abs/1411.2664 29
C. Dwork, F. McSherry, K. Nissim, et al., Calibrating noise to sensitivity in private data analysis, In Theory of Cryptography Conference, pp. 265–284, Springer, 2006. DOI: 10.1007/11681878_14. 20, 29, 30
C. Dwork and K. Nissim, Privacy-preserving data mining on vertically partitioned databases, In Annual International Cryptology Conference, Springer, pp. 528–544, 2004. DOI: 10.1007/978-3-540-28628-8_32. 30
T. ElGamal, A public key cryptosystem and a signature scheme based on discrete logarithms, IEEE Transactions on Information Theory, 31(4):469–472, 1985. DOI: 10.1007/3-540-39568-7_2. 27
C. Fontaine and F. Galand, A survey of homomorphic encryption for nonspecialists, EURASIP Journal on Information Security, (15), 2007. DOI: 10.1186/1687-417x-2007-013801.
A. Gascón, P. Schoppmann, B. Balle, et al., Secure linear regression on vertically partitioned datasets, IACR Cryptology ePrint Archive, 2016.
C. Gentry, Fully homomorphic encryption using ideal lattices, In Proc. of the 41st Annual ACM Symposium on Theory of Computing, 9:169–178, June 2009. DOI: 10.1145/1536414.1536440. 26, 28
N. Gilboa, Two party RSA key generation, In Annual International Cryptology Conference, pp. 116–129, Springer, 1999. DOI: 10.1007/3-540-48405-1_8. 24
O. Goldreich, S. Micali, and A. Wigderson, How to play any mental game, In Proc. of the 19th Annual ACM Symposium on Theory of Computing, pp. 218–229, January 1987. DOI: 10.1145/28395.28420. 22, 23, 25, 71
S. Goldwasser and S. Micali, Probabilistic encryption and how to play mental poker keeping secret all partial information, In Proc. of the 14th Annual ACM Symposium on Theory of Computing, pp. 365–377, 1982. DOI: 10.1145/800070.802212. 26, 27
T. Gu, B. Dolan-Gavitt, and S. Garg, Badnets: Identifying vulnerabilities in the machine learning model supply chain, ArXiv Preprint ArXiv:1708.06733, August 2017. https://arxiv.org/abs/1708.06733
S. Hardy, W. Henecka, H. Ivey-Law, et al., Private federated learning on vertically partitioned data via entity resolution and additively homomorphic encryption, ArXiv Preprint ArXiv:1711.10677, November 2017. https://arxiv.org/abs/1711.10677 28
C. Hazay and Y. Lindell, Efficient secure two-party protocols, In Information Security and Cryptography, 2010. DOI: 10.1007/978-3-642-14303-8. 22
X. He, A. Prasad, S. P. Sethi, et al., A survey of Stackelberg differential game models in supply and marketing channels, Journal of Systems Science and Systems Engineering, 16:385–413, 2007. DOI: 10.1007/s11518-008-5082-x.
R. Impagliazzo and S. Rudich, Limits on the provable consequences of one-way permutations, In Proc. of the 21st Annual ACM Symposium on Theory of Computing (STOC), pp. 44–61, 1989. DOI: 10.1145/73007.73012. 23
Y. Ishai and A. Paskin, Evaluating branching programs on encrypted data, In S.P. Vadhan, Ed., Theory of Cryptography, pp. 575–594, Springer, 2007. DOI: 10.1007/978-3-540-70936-7. 28
Y. Ishai, M. Prabhakaran, and A. Sahai, Founding cryptography on oblivious transfer—efficiently, In David Wagner, Ed., Advances in Cryptology—CRYPTO, pp. 572–591, Springer Berlin Heidelberg, Berlin, Heidelberg, 2008. DOI: 10.1007/978-3-540-85174-5. 22
B. Jayaraman and D. Evans, When relaxations go bad: Differentially-private machine learning, ArXiv Preprint ArXiv:1902.08874, February 2019. https://arxiv.org/abs/1902.08874 30
L. Jiang, R. Tan, X. Lou, et al., On lightweight privacy-preserving collaborative learning for internet-of-things objects, In IoTDI, 2019. DOI: 10.1145/3302505.3310070. 47
A. F. Karr, X. S. Lin, A. P. Sanil, et al., Privacy-preserving analysis of vertically partitioned data using secure matrix products, Journal of Official Statistics, pp. 125–138, September 2009.
M. Kim, Y. Song, S. Wang, et al., Secure logistic regression based on homomorphic encryption: Design and evaluation, JMIR Medical Informatics, 6(2), April 2018. DOI: 10.2196/preprints.8805. 88
Y. Lindell, Secure multiparty computation for privacy preserving data mining, In Encyclopedia of Data Warehousing and Mining, pp. 1005–1009, IGI Global, 2005. DOI: 10.4018/9781591405573.ch189. 21
Y. Lindell, How to simulate it—a tutorial on the simulation proof technique, In Y. Lindell, Ed., Tutorials on the Foundations of Cryptography, Information Security and Cryptography, pp. 277–346, Springer, April 2017. DOI: 10.1007/978-3-319-57048-8. 22
Y. Lindell and B. Pinkas, Secure multiparty computation for privacy-preserving data mining, IACR Cryptology ePrint Archive, 1(1):59–98, April 2009. DOI: 10.4018/9781591405573.ch189. 21
A. López-Alt, E. Tromer, and V. Vaikuntanathan, On-the-fly multiparty computation on the cloud via multi-key fully homomorphic encryption, In Proc. of the 44th Annual ACM Symposium on Theory of Computing, pp. 1219–1234, 2012. DOI: 10.1145/2213977.2214086. 28
V. Lyubashevsky, C. Peikert, and O. Regev, On ideal lattices and learning with errors over rings, In Annual International Conference on the Theory and Applications of Cryptographic Techniques, pp. 1–23, Springer, 2010. DOI: 10.1145/2535925. 28
F. McSherry and K. Talwar, Mechanism design via differential privacy, In FOCS, 7:94–103, 2007. DOI: 10.1109/focs.2007.66. 30
D. Mishra and D. Veeramani, Vickrey–Dutch procurement auction for multiple items, European Journal of Operational Research, 180:617–629, 2007. DOI: 10.1016/j.ejor.2006.04.020. 99
P. Mohassel and P. Rindal, ABY3: A mixed protocol framework for machine learning, In Proc. of the ACM SIGSAC Conference on Computer and Communications Security CCS, pp. 35–52, October 2018. DOI: 10.1145/3243734.3243760.
M. Naor and B. Pinkas, Efficient oblivious transfer protocols, In Proc. of the 12th Annual ACMSIAM Symposium on Discrete Algorithms, pp. 448–457, Society for Industrial and Applied Mathematics, January 2001. 22
A. Narayanan and V. Shmatikov, Robust de-anonymization of large datasets (how to break anonymity of the Netflix prize dataset), University of Texas at Austin, February 2008. 20
P. Paillier, Public-key cryptosystems based on composite degree residuosity classes, In International Conference on the Theory and Applications of Cryptographic Techniques, pp. 223–238, Springer, Berlin, Heidelberg, May 1999. DOI: 10.1007/3-540-48910-x_16. 26, 27, 61, 62, 78
N. Phan, Y. Wang, X. Wu, et al., Differential privacy preservation for deep auto-encoders: an application of human behavior prediction, In 30th AAAI Conference on Artificial Intelligence, February 2016.
M. O. Rabin, How to exchange secrets with oblivious transfer, Harvard University Technical Report, May 1981. https://eprint.iacr.org/2005/187.pdf 22
T. Rabin and M. Ben-Or, Verifiable secret sharing and multiparty protocols with honest majority, In Proc. of the 21st Annual ACM Symposium on Theory of Computing STOC, pp. 73–85, New York, 1989. DOI: 10.1145/73007.73014. 22
L. Reyzin, A. D. Smith, and S. Yakoubov, Turning HATE into LOVE: Homomorphic ad hoc threshold encryption for scalable MPC, IACR Cryptology ePrint Archive, 2018.
R. L. Rivest, A. Shamir, and L. Adleman, A method for obtaining digital signatures and public-key cryptosystems, Communications of the ACM, 21(2):120–126, 1978. DOI: 10.21236/ada606588. 27
B. D. Rouhani, M. S. Riazi, and F. Koushanfar, DeepSecure: Scalable provably-secure deep learning, ArXiv Preprint ArXiv:1705.08963, May, 2017. https://arxiv.org/abs/1705.08963 DOI: 10.1109/dac.2018.8465894. 25
P. Samarati and L. Sweeney, Protecting privacy when disclosing information: k-anonymity and its enforcement through generalization and suppression, Technical Report, SRI International, 1998.
A. P. Sanil, A. F. Karr, X. Lin, et al., Privacy preserving regression modelling via distributed computation, In Proc. of the 10th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 677–682, August 2004. DOI: 10.1145/1014052.1014139.
N. P. Smart, The discrete logarithm problem on elliptic curves of trace one, Journal of Cryptology, 12:193–196, 1999. DOI: 10.1007/s001459900052.
J. Vaidya and C. Clifton, Privacy preserving association rule mining in vertically partitioned data, In Proc. of the 8th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 639–644, July 2002. DOI: 10.1145/775047.775142. 75
M. V. Dijk, C. Gentry, S. Halevi, et al., Fully homomorphic encryption over the integers, In Annual International Conference on the Theory and Applications of Cryptographic Techniques, pp. 24–43, Springer, 2010. DOI: 10.1007/978-3-642-13190-5_2. 28
S. Wagh, D. Gupta, and N. Chandran, SecureNN: Efficient and private neural network training, IACR Cryptology ePrint Archive, 2018. 48
Z. Brakerski and V. Vaikuntanathan, Efficient fully homomorphic encryption from (standard) LWE, In IEEE 52nd Annual Symposium on Foundations of Computer Science, pp. 97–106, October 2011. DOI: 10.1109/focs.2011.12. 28
L. Wan, W. K. Ng, S. Han, et al., Privacy-preservation for gradient descent methods, In Proc. of the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 775–783, August 2007. DOI: 10.1145/1281192.1281275. 45, 46
J.-S. Weng, J. Weng, M. Li, et al., DeepChain: Auditable and privacy-preserving deep learning with blockchain-based incentive, IACR Cryptology ePrint Archive, 2018.
A. F. Westin, Privacy and freedom, Washington Lee Law Review, 1968. 17
C. Xie, S. Koyejo, and I. Gupta, SLSGD: Secure and efficient distributed on-device machine learning, ArXiv Preprint ArXiv:1903.06996, March 2019. https://arxiv.org/abs/1903.06996
Y. Yin, I. Kaku, J. Tang, et al., Application for privacy-preserving data mining, Springer London, London, 2011. DOI: 10.1007/978-1-84996-338-1_14.
M. Chen, R. Mathews, T. Ouyang, et al., Federated learning of out-of-vocabulary words, ArXiv Preprint ArXiv:1903.10635, March 2019. https://arxiv.org/abs/1903.10635 11, 112
A. Sergeev and M. D. Balso, Horovod: Fast and easy distributed deep learning in TensorFlow, ArXiv Preprint ArXiv:1802.05799, February 2018. https://arxiv.org/abs/1802.05799 12
coMind.org, Machine learning network for deep learning, https://comind.org/ 12
V. Smith, C.-K. Chiang, M. Sanjabi, et al., Federated multi-task learning, In Proc. of International Conference on Neural Information Processing Systems (NIPS), December 2017. 11, 50
H. H. Zhuo, W. Feng, Q. Xu, et al., Federated reinforcement learning, ArXiv Preprint ArXiv:1901.08277, January 2019. https://arxiv.org/abs/1901.08277 11, 126, 130
WeBank AI Department, Federated learning white paper V1.0, September 2018. https://aisp-1251170195.cos.ap-hongkong.myqcloud.com/fedweb/1552917186945.pdf 2, 5
S. Pouyanfar, S. Sadiq, Y. Yan, et al., A survey on deep learning: Algorithms, techniques, and applications, ACM Computing Survey, 51(5):1–36, January 2019. DOI: 10.1145/3234150. 1
W. G. Hatcher and W. Yu, A survey of deep learning: Platforms, applications and emerging research trends, IEEE Access, 6(24):411–432, April 2018. DOI: 10.1109/access.2018.2830661. 1
I. Goodfellow, Y. Bengio and A. Courville, Deep Learning, MIT Press, April 2016. http://www.deeplearningbook.org 1, 57
The official GDPR website https://ec.europa.eu/commission/priorities/justice-and-fundamental-rights/data-protection/2018-reform-eu-data-protection-rules_en 2
DLA Piper, Data protection laws of the world: Full handbook, September 2019. https://www.dlapiperdataprotection.com/ 2, 143, 147, 151, 152, 153
Q. Yang, Y. Liu, T. Chen, et al., Federated machine learning: Concept and applications, ArXiv Preprint ArXiv:1902.04885, February 2019. http://arxiv.org/abs/1902.04885 DOI: 10.1145/3298981. xiv, 2, 3, 5, 7, 8, 9, 10, 34, 50, 51, 66, 70, 71, 72, 73, 83, 85, 86, 151, 153
H. B. McMahan, E. Moore, D. Ramage, et al., Communication-efficient learning of deep networks from decentralized data, ArXiv Preprint ArXiv:1602.05629, February 2016a. https://arxiv.org/abs/1602.05629 3, 5, 12, 49, 50, 51, 52, 55, 56, 60, 69, 85
H. B. McMahan, E. Moore, D. Ramage, et al., Federated learning of deep networks using model averaging, February 2016b. https://pdfs.semanticscholar.org/8b41/9080cd37bdc30872b76f405ef6a93eae3304.pdf 3, 44, 45, 55, 56, 57, 58, 59, 60, 64, 118
H. Yu, S. Yang, and S. Zhu, Parallel restarted SGD with faster convergence and less communication: Demystifying why model averaging works for deep learning, ArXiv Preprint ArXiv:1807.06629, July 2018. https://arxiv.org/abs/1807.06629 DOI: 10.1609/aaai.v33i01.33015693. 12, 33, 52, 55, 60
J. Konecný, H. B. McMahan, F. X. Yu, et al., Federated learning: Strategies for improving communication efficiency, ArXiv Preprint ArXiv:1610.05492, October 2016a. http://arxiv.org/abs/1610.05492 3, 47, 63
J. Konecný, H. B. McMahan, D. Ramage, et al., Federated optimization: Distributed machine learning for on-device intelligence, ArXiv Preprint ArXiv:1610.02527, October 2016b. http://arxiv.org/abs/1610.02527 3, 33, 50, 65
F. Hartmann, Federated learning, Master thesis, Free University of Berlin, May 2018. http://www.mi.fu-berlin.de/inf/groups/ag-ti/theses/download/Hartmann_F18.pdf 3, 6, 11, 66
F. Hartmann, Federated learning, August 2018. https://florian.github.io/federated-learning/ 1, 56, 66
Y. Liu, Q. Yang, T. Chen, et al., Federated learning and transfer learning for privacy, security and confidentiality, The 33rd AAAI Conference on Artificial Intelligence (AAAI), January 2019. https://aisp-1251170195.file.myqcloud.com/fedweb/1552916850679.pdf 3, 5, 7, 71, 83
T. Yang, G. Andrew, H. Eichner, et al., Applied federated learning: Improving Google keyboard query suggestions, ArXiv Preprint ArXiv:1812.02903, December 2018. http://arxiv.org/abs/1812.02903 3, 11, 65
A. Hard, K. Rao, R. Mathews, et al., Federated learning for mobile keyboard prediction, ArXiv Preprint ArXiv:1811.03604, November 2018. http://arxiv.org/abs/1811.03604 3, 11, 65
Y. Zhao, M. Li, L. Lai, et al., Federated learning with non-IID data, ArXiv Preprint ArXiv:1806.00582, August 2018. http://arxiv.org/abs/1806.00582 6, 57
F. Sattler, S. Wiedemann, K. Müller, et al., Robust and communication-efficient federated learning from non-IID data, ArXiv Preprint ArXiv:1903.02891, March 2019. http://arxiv.org/abs/1903.02891 6, 56, 57
S. van Lier, Robustness of federated averaging for non-IID data, August 2018. https://www.cs.ru.nl/bachelors-theses/2018/Stan_van_Lier___4256166___Robustness_of_federated_averaging_for_non-IID_data.pdf 6
A. N. Bhagoji, S. Chakraborty, P. Mittal, et al., Analyzing federated learning through an adversarial lens, ArXiv Preprint ArXiv:1811.12470, March 2019. http://arxiv.org/abs/1811.12470 7, 11, 20
B. Han, An overview of federated learning, March 2019. https://medium.com/datadriveninvestor/an-overview-of-federated-learning-8a1a62b0600d 7, 12, 13
J. Mancuso, B. DeCoste, and G. Uhma, Privacy-preserving machine learning 2018: A year in review, January 2019. https://medium.com/dropoutlabs/privacy-preserving-machine-learning-2018-a-year-in-review-b6345a95ae0f 10, 17, 34, 145, 153
K. Cheng, T. Fan, Y. Jin, et al., Secureboost: A lossless federated learning framework, ArXiv Preprint ArXiv:1901.08755, January 2019. http://arxiv.org/abs/1901.08755 10, 42, 72, 76, 78, 79
Y. Liu, T. Chen, and Q. Yang, Secure federated transfer learning, ArXiv Preprint ArXiv:1812.03337, January 2018. http://arxiv.org/abs/1812.03337 10, 29, 48, 83, 85, 86, 89, 90
F. Chen, Z. Dong, Z. Li, et al., Federated meta-learning for recommendation, ArXiv Preprint ArXiv:1802.07876, February 2018. http://arxiv.org/abs/1802.07876 11, 118
D. Liu, T. Miller, R. Sayeed, et al., FADL: Federated-autonomous deep learning for distributed electronic health record, ArXiv Preprint ArXiv:1811.11400, November 2018. http://arxiv.org/abs/1811.11400 11
L. Huang and D. Liu, Patient clustering improves efficiency of federated machine learning to predict mortality and hospital stay time using distributed electronic medical records, ArXiv Preprint ArXiv:1903.09296, March 2019. http://arxiv.org/abs/1903.09296 DOI: 10.1016/j.jbi.2019.103291. 11
OpenMined. https://www.openmined.org/ 13
Horovod. https://github.com/horovod 12
T. Ryffel, A. Trask, M. Dahl, et al., A generic framework for privacy preserving deep learning, ArXiv Preprint ArXiv:1811.04017, November 2018. http://arxiv.org/abs/1811.04017 13
OpenMined/PySyft. https://github.com/OpenMined/PySyft 13, 144
T. Ryffel, Federated learning with PySyft and PyTorch, March 2019. https://blog.openmined.org/upgrade-to-federated-learning-in-10-lines/ 13
WeBank AI Department, Federated AI technology enabler (FATE). https://github.com/FederatedAI/FATE xv, 12, 14, 144
WeBank AI Department, The federated AI ecosystem project. https://www.fedai.org/#/ 12, 14, 143
Tensorflow.org, Tensorflow federated (TFF): Machine learning on decentralized data. https://www.tensorflow.org/federated 12, 64
A. Ingerman and K. Ostrowski, Introducing tensorflow federated, March 2019. https://medium.com/tensorflow/introducing-tensorflow-federated-a4147aa20041 12
Tensorflow/federated. https://github.com/tensorflow/federated 12
TensorFlow/Encrypted. https://github.com/tf-encrypted/tf-encrypted 12
coMindOrg/federated-averaging-tutorials. https://github.com/coMindOrg/federated-averaging-tutorials 12
IEEE P3652.1—Guide for architectural framework and application of federated machine learning. https://standards.ieee.org/project/3652_1.html and https://sagroups.ieee.org/3652-1/ xv, 14
The general data protection regulation (GDPR), April 2016. https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX:32016R0679&from=EN 146, 147, 148, 149
GDPR Info. https://gdpr-info.eu/ 145, 148, 149
EU GDPR.ORG. https://eugdpr.org/ 145
Overview of the general data protection regulation (GDPR). https://ico.org.uk/media/for-organisations/data-protection-reform/overview-of-the-gdpr-1-13.pdf 145, 148, 149
TechRepublic, GDPR: A cheat sheet. https://www.techrepublic.com/article/the-eu-general-data-protection-regulation-gdpr-the-smart-persons-guide/ 147
A. Kotsios, M. Magnani, L. Rossi, et al., An analysis of the consequences of the general data protection regulation (GDPR) on social network research, ArXiv Preprint ArXiv:1903.03196, March 2019. http://arxiv.org/abs/1903.03196 148, 149
A. Shah, V. Banakar, S. Shastri, et al., Analyzing the impact of GDPR on storage systems, ArXiv Preprint ArXiv:1903.04880, March 2019. http://arxiv.org/abs/1903.04880 154
A. Dasgupta and A. Ghosh, Crowdsourced judgement elicitation with endogenous proficiency, In WWW, pp. 319–330, 2013. DOI: 10.1145/2488388.2488417. 97
B. Faltings and G. Radanovic, Game Theory for Data Science: Eliciting Truthful Information, Morgan & Claypool Publishers, 2017. DOI: 10.2200/s00788ed1v01y201707aim035. 97
R. Jia, D. Dao, B. Wang, F. A. Hubis, N. Hynes, N. M. Gurel, B. Li, C. Zhang, D. Song and C. Spanos, Towards efficient data valuation based on the Shapley value, In PLMR, pp. 1167–1176, 2019. 98
Y. Kong and G. Schoenebeck, An information theoretic framework for designing information elicitation mechanisms that reward truth-telling, ACM Transactions on Economics and Computation, 7(1), article 2, 2019. DOI: 10.1145/3296670. 97
G. Radanovic, B. Faltings and R. Jurca, Incentives for effort in crowdsourcing using the peer truth serum, ACM Transactions on Intelligent Systems and Technology, 7(4), article 48, 2016. DOI: 10.1145/2856102. 98
A. Richardson, A. Filos-Ratsikas and B. Faltings, Rewarding high-quality data via influence functions, arXiv 1908.11598, 2019. 98
A. Singla and A. Krause, Truthful incentives in crowdsourcing tasks using regret minimization mechanisms, In WWW, pp. 1167–1178, 2013. DOI: 10.1145/2488388.2488490. 97
V. Shnayder, A. Agarwal, R. Frongillo, and D. C. Parkes, Informed truthfulness in multi-task peer prediction, In ACM EC, pp. 179–196, 2016. DOI: 10.1145/2940716.2940790. 97
S. Sirur, J. R. C. Nurse, and H. Webb, Are we there yet? understanding the challenges faced in complying with the general data protection regulation (GDPR), ArXiv Preprint ArXiv:1808.07338, September 2018. http://arxiv.org/abs/1808.07338 DOI: 10.1145/3267357.3267368.
University of Groningen, Understanding the GDPR. https://www.futurelearn.com/courses/general-data-protection-regulation/0/steps/32412 148
T. McGavisk, The positive and negative impact of GDPR. https://www.timedatasecurity.com/blogs/the-positive-and-negative-implications-of-gdpr 150, 151
D. Roe, Understanding GDPR and its impact on the development of AI, April 2018. https://www.cmswire.com/information-management/understanding-gdpr-and-its-impact-on-the-development-of-ai/ 151
J. Pierce, Privacy and cybersecurity: A global year-end review, December 2018. https://www.insideprivacy.com/data-privacy/privacy-and-cybersecurity-a-global-year-end-review/ 151, 153
The California consumer privacy act (CCPA). https://www.caprivacy.org/ DOI: 10.2307/j.ctvjghvnn. 152
Information security technology—Personal information security specification. http://www.gb688.cn/bzgk/gb/newGbInfo?hcno=4FFAA51D63BA21B9EE40C51DD3CC40BE 153
G. Liang and S. S. Chawathe, Privacy-preserving inter-database operations, In International Conference on Intelligence and Security Informatics, pp. 66–82, Springer, 2004. DOI: 10.1007/978-3-540-25952-7_6. 71, 76
M. Scannapieco, I. Figotin, E. Bertino, et al., Privacy preserving schema and data matching, In Proc. of the ACM SIGMOD International Conference on Management of Data, pp. 653–664, 2007. DOI: 10.1145/1247480.1247553. 71
R. Nock, S. Hardy, W. Henecka, et al., Entity resolution and federated learning get a federated resolution, ArXiv Preprint ArXiv:1803.04035, March 2018. http://arxiv.org/abs/1803.04035
S. J. Pan and Q. Yang, A survey on transfer learning, IEEE Transactions on Knowledge and Data Engineering, 22(10):1345–1359, 2010. DOI: 10.1109/tkde.2009.191. 9, 83, 84
S. J. Pan, I. W. Tsang, J. T. Kwok, and Q. Yang, Domain adaptation via transfer component analysis, Proc. of the 21st International Joint Conference on Artificial Intelligence, pp. 1187–1192, 2009. DOI: 10.1109/tnn.2010.2091281. 84
J. Augustine, N. Chen, E. Elkind, et al., Dynamics of profit-sharing games, Internet Mathematics, 1:1–22, 2015. DOI: 10.1080/15427951.2013.830164. 97
S. Barbara and M. Jackson, Maximin, leximin, and the protective criterion: Characterizations and comparisons, Journal of Economic Theory, 46(1):34–44, 1988. DOI: 10.1016/0022-0531(88)90148-2.
The Belmont report. Technical Report, National commission for the protection of human subjects of biomedical and behavioral research, Department of Health, Education and Welfare, United States Government Printing Office, Washington, DC, 1978. 101
G. Christodoulou, K. Mehlhorn, and E. Pyrga, Improving the price of anarchy for selfish routing via coordination mechanisms, In ESA, pp. 119–130, 2011. DOI: 10.1007/978-3-642-23719-5_11.
Regulation (EU) 2016/679 of the European Parliament and of the Council 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing directive 95/46/ec (general data protection regulation), Technical Report, European Union, 2016.
B. Faltings, G. Radanovic, and R. Brachman, Game Theory for Data Science: Eliciting Truthful Information, Morgan & Claypool Publishers, 2017. DOI: 10.2200/s00788ed1v01y201707aim035.
S. Gollapudi, K. Kollias, D. Panigrahi, et al., Profit sharing and efficiency in utility games, In ESA, pp. 1–16, 2017. 96, 97
K. Kollias and T. Roughgarden, Restoring pure equilibria to weighted congestion games, ACM Transactions on Economics and Computation, 3(4):21:1–21:24, 2015. DOI: 10.1145/2781678.
T. Luo, S. S. Kanhere, J. Huang, et al., Sustainable incentives for mobile crowdsensing: Auctions, lotteries, and trust and reputation systems, IEEE Communications Magazine, 55(3):68–74, 2017. DOI: 10.1109/mcom.2017.1600746cm.
J. R. Marden and A. Wierman, Distributed welfare games, Operations Research, 61(1):155–168, 2013. DOI: 10.1287/opre.1120.1137.
M. J. Neely, Stochastic Network Optimization with Application to Communication and Queueing Systems, Morgan & Claypool Publishers, 2010. DOI: 10.2200/s00271ed1v01y201006cnt007. 101
R. Shokri and V. Shmatikov, Privacy-preserving deep learning, In Proc. of the ACM SIGSAC Conference on Computer and Communications Security (CCS), pp. 1310–1321, October 2015. DOI: 10.1109/allerton.2015.7447103. 43, 44, 47, 49, 89
P. von Falkenhausen and T. Harks, Optimal cost sharing protocols for scheduling games, In Proc. of the 12th ACM Conference on Electronic Commerce (EC), pp. 285–294, June 2011. DOI: 10.1145/1993574.1993618.
S. Yang, F. Wu, S. Tang, et al., On designing data quality-aware truth estimation and surplus sharing method for mobile crowdsensing, IEEE Journal on Selected Areas in Communications, 35(4):832–847, 2017. DOI: 10.1109/jsac.2017.2676898. 96
H. Yu, C. Miao, Z. Shen, et al., Efficient task sub-delegation for crowdsourcing, In 29th AAAI Conference on Artificial Intelligence, pp. 1305–1311, February 2015. 101
H. Yu, C. Miao, C. Leung, et al., Mitigating herding in hierarchical crowdsourcing networks, Scientific Reports, 6(4), 2016. DOI: 10.1038/s41598-016-0011-6. 101
H. Yu, Z. Shen, C. Miao, et al., Building ethics into artificial intelligence, ArXiv Preprint ArXiv:1812.02953, December 2018. http://arxiv.org/abs/1812.02953 101
H. Yu, C. Miao, Y. Zheng, et al., Ethically aligned opportunistic scheduling for productive laziness, ArXiv Preprint ArXiv:1901.00298, January 2019. http://arxiv.org/abs/1901.00298 DOI: 10.1145/3306618.3314240. 101
S. Ruder, Neural Transfer Learning for Natural Language Processing, National University of Ireland, Galway, 2019. 92
E. Bagdasaryan, A. Veit, Y. Hua, et al., ImageNet: A large-scale hierarchical image database, In IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255, 2009. DOI: 10.1109/CVPR.2009.5206848. 92
H. B. McMahan and D. Ramage, Federated learning: Collaborative machine learning without centralized training data, April 2017. https://ai.googleblog.com/2017/04/federated-learning-collaborative.html 69, 139
K. Xu, W. Hu, J. Leskovec, et al., How powerful are graph neural networks?, ArXiv Preprint ArXiv:1810.00826, October 2018. http://arxiv.org/abs/1810.00826
D. Preuveneers, V. Rimmer, I. Tsingenopoulos, et al., Chained anomaly detection models for federated learning: An intrusion detection case study, In Applied Sciences, December 2018. DOI: 10.3390/app8122663. 21, 65, 140
Y. Zheng, F. Liu, and H. Hsieh, U-Air: When urban air quality inference meets big data, In Proc. of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), pp. 1436–1444, New York, 2013. https://doi.org/10.1145/2487575.2488188 DOI: 10.1145/2487575.2488188. 136
CNNIC publishes the 41st statistical report on China’s Internet development in China. https://www.lexology.com/library/detail.aspx?g=911ae57f-50da-4c53-ab75-2376272b2021 139
eMarketer publishes Worldwide Internet and mobile users: eMarketer’s updated estimates and forecast for 2017–2021. https://www.emarketer.com/Report/Worldwide-Internet-Mobile-Users-eMarketers-Updated-Estimates-Forecast-20172021/2002147 139
R. S. Sutton and A. G. Barto, Introduction to Reinforcement Learning, MIT Press, 1998. 121
G. A. Rummery and M. Niranjan, On-Line Q-Learning Using Connectionist Systems, Cambridge University Engineering Department, 1994. 124
C. Watkins and P. Dayan, Q-learning, In Machine Learning, pp. 279–292, 1992. DOI: 10.1007/bf00992698. 124
J. Chen, X. Pan, R. Monga, et al., Revisiting distributed synchronous SGD, March 2017. http://arxiv.org/abs/1604.00981 33, 51, 126
V. Mnih, A. P. Badia, M. Mirza, et al., Asynchronous methods for deep reinforcement learning, In Proc. of the 33rd International Conference on Machine Learning, pp. 1928–1937, June 2016. 125, 126
A. Nair, P. Srinivasan, S. Blackwell, et al., Massively parallel methods for deep reinforcement learning, July 2015. http://arxiv.org/abs/1507.04296 125
A. V. Clemente, H. N. Castejón, and A. Chandra, Efficient parallel methods for deep reinforcement learning, May 2017. http://arxiv.org/abs/1705.04862 126
V. Chen, V. Pastro, and M. Raykova, Secure computation for machine learning with SPDZ, January 2019. https://arxiv.org/abs/1901.00329 26
R. Cramer, I. Damgård, D. Escudero, et al., SPDZ2k: Efficient MPC mod 2k for dishonest majority, In Annual International Cryptology Conference, pp. 769–798, Springer, 2018. 26
R. Gilad-Bachrach, N. Dowlin, K. Laine, et al., CryptoNets: Applying neural networks to encrypted data with high throughput and accuracy, In International Conference on Machine Learning, pp. 201–210, 2016. 28
C. Juvekar, V. Vaikuntanathan, and A. Chandrakasan, Gazelle: A low latency framework for secure neural network inference, In USENIX Security Symposium, 2018. 29
D. Chai, L. Wang, K. Chen, and Yang, Secure federated matrix factorization, June 2019. https://arxiv.org/abs/1906.05108 29
N. Phan, X. Wu, and D. Dou, Preserving differential privacy in conutional deep belief networks, In Machine Learning, 106(9):1681–1704, October 2017. DOI: 10.1007/s10994-017-5656-2. 32
A. Triastcyn and B. Faltings, Generating differentially private datasets using GANs, February 2018. https://openreview.net/pdf?id=rJv4XWZA- 32
L. Yu, L. Liu, C. Pu, et al., Differentially private model publishing for deep learning, May 2019. https://arxiv.org/abs/1904.02200 DOI: 10.1109/sp.2019.00019. 32
X. Chen, T. Chen, H. Sun, et al., Distributed training with heterogeneous data: Bridging median- and mean-based algorithms, June 2019. https://arxiv.org/abs/1906.01736 56, 63
L. Li, W. Xu, T. Chen, et al., RSA: Byzantine-robust stochastic aggregation methods for distributed learning from heterogeneous datasets, November 2018. https://arxiv.org/abs/1811.03761 DOI: 10.1609/aaai.v33i01.33011544. 56
M. Duan, Astraea: Self-balancing federated learning for improving classification accuracy of mobile deep learning applications, July 2019. https://arxiv.org/abs/1907.01132 56
iResearch, Report on China’s smart cities development, 2019. https://www.iresearch.com.cn/Detail/report?id=3350&isfree=0 137
A. Chen, IBM’s Watson gave unsafe recommendations for treating cancer, July 2018. https://www.theverge.com/2018/7/26/17619382/ibms-watson-cancer-ai-healthcare-science 134
L. Mearian, Did IBM overhype Watson health’s AI promise?, November 2018. https://www.computerworld.com/article/3321138/did-ibm-put-too-much-stock-in-watson-health-too-soon.html 134
A. van den Oord, S. Dieleman, H. Zen, et al., WaveNet: A generative model for raw audio, September 2016. https://arxiv.org/abs/1609.03499 112
F. Baldimtsi, D. Papadopoulos, S. Papadopoulos, et al., Server-aided secure computation with off-line parties, In Computer Security—ESORICS, pp. 103–123, 2017. DOI: 10.1007/978-3-319-66402-6_8. 81
R. Bost, R. A. Popa, S. Tu, and S. Goldwasser, Machine learning classification over encrypted data, In NDSS, pp. 103–123, February 2015. DOI: 10.14722/ndss.2015.23241. 81
Covington and Burling LLP, Inside privacy: Updates on developments in data privacy and cybersecurity, July 2019. https://www.insideprivacy.com/uncategorized/china-releases-draft-measures-for-the-administration-of-data-security/ 154
H. Guo, R. Tang, Y. Ye, et al., DeepFM: A factorization-machine based neural network for CTR prediction, In Proc. of the 26th International Joint Conference on Artificial Intelligence, IJCAI, pp. 1725–1731, August 2017. https://doi.org/10.24963/ijcai.2017/239 DOI: 10.24963/ijcai.2017/239. 118
O. Habachi, M.-A. Adjif, and J.-P. Cances, Fast uplink grant for NOMA: A federated learning based approach, March 2019. https://arxiv.org/abs/1904.07975 141
S. Niknam, H. S. Dhillon, and J. H. Reed, Federated learning for wireless communications: Motivation, opportunities and challenges, September 2019. https://arxiv.org/abs/1908.06847 141
K. B. Letaief, W. Chen, Y. Shi, et al., The roadmap to 6G—AI empowered wireless networks, July 2019. https://arxiv.org/abs/1904.11686 DOI: 10.1109/mcom.2019.1900271. 141
Z. Zhou, X. Chen, E. Li, et al., Edge intelligence: Paving the last mile of artificial intelligence with edge computing, May 2019. https://arxiv.org/abs/1905.10083 DOI: 10.1109/jproc.2019.2918951. 141
G. Zhu, D. Liu, Y. Du, et al., Towards an intelligent edge: Wireless communication meets machine learning, September 2018. https://arxiv.org/abs/1809.00343 141
S. Samarakoon, M. Bennis, W. Saad, and M. Debbah, Federated learning for ultra-reliable low-latency V2V communication, In Proc. of the IEEE Globecom, 2018. DOI: 10.1109/glocom.2018.8647927. 141
E. Jeong, S. Oh, H. Kim, et al., Communication-efficient on-device machine learning: Federated distillation and augmentation under non-IID private data, NIPS Workshop, Montreal, Canada, 2018. 141
M. Bennis, Trends and challenges of federated learning in the 5G network, July 2019. https://www.comsoc.org/publications/ctn/edging-towards-smarter-network-opportunities-and-challenges-federated-learning 141
J. Park, S. Samarakoon, M. Bennis, and M. Debbah, Wireless network intelligence at the edge, September 2019. https://arxiv.org/abs/1812.02858 DOI: 10.1109/jproc.2019.2941458. 141
Q. Li, Z. Wen, and B. He, Federated learning systems: Vision, hype and reality for data privacy and protection, July 2019a. http://arxiv.org/abs/1907.09693 66
T. Li, A. K. Sahu, A. Talwalkar, and V. Smith, Federated learning: Challenges, methods, and future directions, August 2019. https://arxiv.org/abs/1908.07873 66
F. Mo and H. Haddadi, Efficient and private federated learning using TEE, March 2019. https://eurosys2019.org/wp-content/uploads/2019/03/eurosys19posters-abstract66.pdf 21
R. C. Geyer, T. Klein, and M. Nabi, Differentially private federated learning: A client level perspective, March 2018. https://arxiv.org/abs/1712.07557 32
M. Al-Rubaie and J. M. Chang, Reconstruction attacks against mobile-based continuous authentication systems in the cloud, In IEEE Transactions on Information Forensics and Security, 11(12):2648–2663, 2016. DOI: 10.1109/tifs.2016.2594132. 20
M. Fredrikson, S. Jha, and T. Ristenpart, Model inversion attacks that exploit confidence information and basic countermeasures, In Proc. of the 22nd ACM SIGSAC Conference on Computer and Communications Security, pages 1322–1333, 2015. DOI: 10.1145/2810103.2813677. 19
P. Xie, B. Wu, and G. Sun, BAYHENN: Combining Bayesian deep learning and homomorphic encryption for secure DNN inference, In Proc. of the 28th International Joint Conference on Artificial Intelligence, IJCAI, pages 4831–4837, Macao, China, August 10–16, 2019. DOI: 10.24963/ijcai.2019/671. 20
R. Shokri, M. Stronati, C. Song, and V. Shmatikov, Membership inference attacks against machine learning models, In IEEE Symposium on Security and Privacy (SP), pages 3–18, 2017. DOI: 10.1109/sp.2017.41. 20
D. Wang, L. Zhang, N. Ma, and Xiaobo Li, Two secret sharing schemes based on Boolean operations, Pattern Recognition, 40(10):2776–2, 2017. DOI: 10.1016/j.patcog.2006.11.018. 24
K. Xu, H. Mi, D. Feng, et al., Collaborative deep learning across multiple data centers, October 2018. https://arxiv.org/abs/1810.06877 52, 55
I. Cano, M. Weimer, D. Mahajan, et al., Towards geo-distributed machine learning, March 2016. https://arxiv.org/abs/1603.09035 52, 55
A. Reisizadeh, A. Mokhtari, H. Hassani, et al., Fed PAQ: A Communication-efficient federated learning method with periodic averaging and quantization, October 2019. https://arxiv.org/abs/1909.13014 63
L. Wang, W. Wang, and B. Li, CMFL: Mitigating communication overhead for federated learning, In Proc. of the 39th IEEE International Conference on Distributed Computing Systems (ICDCS), July 2019. 64
K. Hsieh, A. Harlap, N. Vijaykumar, et al., Gaia: Geo-distributed machine learning approaching LAN speeds, In NSDI, pp. 629–647, 2017. 52, 64
A. Zhang, Z. C. Lipton, M. Li, and A.J. Smola, Dive into deep learning, October 2019. https://en.d2l.ai/d2l-en.pdf 57
T.-Y. Liu, W. Chen, T. Wang, and F. Gao, Distributed Machine Learning: Theories, Algorithms, and Systems, China Machine Press, September 2018. (In Chinese, ISBN 978-7-111-60918-6.) 33
J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, You only look once: Unified, real-time object detection, May 2016. Available: https://arxiv.org/abs/1506.02640 109
H. Yu, Z. Liu, Y. Liu, T. Chen, M. Cong, X. Weng, D. Niyato, and Q. Yang, A fairness-aware incentive scheme for federated earning, In Proc. of the 3rd AAAI/ACM Conference on Artificial Intelligence, Ethics, and Society (AIES-20), 2020. 95
F. Haddadpour, M. M. Kamani, M. Mahdavi, and V. R. Cadambe, Local SGD with periodic averaging: Tighter analysis and adaptive synchronization, October 2019. https://arxiv.org/abs/1910.13598 55
P. Kairouz, H.B. McMahan, B. Avent, et al., Advances and open problems in federated learning, December 2019. https://arxiv.org/abs/1912.04977 7, 49