Deep learning is the process of learning nonlinear features and functions from complex data. Deep learning has been shown to outperform traditional techniques for speech recognition, image recognition, and face detection. Deep learning aims to extract complex features from high-dimensional data and use them to build a model that relates inputs to outputs (e.g., classes). Deep learning architectures are usually constructed as multi-layer networks so that more abstract features are computed as nonlinear functions of lower-level features.
Privacy in deep learning consists of three aspects: privacy of the data used for learning a model or as input to an existing model, privacy of the model, and privacy of the model’s output.
This paper provides a detailed and clear background. [2015-Reza Shokri] design, implement, and evaluate a practical system that enables multiple parties to jointly learn an accurate neural network model for a given objective without sharing their input datasets, their system achieves all three privacy objectives in the context of collaborative neural-network training: it protects privacy of the training data, enables participants to control the learning objective and how much to reveal about their individual models, and lets them apply the jointly learned model to their own inputs without revealing the inputs or the outputs.
[2016-Martín Abadi] develop differential privacy SGD for neural network by applying gaussian noise with tighter error bounds in centralized setting.
[2017-NhatHai Phan] criticizes all three methods above and aiming at following three objects:
- It is totally independent of the number of training epochs in consuming privacy budget
- It has the ability to adaptively inject noise into features based on the contribution of each to the model output
- It can be applied in a variety of deep neural networks
[2016-Martín Abadi]
The following summary is from paper [2017-NhatHai Phan].
Reference
[2015-Reza Shokri] Privacy-Preserving Deep Learning
[2016-Martín Abadi] Deep Learning with Differential Privacy
[2016-NhatHai Phan] Differential privacy preservation for deep auto-encoders: an application of human behavior prediction
[2017-NhatHai Phan] Adaptive Laplace Mechanism: Differential Privacy
Preservation in Deep Learning