adaptive moment estimation
Hyperparameter
It cannot be learnt from the training dataset.
Validation set
While test dataset is used to estimate the generalization error of a model, validation dataset is used to choose the model, i.e., determining hyperparameters. It is constructed from the training dataset.
Variance and Standard Variance
The variance or standard error, of an estimator provides a measure of how we would expect the estimator we compute from data to vary as we independently resample the dataset from the underlying data-generating process.
Bias
where the expectation is over the data (seen as samples from a random variable) and $\hat \theta$ is the true underlying value of $\theta$ used to define the data-generating distribution.
Bias and variance measure two different sources of error in an estimator. Bias measures the expected deviation from the true value of the function or parameter. Variance on the other hand, provides a measure of the deviation from the expected estimator value that any particular sampling of the data is likely to cause.