Advances in learning theory: methods, models, and applications

Contents......Page 16
Preface......Page 6
Organizing committee......Page 10
List of chapter contributors......Page 12
1 An Overview of Statistical Learning Theory......Page 24
1.1.3 Three main learning problems......Page 25
1.1.5 Empirical risk minimization principle and the classical methods......Page 27
1.1.6 Four parts of learning theory......Page 28
1.2.1 The key theorem of the learning theory......Page 29
1.2.2 The necessary and sufficient conditions for uniform convergence......Page 30
1.2.3 Three milestones in learning theory......Page 32
1.3 Bounds on the Rate of Convergence of the Learning Processes......Page 33
1.3.2 Equivalent definition of the VC dimension......Page 34
1.3.3 Two important examples......Page 35
1.3.4 Distribution independent bounds for the rate of convergence of learning processes......Page 36
1.3.5 Problem of constructing rigorous (distribution dependent) bounds......Page 37
1.4.1 Structural risk minimization induction principle......Page 38
1.5.1 Methods of separating hyperplanes and their generalization......Page 40
1.5.2 Sigmoid approximation of indicator functions and neural nets......Page 41
1.5.3 The optimal separating hyperplanes......Page 42
1.5.4 The support vector network......Page 44
1.5.5 Why can neural networks and support vectors networks generalize?......Page 46
1.6 Conclusion......Page 47
2 Best Choices for Regularization Parameters in Learning Theory: On the Bias-Variance Problem......Page 52
2.2 RKHS and Regularization Parameters......Page 53
2.3 Estimating the Confidence......Page 55
2.4 Estimating the Sample Error......Page 61
2.5 Choosing the optimal γ......Page 63
2.6 Final Remarks......Page 64
3 Cucker Smale Learning Theory in Besov Spaces......Page 70
3.2 Cucker Smale Functional and the Peetre K-Functional......Page 71
3.3 Estimates for the CS-Functional in Anisotropic Besov Spaces......Page 75
4 High-dimensional Approximation by Neural Networks......Page 92
4.1 Introduction......Page 93
4.2 Variable-basis Approximation and Optimization......Page 94
4.3 Maurey-Jones-Barron's Theorem......Page 96
4.4 Variation with respect to a Set of Functions......Page 98
4.5 Rates of Approximate Optimization over Variable Basis Functions......Page 100
4.6 Comparison with Linear Approximation......Page 102
4.7 Upper Bounds on Variation......Page 103
4.8 Lower Bounds on Variation......Page 105
4.9 Rates of Approximation of Real-valued Boolean Functions......Page 106
5 Functional Learning through Kernels......Page 112
5.1 Some Questions Regarding Machine Learning......Page 113
5.2.2 r.k.h.s and learning in the literature......Page 114
5.3.1 The learning problem......Page 115
5.3.3 Continuity of the evaluation functional......Page 116
5.3.5 R[sup(χ)] the set of the pointwise denned functions on χ......Page 117
5.4 Reproducing Kernel Hilbert Space (r.k.h.s)......Page 118
5.5.1 How to build r.k.h.s?......Page 120
5.5.2 Carleman operator and the regularization operator......Page 121
5.6.1 Evaluation spaces......Page 122
5.6.2 Reproducing kernels......Page 123
5.7 Representer Theorem......Page 127
5.8.1 Examples in Hilbert space......Page 128
5.9 Conclusion......Page 130
6 Leave-one-out Error and Stability of Learning Algorithms with Applications......Page 134
6.1 Introduction......Page 135
6.2 General Observations about the Leave-one-out Error......Page 136
6.3.1 Early work in non-parametric statistics......Page 139
6.3.2 Relation to VC-theory......Page 140
6.3.3 Stability......Page 141
6.4 Kernel Machines......Page 142
6.4.1 Background on kernel machines......Page 143
6.4.2 Leave-one-out error for the square loss......Page 144
6.4.3 Bounds on the leave-one-out error and stability......Page 145
6.5.2 Feature selection and rescaling......Page 146
6.6.2 Open problems......Page 147
7 Regularized Least-Squares Classification......Page 154
7.1 Introduction......Page 155
7.2 The RLSC Algorithm......Page 157
7.3 Previous Work......Page 158
7.4 RLSC vs. SVM......Page 159
7.5 Empirical Performance of RLSC......Page 160
7.6 Approximations to the RLSC Algorithm......Page 162
7.6.1 Low-rank approximations for RLSC......Page 164
7.6.2 Nonlinear RLSC application: image classification......Page 165
7.7 Leave-one-out Bounds for RLSC......Page 169
8 Support Vector Machines: Least Squares Approaches and Extensions......Page 178
8.1 Introduction......Page 179
8.2.1 LS-SVM classifiers and link with kernel FDA......Page 181
8.2.3 Issues of sparseness and robustness......Page 184
8.3.1 Kernel PCA as a one-class modelling problem and a primal-dual derivation......Page 186
8.3.2 A support vector machine formulation to Kernel CCA......Page 189
8.4.1 Nyström method......Page 191
8.4.2 Basis construction in the feature space using fixed size LS-SVM......Page 192
8.5 Recurrent Networks and Control......Page 195
8.6 Conclusions......Page 196
9 Extension of the ν-SVM Range for Classification......Page 202
9.1 Introduction......Page 203
9.2 ν Support Vector Classifiers......Page 204
9.3 Limitation in the Range of ν......Page 208
9.4 Negative Margin Minimization......Page 209
9.5 Extended ν-SVM......Page 211
9.5.1 Kernelization in the dual......Page 212
9.6 Experiments......Page 214
9.7 Conclusions and Further Work......Page 217
10 Kernels Methods for Text Processing......Page 220
10.2 Overview of Kernel Methods......Page 221
10.3 From Bag of Words to Semantic Space......Page 222
10.4 Vector Space Representations......Page 224
10.4.1 Basic vector space model......Page 226
10.4.3 Semantic smoothing for vector space models......Page 227
10.4.4 Latent semantic kernels......Page 228
10.4.5 Semantic diffusion kernels......Page 230
10.5 Learning Semantics from Cross Language Correlations......Page 234
10.6 Hypertext......Page 238
10.7 String Matching Kernels......Page 239
10.7.1 Efficient computation of SSK......Page 242
10.8 Conclusions......Page 243
11 An Optimization Perspective on Kernel Partial Least Squares Regression......Page 250
11.1 Introduction......Page 251
11.2.1 PGA regression review......Page 252
11.2.2 PLS analysis......Page 254
11.2.3 Linear PLS......Page 255
11.2.4 Final regression components......Page 257
11.3.1 Feature space K-PLS......Page 259
11.3.2 Direct kernel partial least squares......Page 260
11.4 Computational Issues in K-PLS......Page 261
11.5.1 Methods......Page 262
11.5.3 Data preparation and parameter tuning......Page 263
11.5.4 Results and discussion......Page 264
11.7 Feature Selection with K-PLS......Page 266
11.8 Thoughts and Conclusions......Page 268
12 Multiclass Learning with Output Codes......Page 274
12.1 Introduction......Page 275
12.2 Margin-based Learning Algorithms......Page 276
12.3 Output Coding for Multiclass Problems......Page 280
12.4 Training Error Bounds......Page 283
12.5 Finding Good Output Codes......Page 285
12.6 Conclusions......Page 286
13 Bayesian Regression and Classification......Page 290
13.1.1 Least squares regression......Page 291
13.1.3 Probabilistic models......Page 292
13.1.4 Bayesian regression......Page 294
13.2 Support Vector Machines......Page 295
13.3.1 Model specification......Page 296
13.3.2 The effective prior......Page 298
13.3.3 Inference......Page 299
13.3.4 Making predictions......Page 300
13.3.5 Properties of the marginal likelihood......Page 301
13.3.6 Hyperparameter optimization......Page 302
13.3.7 Relevance vector machines for classification......Page 303
13.4.1 Illustrative synthetic data: regression......Page 304
13.4.2 Illustrative synthetic data: classification......Page 306
13.4.3 Benchmark results......Page 307
13.5 Discussion......Page 308
14 Bayesian Field Theory: from Likelihood Fields to Hyperfields......Page 312
14.2.1 The basic probabilistic model......Page 313
14.2.2 Bayesian decision theory and predictive density......Page 314
14.2.3 Bayes' theorem: from prior and likelihood to the posterior......Page 316
14.3.1 Log–probabilities, energies, and density estimation......Page 318
14.3.2 Regression......Page 320
14.3.3 Inverse quantum theory......Page 321
14.4.1 Gaussian prior factors and approximate symmetries......Page 322
14.4.2 Hyperparameters and hyperfields......Page 326
14.4.3 Hyperpriors for hyperfields......Page 331
14.4.4 Auxiliary fields......Page 332
14.5 Summary......Page 335
15 Bayesian Smoothing and Information Geometry......Page 342
15.1 Introduction......Page 343
15.2 Problem Statement......Page 344
15.3 Probability-Based Inference......Page 345
15.4 Information-Based Inference......Page 347
15.5 Single-Case Geometry......Page 350
15.6 Average-Case Geometry......Page 354
15.7 Similar-Case Modeling......Page 355
15.8 Locally Weighted Geometry......Page 359
15.9 Concluding Remarks......Page 360
16 Nonparametric Prediction......Page 364
16.2 Prediction for Squared Error......Page 365
16.3 Prediction for 0 – 1 Loss: Pattern Recognition......Page 369
16.4 Prediction for Log Utility: Portfolio Selection......Page 371
17 Recent Advances in Statistical Learning Theory......Page 380
17.2.1 Uniform convergence of empirical means......Page 381
17.2.2 Probably approximately correct learning......Page 383
17.3.1 Fixed distribution case......Page 385
17.3.2 Distribution-free case......Page 387
17.4.1 Intermediate families of probability measures......Page 388
17.4.2 Learning with prior information......Page 389
17.5.1 Problem formulations......Page 390
17.5.2 Definition of β-mixing......Page 391
17.5.3 UCEM and PAC learning with β-mixing inputs......Page 392
17.6 Applications to Learning with Inputs Generated by a Markov Chain......Page 394
17.7 Conclusions......Page 395
18 Neural Networks in Measurement Systems (an engineering view)......Page 398
18.1 Introduction......Page 399
18.2 Measurement and Modeling......Page 400
18.3 Neural Networks......Page 406
18.4 Support Vector Machines......Page 412
18.5 The Nature of Knowledge, Prior Information......Page 416
18.6 Questions Concerning Implementation......Page 417
18.7 Conclusions......Page 419
List of participants......Page 426
E......Page 434
L......Page 435
R......Page 436
W......Page 437
Author Index......Page 438