view · edit · attach · print · history

Large Circle-in-the-Square (CIS) dataset: train on 1,000 points

Benchmark data

CN550 Spring 2009

CIS Large - train on 1,000 points

Format for CIS results

# Name System Parameters-Settings % Correct C-index Runtime Details Link
1 Best All predictions correct   100 1.0 0   
2 Worst All predictions wrong   0 0 0   
3 All IN All predictions IN   50 0.5 0   
4 All OUT All predictions OUT   50 0.5 0   
5 Chance Random IN/OUT   50 0.5 0   
6 Your name System Parameters-Settings % Correct C-index Runtime Details Link  

Format for CIS confusion matrices

# CIS confusion matrix # actual IN # actual OUT total
1 # predicted IN TP: true IN FP: false IN TP + FP
2 # predicted OUT FN: false OUT TN: true OUT TN + FN
3 total 4,997 5,003 10,000

CN550 Spring 2008

# Name System Parameters-Settings % Correct C-index Runtime Details Link
1 Best All predictions correct   100 1.0 0   
2 Worst All predictions wrong   0 0 0   
3 All IN All predictions IN   50 0.5 0   
4 All OUT All predictions OUT   50 0.5 0   
5 Chance Random IN/OUT   50 0.5 0   
6 Jeff SVM libsvm package, poly kernel (3rd), qp 98.92 .978 644.3sec (train) .704sec (test) Details  
7 Peifeng Fuzzy ARTMAP learning rate = 1 baseline vigilance = 0.7 95.75 .916 1.3824sec (train) 2.764sec (test) Details  
8 Peifeng SVM Using Platt's SMO C = positive infinity, Kernel = Gaussian RBF, using matlab with C++ mex 98.81 .976 .9205sec (train) .0216sec (test) Details  
9 Jeff MADALINE MRII algorithm, learning rate =.4 52 .2553 11.1sec (train) .435sec (test) Details  
10 Neel Simplified Fuzzy ARTMAP (Kasuba, 1993) learning rate = 1 baseline vigilance = 0.75 95.17 0.956 8.9 seconds (training) 3.8 seconds (test) Details  
11 Rohit KNN K=2, City block metric 97.12 0.9802 4.9 sec (train + test) Details  
12 Roger Fuzzy Artmap learning rate=1, baseline vigilance=0+e 95.17   36.938 sec   
13 Sean KNN K=1, City block metric 97.12 0.9802 67.27 sec Details
14 Charles Cascade Correlation In-house code, fixed cascades, epochs (10 and 100, respectively), eta = 0.5 (arbitrary), error threshold = 0.5 82.41 0.8237 1137.74 sec Details
15 Melissa Backprop w/Momentum learning rate = 0.25, momentum = 0.80 97.67 0.9534 451.64 seconds Details
16 Todd SVM libSVM with Polynomial Kernel (3rd order) 99.13 0.9943 0.044s (trn) 0.07s (test) Details
17 Karthik MADALINE In-House code MRII, LMS, rate=0.5 51 0.25 20s (train) 5s(test) Performance is closer to guess work Results
18 Cloud Backprop w/Momentum Neurons in the hidden layer: 20, learning rate = 0.15, momentum = 0.6 (one possible selection) 96   29.445s (train) Details
19 Rohit fuzzy ARTMAP learning rate = 1, baseline vigilance = 0.9 96.16 0.9725 3.994 sec (train) + 42.1505 sec (test) Details  
20 Charles Maximum Likelihood In-house code, Severely Hacked (see details) 80.89 0.8035 2.273 sec Details
21 Cloud Fuzzy KNN k=1, L2 97   7.884s (train+cross validation+test) Details
22 Melissa PNN sigma = 0.62 78.58 0.5991 3.79 seconds Details
23 Roger PNN spread= 0.005 97.14   44.444 seconds  
24 Neel Genetic Simplified Fuzzy ARTMAP population size = 80, generations = 2000 90.67 0.724 2 hours (training) 1.2 seconds (test) Details  
25 Joe Backprop 2000 epochs, eta = .5 98   584.99 seconds (training + testing)  
26 Joe Maximum Likelihood 1 dimension 92   seconds (training + testing)  
27 Karthik SVM svmlite,polynomial kernels 96 .93 5 seconds (train) 1.2 seconds (test) works fine for kernels of order 3 or more Results
28 Ben kNN k=1 97.12   ~0.3 seconds Details  
29 Todd EBF Centers = 1 97.56 0.9821 0.331s (trn) 0.672s (test) Details
30 Sean Fuzzy ARTMAP beta=1; rho=0.9 96.16 0.9725 2.715 sec (train); 26.71 sec (test) Details
31 Sean Radial Basis Function RBF Nodes=25; sigma=0.7 98.94   24.47 sec Details

6 -Jeff (SVM)

# CIS 6 - Jeff # actual positive (+) # actual negative (–) ____total____
1 # predicted + 4897 8 4905
2 # predicted – 100 4995 5095
3 total 4997 5003 10000

9 -Jeff (MADALINE)

# CIS 9 - Jeff # actual positive (+) # actual negative (–) ____total____
1 # predicted + 3383 1887 5284
2 # predicted – 1614 3116 4716
3 total 4997 5003 10000

7 - Peifeng (Fuzzy ARTMAP)

# CIS 7 - Peifeng # actual IN # actual OUT total
1 # predicted IN 4720 148 4868
2 # predicted OUT 277 4855 5132
3 total 4997 5003 10000

8 - Peifeng(SVM Using Platt's SMO)

# CIS 8 - Peifeng # actual IN # actual OUT total
1 # predicted IN 4885 7 4892
2 # predicted OUT 112 4996 5118
3 total 4997 5003 10000

10 - Neel (Simplified Fuzzy ARTMAP, (Kasuba, 1993))

# confusion matrix # actual positive (+) # actual negative (–) ____total____
1 # predicted + 4701 187 4888
2 # predicted – 296 4816 5112
3 total 5121 4879 10000

11 - Rohit (KNN, K=2, City block metric)

# confusion matrix # actual positive (+) # actual negative (-) ____total____
1 # predicted + 4918 85 5003
2 # predicted - 203 4794 4997
3 total 5121 4879 10000

12 - Roger (Fuzzy Artmap)

# confusion matrix # actual positive (+) # actual negative (-) ____total____
1 # predicted + 4701 187 4898
2 # predicted - 296 4816 5112
3 total 4997 5003 10000

13 - Sean (KNN, K=1, City block metric)

# confusion matrix # actual IN # actual OUT total
1 # predicted IN 4794 203 4892
2 # predicted OUT 85 4918 5118
3 total 4879 5121 10000

14 - Charles (Cascade Correlation, 10 Cascades)

# confusion matrix # actual IN # actual OUT total
1 # predicted IN 4047 809 4856
2 # predicted OUT 950 4194 5144
3 total 4997 5003 10000

15 - Melissa (Backprop with Momentum)

# CIS confusion matrix # actual IN # actual OUT total
1 # predicted IN TP: 4764 FP: 0 4764
2 # predicted OUT FN: 233 TN: 5003 5236
3 total 4997 5003 10000

16 - Todd (SVM)

# confusion matrix # actual positive (+) # actual negative (–) __total__
1 # predicted + 4928 18 4946
2 # predicted – 69 4985 5054
3 total 4997 5003 10000

18 - Cloud (Backprop with Momentum)

# CIS confusion matrix # actual IN # actual OUT total
1 # predicted IN TP: 4897 FP: 300 5197
2 # predicted OUT FN: 100 TN: 4703 4803
3 total 4997 5003 10000

19 - Rohit (fuzzy ARTMAP, baseline vigilance = 0.9, learning rate = 1)

# confusion matrix # actual positive (+) # actual negative (-) ____total____
1 # predicted + 4927 76 5003
2 # predicted - 308 4689 4997
3 total 5235 4765 10000

20 - Charles (Maximum Likelihood)

# confusion matrix # actual IN # actual OUT total
1 # predicted IN 4754 1668 6422
2 # predicted OUT 243 3335 3578  
3 total 4997 5003 10000

21 - Cloud (Fuzzy KNN)

# confusion matrix # actual IN # actual OUT total
1 # predicted IN 4847 135 4982
2 # predicted OUT 150 4868 5018
3 total 4997 5003 10000

22 - Melissa (PNN)

# CIS confusion matrix # actual IN # actual OUT total
1 # predicted IN TP: 3252 FP: 397 3649
2 # predicted OUT FN: 1745 TN: 4606 6351
3 total 4997 5003 10000

23 - Roger (PNN)

# CIS confusion matrix # actual IN # actual OUT total
1 # predicted IN TP: 4798 FP: 87 4885
2 # predicted OUT FN: 199 TN: 4916 5115
3 total 4997 5003 10000

24 - Neel (GFAM)

# CIS confusion matrix # actual IN # actual OUT total
1 # predicted IN TP: 4257 FP: 193 4450
2 # predicted OUT FN: 740 TN: 4810 5550
3 total 4997 5003 10000

25 - Joe (Backprop)

# CIS confusion matrix # actual IN # actual OUT total
1 # predicted IN TP: 4877 FP: 77 4954
2 # predicted OUT FN: 120 TN: 4926 5046
3 total 4997 5003 10000

25 - Joe (Maximum Likelihood)

# CIS confusion matrix # actual IN # actual OUT total
1 # predicted IN TP: 5003 FP: 803 5806
2 # predicted OUT FN: 0 TN: 4194 4194
3 total 5003 4997 10000

26 -Karthik (MADALINE)

# CIS confusion matrix # actual positive (+) # actual negative (–) ____total____
1 # predicted + 3368 1904 5272
2 # predicted – 1634 3094 4728
3 total 5002 4998 10000

27 -Karthik (SVMlite)

# CIS confusion matrix # actual positive (+) # actual negative (–) ____total____
1 # predicted + 4897 68 4965
2 # predicted – 100 4935 5035
3 total 4997 5003 10000

28 - Ben (kNN k=1)

# CIS confusion matrix # actual positive (+) # actual negative (–) ____total____
1 # predicted + 4794 85 4879
2 # predicted – 203 4918 5121
3 total 4997 5003 10000

29 - Todd (Elliptical Basis Function)

# confusion matrix # actual positive (+) # actual negative (–) __total__
1 # predicted + 4830 77 4907
2 # predicted – 167 4926 5093
3 total 4997 5003 10000

30 - Sean (Fuzzy ARTMAP)

# CIS confusion matrix # actual IN # actual OUT total
1 # predicted IN 4927 76 5003
2 # predicted OFF 308 4689 4997
3 total 5235 4765 10000

31 - Sean (Radial Basis Function)

# CIS confusion matrix # actual IN # actual OUT total
1 # predicted IN 4911 86 4997
2 # predicted OUT 20 4983 5003
3 total 4931 5069 10000

CN550 Spring 2007

# SVM Name % Correct C-index More details
1 Gaussian kSVM Jing 98.68   Methods and Results
2 SVM (Platt) Andy 99.06   details
3 SVM (Osuna) Eugene    
4 SVM Paramesh 98.43   Results

# ARTMAP Name % Correct C-index More details
1 Fuzzy ARTMAP Tim 89.3 0.86 Vote Vis Vigilance
2 Fuzzy ARTMAP Gary 96.99 0.94 result parameters
3 Fuzzy ARTMAP Yohan 97.20 0.94 details
4 Distr ARTMAP Rob 95.3   
5 GA+Distr ARTMAP Jesse 95.12   519 nodes, Dstrdness: 0.706% (~3.67 nodes)
6 Default ARTMAP Hee 95.91 0.90 Details_DefA_CIS
7 Default ARTMAP Paramesh 95.96 0.9203 Results
8 ARTMAP-IC Jeff 96.89   

# KNN Name % Correct C-index More details
1 KNN Andy 97.17 0.94 details
2 KNN Hee 97.13 0.94 Details_KNN

# Bayesian Est. Name % Correct C-index More details
1 Bayesian Est. Jing Xia 91.97 0.84 result
2 Bayesian Est. Tim 92 0.84 result

# RBF Name % Correct C-index More details
1 RBF Network Jing Bai 97.6 0.95 Methods and Results
2 RBF Jing Xia 97.66 0.95 method result
view · edit · attach · print · history
Page last modified on January 21, 2009, at 04:17 AM