view · edit · attach · print · history

Small Circle-in-the-Square (CIS) dataset: train on 100 points

Benchmark data

CN550 Spring 2009

Format for CIS results

CIS Small - train on 100 points

# Name System Parameters-Settings % Correct C-index Runtime Details Link
1 Best All predictions correct   100 1.0 0   
2 Worst All predictions wrong   0 0 0   
3 All IN All predictions IN   50 0.5 0   
4 All OUT All predictions OUT   50 0.5 0   
5 Chance Random IN/OUT   50 0.5 0   

Format for CIS confusion matrices

# CIS confusion matrix # actual IN # actual OUT total
1 # predicted IN TP: true IN FP: false IN TP + FP
2 # predicted OUT FN: false OUT TN: true OUT TN + FN
3 total 4,997 5,003 10,000

1 - Best

# CIS 1 - Best # actual IN # actual OUT total
1 # predicted IN 4,997 0 4,997
2 # predicted OUT 0 5,003 5,003
3 total 4,997 5,003 10,000

2 - Worst

# CIS 2 - Worst # actual IN # actual OUT total
1 # predicted UP 0 5,003 5,003
2 # predicted DOWN 4,997 0 4,997
3 total 4,997 5,003 10,000

3 - All IN

# CIS 3 - All IN # actual IN # actual OUT total
1 # predicted IN 4,997 5,003 10,000
2 # predicted OUT 0 0 0
3 total 4,997 5,003 10,000

4 - All OUT

# CIS 4 - All OUT # actual IN # actual OUT total
1 # predicted IN 0 0 0
2 # predicted OUT 4,997 5,003 10,000
3 total 4,997 5,003 10,000

5 -Chance

# CIS 5 - Chance # actual IN # actual OUT total
1 # predicted IN ~2,498 ~2,502 ~5,000
2 # predicted OUT ~2,499 ~2,501 ~5,000
3 total 4,997 5,003 10,000

CN550 Spring 2008

6 Jeff SVM libSVM package, poly kernel (3rd), qp 97.06 .942 .963 seconds (train) .659 seconds (test) Details  
7 Peifeng Fuzzy ARTMAP learning rate = 1 baseline vigilance = 0.72 92.55 0.855 .0954sec (train) 2.1445sec (test) Details
8 Peifeng SVM Using Platt's SMO C = positive infinity, Kernel = Gaussian RBF, Matlab with c++ mex 97.29 0.946 .1371sec (train) .018sec (test) Details
9 Jeff MADALINE MRII algorithm (LMS-a), learning rate = .3 54.5 .291 1.2 seconds (training) .549 seconds (test) Details  
10 Neel Simplified Fuzzy ARTMAP (Kasuba, 1993) learning rate = 1 baseline vigilance = 0.75 86.76 0.8794 0.5 seconds (training) 2.5 seconds (test) Details
11 Rohit KNN K=2 95.55 0.9654 0.7699 sec (training + testing) Details
12 Sean KNN K=1 95.55 0.9654 6.65 sec Details
13 Charles Cascade Correlation In-house Code, Fixed Iterations/Cascades (10 Cascades, 100 epochs each), eta = 0.5 (arbitrary), error Threshold = 0.5 66.90 0.6604 111.144 sec Details
14 Melissa Backprop w/Momentum learning rate = 0.25, momentum = 0.8 97.45 0.9493 39.95 secondsDetails
15 Todd SVM libSVM using Polynomial kernel (3rd) 97.38 .9827 0.006s (Trn) 0.027s (Test) Details
16 Karthik Madaline inhouse code MRII Learning Law,LMS, l = 0.4 52 0.256 3.0 s(training) 1.2 s (test) Chance level, just breaks even Results
17 Cloud Backprop w/Momentum Neurons in the hidden layer: 20, learning rate = 0.15, momentum = 0.8 (one possible selection) 91   9.047s (train) I report the results in a slightly different way because of the random initilization process of BP network. Please see the Details
18 Rohit fuzzy ARTMAP learning rate = 1 baseline vigilance = 0.9 87.97 0.9117 0.3245(train) + 16.6011(test) Details
19 Charles Maximum Likelihood In-house Code, Severely Hacked (see notes) 80.26 0.7844 2.150113 sec Details
20 Cloud Fuzzy KNN k=1, L2 95   1.4s (train+cross validation+test) Details
21 Melissa PNN sigma = 0.62 68.24 0.3762 1.73 secondsDetails
22 Neel Genetic Simplified Fuzzy ARTMAP population size = 80, generations = 2000 90.34 0.898 19.5 minutes (training) 1.2 seconds (test) Details
23 Roger Fuzzy ARTMAP vigilance =.001 89.12   22.23 sec (train,test, visualize)  
23 Roger PNN spread =.005 95.22   23.24 sec (train,test, visualize)  
24 Joe Backprop eta =.5, 2000 epochs 96   90.67 sec (train,test, visualize)  
25 Joe Maximum Likelihood   90   sec (train,test, visualize)  
26 Karthik SVM svmlite,polynomial kernels 94 .91 2.5 seconds (train) 0.8 seconds (test) works fine for kernels of order 3 or more Results
27 Ben kNN k=1 95.55   ~0.1 seconds Details  
28 Todd EBF Centers = 1 96.34 0.9782 0.036s (trn) 0.666s (test) Details
29 Sean Fuzzy ARTMAP beta=1, rho=0.9 87.97 0.91 0.129 sec (train); 11.334 sec (test) Details
30 Sean Radial Basis Function RBF Nodes=25; sigma=0.7 92.66   23.99 sec Details

6 -Jeff (SVM)

# CIS 6 - Jeff # actual positive (+) # actual negative (–) ____total____
1 # predicted + 4828 125 4953
2 # predicted – 169 4878 5047
3 total 4997 5003 10000

9 -Jeff (MADALINE)

# CIS 9 - Jeff # actual positive (+) # actual negative (–) ____total____
1 # predicted + 2322 1875 5092
2 # predicted – 2675 3128 4908
3 total 4997 5003 10000

7 - Peifeng (Fuzzy ARTMAP)

# CIS 7 - Peifeng # actual IN # actual OUT total
1 # predicted IN 4434 182 4616
2 # predicted OUT 563 4821 5384
3 total 4997 5003 10000

8 - Peifeng (SVM Using Platt's SMO)

# CIS 8 - Peifeng # actual IN # actual OUT total
1 # predicted IN 4944 59 5003
2 # predicted OUT 212 4785 4997
3 total 5156 4844 10000

10 - Neel (Simplified Fuzzy ARTMAP (Kasuba, 1993))

# confusion matrix # actual positive (+) # actual negative (–) ____total____
1 # predicted + 4310 551 4861
2 # predicted – 685 4366 5051
3 total 4997 5003 10000

11 - Rohit (KNN, K=2, city block metric)

# confusion matrix # actual positive (+) # actual negative (-) ____total____
1 # predicted + 4834 169 5003
2 # predicted - 276 4721 4997
3 total 5110 4890 10000

12 - Sean (KNN, K=1, city block metric)

# confusion matrix # actual IN # actual OUT total
1 # predicted IN 4721 276 4997
2 # predicted OUT 169 4834 5003
3 total 4890 5110 10000

13 - Charles (Cascade Correlation)

# confusion matrix # actual IN # actual OUT total
1 # predicted IN 4666 1643 6309
2 # predicted OUT 331 3360 3691
3 total 4997 5003 10000

14 - Melissa (Backprop with momentum)

# CIS confusion matrix # actual IN # actual OUT total
1 # predicted IN TP: 4774 FP: 32 4806
2 # predicted OUT FN: 223 TN: 4971 5194
3 total 4997 5003 10000

15 - Todd (SVM)

# confusion matrix # actual positive (+) # actual negative (–) __total__
1 # predicted + 4792 57 4849
2 # predicted – 205 4946 5151
3 total 4997 5003 10000

17 - Cloud (Backprop with momentum)

# CIS confusion matrix # actual IN # actual OUT total
1 # predicted IN TP: 4847 FP: 750 5597
2 # predicted OUT FN: 150 TN: 4253 4403
3 total 4997 5003 10000

18 - Rohit (fuzzy ARTMAP)

# CIS confusion matrix # actual IN # actual OUT total
1 # predicted IN 4638 365 5003
2 # predicted OUT 838 4159 4997
3 total 5476 4524 10000

19 - Charles (Maximum Likelihood)

# confusion matrix # actual IN # actual OUT total
1 # predicted IN 4430 1645 6075
2 # predicted OUT 567 3358 3925
3 total 4997 5003 10000

20 - Cloud (Fuzzy KNN)

# confusion matrix # actual IN # actual OUT total
1 # predicted IN 4747 245 4992
2 # predicted OUT 250 4758 5008
3 total 4997 5003 10000

21 - Melissa (PNN)

# CIS confusion matrix # actual IN # actual OUT total
1 # predicted IN TP: 1887 FP: 66 1953
2 # predicted OUT FN: 3110 TN: 4937 8047
3 total 4997 5003 10000

22 - Neel (GSFAM)

# CIS confusion matrix # actual IN # actual OUT total
1 # predicted IN TP: 4449 FP: 418 4867
2 # predicted OUT FN: 548 TN: 4585 5133
3 total 4997 5003 10000

23 - Roger (Fuzzy Artmap)

# CIS confusion matrix # actual IN # actual OUT total
1 # predicted IN TP: 4077 FP: 168 4245
2 # predicted OUT FN: 920 TN: 4835 5755
3 total 4997 5003 10000

24 - Roger (PNN)

# CIS confusion matrix # actual IN # actual OUT total
1 # predicted IN TP: 4697 FP: 178 4875
2 # predicted OUT FN: 300 TN: 4825 5125
3 total 4997 5003 10000

24 - Joe (Backprop)

# CIS confusion matrix # actual IN # actual OUT total
1 # predicted IN TP: 4726 FP: 159 4885
2 # predicted OUT FN: 271 TN: 4844 5115
3 total 4997 5003 10000

25 - Joe (Maximum Likelihood)

# CIS confusion matrix # actual IN # actual OUT total
1 # predicted IN TP: 5003 FP: 1009 6012
2 # predicted OUT FN: 0 TN: 3988 3988
3 total 5003 4997 10000

26 -Karthik (MADALINE)

# CIS confusion matrix # actual positive (+) # actual negative (–) ____total____
1 # predicted + 2282 1915 5197
2 # predicted – 2695 3108 4803
3 total 4977 5023 10000

27 -Karthik (SVMlite)

# CIS confusion matrix # actual positive (+) # actual negative (–) ____total____
1 # predicted + 4731 312 5043
2 # predicted – 266 4691 4957
3 total 4997 5003 10000

28 - Ben (kNN, k=1)

# CIS confusion matrix # actual positive (+) # actual negative (–) ____total____
1 # predicted + 4721 169 4890
2 # predicted – 276 4834 5110
3 total 4997 5003 10000

28 - Todd (Elliptical Basis Functions)

# confusion matrix # actual positive (+) # actual negative (–) __total__
1 # predicted + 4867 236 5103
2 # predicted – 130 4767 4897
3 total 4997 5003 10000

29 - Sean (Fuzzy ARTMAP)

# CIS confusion matrix # actual IN # actual OUT total
1 # predicted IN 4638 365 5003
2 # predicted OUT 838 4159 4997
3 total 5476 4524 10000

30 - Sean (Radial Basis Function)

# CIS confusion matrix # actual IN # actual OUT total
1 # predicted IN 4809 188 4997
2 # predicted OUT 546 4457 5003
3 total 5355 4645 10000

CN550 Spring 2007

# SVM Name % Correct C-index More details
1 Gaussian kSVM Jing Bai 96.08   Methods and Results
2 SVM (Platt) Andy 97.44   details
3 SVM (Osuna) Eugene    Work in progress
4 SVM Paramesh 93.84   Results

# ARTMAP Name % Correct C-index More details
1 Fuzzy ARTMAP Tim 92.8 0.86 Vote Vis Vigilance
2 Fuzzy ARTMAP Gary 94.19 0.89 result parameters
3 Fuzzy ARTMAP Yohan 93.79 0.88 details
4 Distr ARTMAP Rob 94.5 N/A  
5 GA+Distr ARTMAP Jesse 95.3   Nodes:89, Dstrdness:1.36% (1.21 nodes), a:0.007 | b:1.01 | ratio:1.21 | v: 0.0082 | e:0 | g:0.65
6 Default ARTMAP Hee 92.83 0.86 Details_DefA_CIS
7 Default ARTMAP Paramesh 92.04 0.8459 Results
8 ARTMAP-IC Jeff 93.97   

# KNN Name % Correct C-index More details
1 KNN Andy 95.22 0.90 details
2 KNN Hee 95.22 0.90 Details_KNN

# Bayesian Est. Name % Correct C-index More details
1 Bayesian Est. Jing Xia 89.91 0.80 result
2 Bayesian Est. Tim 89.9 0.80 result

# RBF Name % Correct C-index More details
1 RBF Network Jing Bai 95.5 0.91 Methods and Results
2 RBF Jing Xia 97.43 0.95 method result

view · edit · attach · print · history
Page last modified on January 21, 2009, at 04:09 AM