DREAM4: Prediction Evaluations by Challenge

Challenge 3 : SignalingNetworkOriginal

Refer to the Challenge 3 Description for more information.

Updated results and Original results

The results table called SignalingNetworkOriginal was based on a cost per link that was determined empirically from the team with the lowest value of prediction_score / number of links. It later came to light that the team upon which the cost per link value was based made a submission error in their phosphoprotein predictions. Therefore, we re-scored the challenge using this team's (unofficial) corrected predictions, which affected the empirically determined cost per link, which in turn affected the overall score for every team. The updated results table is called SignalingNetworkUpdated. The three best-performers remain the same using either derivation of cost per link, but there are subtle changes in the rankings of the remainng teams.

Definitions

• Normalized Squared Error for protein P (P = AKT, ERK12, IKB, JNK12, P38, HSP27, MEK12): The sum of the normalized squared prediction errors for protein P. The squared error for each data point was normalized by an estimate of the magnitude of the measurement error as:

                            


• P-value for protein P: Given the NSE(P) for a submitted prediction, the P-value for protein P is the probability of obtaining an equal or larger NSE(P) under the null distribution. The null distribution was produced empirically by assuming that the random predictions of protein P activity were values sampled at random from the actual set of values for protein P. Note that given that the dynamic range differs for different proteins, the distributions of NSE(P) for two proteins could be quite different.
• Prediction Score: A log-transformed "average" of the P-values for the 7 measured proteins, computed as -log10 ( P-value AKT * ... * P-value MEK12 )/7. Larger scores indicate greater statistical significance of the prediction.
• Edge Count: The number of links in the submitted network, on which the predictions were based.
• Overall Score: The excess score obtained by a team after discounting a minimum gain in score expected by the simple addition of a link in the submitted network.
  
        Overall Score = Prediction Score - r * (Edge Count)
  
  where r, the score per edge, is defined as:
  
        r = min {Prediction Score / Edge Count}
  
  where the minimum is taken over all the submissions. For this challenge the resulting value of r was 0.0581

Predictive Signaling Network Modeling Results

TEAM	OVERALL SCORE	EDGE COUNT	PREDICTION SCORE	AKT PVAL	ERK12 PVAL	IKB PVAL	JNK12 PVAL	P38 PVAL	HSP27 PVAL	MEK12 PVAL
Team 395	3.075	15.000	3.946	7.927e-04	9.758e-08	1.032e-04	8.359e-05	9.302e-03	1.461e-02	2.625e-06
Team 533	3.610	26.000	5.121	1.327e-03	2.801e-09	3.062e-07	1.746e-06	1.166e-04	2.355e-03	2.625e-06
Team 471	0.477	54.000	3.614	4.426e-03	6.056e-07	1.481e-04	1.189e-06	1.122e-03	1.247e-02	7.590e-03
Team 347	0.740	19.000	1.844	3.811e-03	6.802e-01	5.080e-05	1.000e+00	1.620e-03	9.051e-03	6.406e-02
Team 441	3.895	18.000	4.941	1.653e-03	1.337e-07	3.062e-07	1.451e-06	5.206e-05	6.246e-04	8.148e-06
Team 491	-0.000	18.000	1.046	2.380e-02	2.424e-05	7.147e-01	9.608e-01	8.457e-01	5.312e-01	2.695e-01
Team 544	0.364	18.000	1.410	4.172e-03	4.489e-05	8.712e-02	1.000e+00	9.949e-01	1.134e-01	7.307e-02
Team 476	3.830	17.000	4.818	4.300e-03	7.457e-09	5.715e-07	1.451e-06	2.462e-05	2.869e-03	1.006e-04
Team 451	3.355	17.000	4.343	1.490e-03	4.123e-08	1.389e-06	2.151e-04	1.967e-04	1.181e-03	9.389e-05
Team 504	1.858	10.000	2.439	4.624e-03	1.626e-05	7.385e-03	8.713e-04	2.573e-03	3.385e-02	2.000e-01
Team 256	3.287	22.000	4.565	1.811e-03	4.090e-07	3.909e-07	1.451e-06	4.901e-04	6.040e-03	8.858e-06
Team 314	2.582	27.000	4.151	2.432e-03	1.382e-05	1.908e-06	2.075e-06	4.901e-04	6.788e-03	1.990e-05