Daniel Marbach - Swiss Federal Institute of Technology LausanneFeb 16, 2008 - 10:35 am
(1) I agree with Pablo Verdes that the primary goal of reverse engineering is identifying the system, and not predicting data. From a black-box model that only predicts data we cannot learn much about the functioning of a biological system. Furthermore, not all reverse engineering methods construct a predictive model for the data. (2) It has been argued that since a large class of networks may be consistent with the data, reverse engineering methods should not be compared on a single network prediction. I agree, but I believe that the current format (a ranked list of link predictions) is a very good one, precisely for this reason. This list is *not* a network, and in a sense it is even wrong to interpret it as a network. Instead, the list gives for every link the estimated confidence level (e.g., the posterior probability given the data and the prior knowledge) that it is present in the gold standard network. For example, we obtained this list by combining the information contained within an ensemble of different networks that fit the data well. (3) Methods should be compared on several gold standard networks to compare their performance (at least for the in silico benchmarks, where many gold standards can easily be generated).
What should we predict?
Daniel Marbach - Swiss Federal Institute of Technology LausanneFeb 16, 2008 - 10:35 am
(1) I agree with Pablo Verdes that the primary goal of reverse engineering is identifying the system, and not predicting data. From a black-box model that only predicts data we cannot learn much about the functioning of a biological system. Furthermore, not all reverse engineering methods construct a predictive model for the data. (2) It has been argued that since a large class of networks may be consistent with the data, reverse engineering methods should not be compared on a single network prediction. I agree, but I believe that the current format (a ranked list of link predictions) is a very good one, precisely for this reason. This list is *not* a network, and in a sense it is even wrong to interpret it as a network. Instead, the list gives for every link the estimated confidence level (e.g., the posterior probability given the data and the prior knowledge) that it is present in the gold standard network. For example, we obtained this list by combining the information contained within an ensemble of different networks that fit the data well. (3) Methods should be compared on several gold standard networks to compare their performance (at least for the in silico benchmarks, where many gold standards can easily be generated).