At the end of DREAM2 there were a number of good ideas that people suggested. Here are the few of them that I could capture. If you made the comment, please feel free to ellaborate on it, as in some cases I could only capture the general idea, but none of the details. 1) Make the data (used in the challenges) comparable accross years. 2) Invite the pharmaceutical industry to participate more actively in the reverse engineering challenges. 3) Have some ongoing benchmark in the website. 4) Create a protocol for the comparison of reverse engineering methods. 5) It is unclear whether the networks underlying the data can be inferred from the available data. This is the issue of model identifiability. There is also the isuue of model distinguishability: there can be more than one model consistent with the data. Which one is the "right" one? 6) Related to the previous item, there is the question of what is it that we should infer? The "network" that we are trying to reverse engineer should not be what we are predicting. May be the emphasis should be on predicting experimental measurements. In other words, instead of predicting networks, we should be predicting data. 7) There was a suggestion to add the methods that were used in the network prediction challenges on the web. 8) Some people think that it is necessary to clearly say who the participating teams are, not just the best performers. I would like some feedback on this, as there are pros and cons in both cases. 9) Some people felt that we were ignoring the last 30 years of molecular biology in the design of the challenges. The question is how to make predictions in addition of what is known. 10) At the end of DREAM2 there was a 100% agreement that a DREAM3 network inference challenge would be useful. Do you agree? These are the ideas and suggestions that were made at the conclusion of DREAM2. Please feel free to comment on any of these or on anything else.
Food for Thought from DREAM2
Gustavo Stolovitzky - IBM Research/DREAM organizerDec 28, 2007 - 4:59 pm
At the end of DREAM2 there were a number of good ideas that people suggested. Here are the few of them that I could capture. If you made the comment, please feel free to ellaborate on it, as in some cases I could only capture the general idea, but none of the details. 1) Make the data (used in the challenges) comparable accross years. 2) Invite the pharmaceutical industry to participate more actively in the reverse engineering challenges. 3) Have some ongoing benchmark in the website. 4) Create a protocol for the comparison of reverse engineering methods. 5) It is unclear whether the networks underlying the data can be inferred from the available data. This is the issue of model identifiability. There is also the isuue of model distinguishability: there can be more than one model consistent with the data. Which one is the "right" one? 6) Related to the previous item, there is the question of what is it that we should infer? The "network" that we are trying to reverse engineer should not be what we are predicting. May be the emphasis should be on predicting experimental measurements. In other words, instead of predicting networks, we should be predicting data. 7) There was a suggestion to add the methods that were used in the network prediction challenges on the web. 8) Some people think that it is necessary to clearly say who the participating teams are, not just the best performers. I would like some feedback on this, as there are pros and cons in both cases. 9) Some people felt that we were ignoring the last 30 years of molecular biology in the design of the challenges. The question is how to make predictions in addition of what is known. 10) At the end of DREAM2 there was a 100% agreement that a DREAM3 network inference challenge would be useful. Do you agree? These are the ideas and suggestions that were made at the conclusion of DREAM2. Please feel free to comment on any of these or on anything else.