Predictive Signaling Network Modeling
DREAM4, Challenge 3
Note: Both the data and pathway map cannot be used for purposes other than this challenge without the explicit permission of the data providers. (See below for contact information.)
This challenge explores the extent to which our current knowledge of signaling pathways, collected from a variety of cell types, agrees with cell-type specific high-throughput experimental data. Specifically, we ask the challenge participants to create a cell-type specific model of signal transduction using the measured activity levels of signaling proteins in HepG2 cell lines. The model, which can leverage prior information encoded in a generic signaling pathway provided in the challenge, should be biologically interpretable as a network, and capable of predicting the outcome of new experiments.
It is an open question how to make use of the accumulated body of knowledge of signaling pathways to create mechanistic, predictive signaling network models. The network depicted in Figure 1 is representative of the type of information about the topology of signaling pathways that can be culled from the literature . Figure 1 depicts canonical pathways downstream of major receptors to four ligands (represented by green nodes): two inflammatory (TNFa, IL1a), one insulin (IGF-I), and one growth factor (TGFa). Note that this pathway map is not cell-type specific.
In addition to the topology of "canonical" signaling pathways based on the accumulation of evidence from multiple cell-types, we have at our disposal a data set consisting of measurements of phosphoprotein activity levels in the HepG2 cell line using the Luminex xMAP sandwich assay. Measurements of certain phosphoprotein activities were measured under various perturbations of the signaling pathways . The signaling pathways were stimulated with one or more of the ligands mentioned above. The pathways were also perturbed with chemical inhibitors of specific phosphoproteins. In Figure 1, blue and magenta nodes indicate the phosphoproteins measured by the xMAP assay; red and magenta nodes indicate the phosphoproteins that were inhibited.
Figure 1. Pathway map summarizing the current public knowledge of the signaling pathway pertaining to this challenge, simplified from . Green nodes represent stimuli. Red nodes represent inhibited proteins. Blue nodes indicate proteins whose phosphorylation is measured. Magenta nodes represent proteins that are both inhibited and measured. Grey nodes represent proteins considered to be involved in the relevant pathways. This figure was created with Cytoscape http://www.cytoscape.org/) from data obtained from the Ingenuity knowledgebase.
The Hep2G data set is plotted in Figure 2, which is organized into panels corresponding to the various ligands. Upon pretreatment with an inhibitor (or no inhibitor), measurements (phosphoprotein activities) of seven proteins at three time points (0, 30 minutes, and 3 hours post stimulus) were acquired.
The challenge entails "customizing" the provided pathway map (Figure 1) so that it is an accurate representation of the provided data set (Figure 2). Specifically, we are soliciting
- A revised network specific to the HepG2 cell line. The revised network could be produced by removal of links that are not supported by the provided data set from the pathway map of Figure 1, and/or, addition of links that are supported by data, but absent from the pathway map of Figure 1.
- The predicted values of the 7 measured phosphoproteins for all 20 possible pairwise combinations of the following stimuli and inhibitors which comprise a "test set."
Stimuli: IL1a, IGF1, TGFa, and TGFa+IGF1
Inhibitors: pp38+MEK, PI3K+MEK, p38+PI3K, p38+IKK, and PI3K+IKK
In the above list, TGFa+IGF1 indicates that both TGFa and IGF1 were simultaneously applied to the cells. The same is true for simultaneous application of inhibitors such as PI3K+IKK. The answer to the challenge should entail some interplay between predictive modeling and network reconstruction. Any modeling formalism may be used as long as the model is amenable to be interpreted as a network. A wide range of modeling formalisms can be applied and model relevance will be ascertained by how close the model predicts the response to the set of test stimuli and inhibitors.
Figure 2. Training data set. Time courses for the phosphorylation of 7 key proteins (rows) in the cancer cell line HepG2 treated with 5 different protein inhibitors (including no inhibitor) under 5 different conditions of cytokine stimulation (panels, including no cytokine stimulus) . When the measured molecule is inhibited the measurement cannot be used (denoted with a big red X). The numbers at the right of the figure indicate the maximum value for the signals across all conditions (i.e., the maximum value of the corresponding row) and it is in arbitrary units (fluorescent intensity). The figure was created using DataRail .
The canonical pathway map (Figure 1) is provided in several formats with filenames
where ext is one of the following: pdf, gml, xgmml, cys, and sif. All formats were created with Cytoscape. The sif (simple interaction file ) format contains the human readable list of edges in source/target format; Gml (Graph markup language ) and xgmml (extensible graph markup and modeling language) provide additional information about the network visualization (not relevant for the analysis), and cys is the intern Cytoscape format.
The data set (Figure 2) is provided as comma-separated-value files in two formats, DataRail’s MIDAS (Minimum Information for Data Analysis in Systems Biology) format , and a simple table, with the filenames:
Important information regarding measurements
(a) Data integrity / linearity. Significant effort was dedicated to data integrity. The data are reported as arbitrary (fluorescence) units in the range between 0 and ~29000. The upper limit (~29000) corresponds to the saturation limit of the detector. Experiments were performed in such a way that measurements are as much as possible within the linear range of the detector. In general, data can be considered linear but there are a few cases that measurements are closer to the upper detection limit of ~29000 (e.g. some AKT measurements) where linearity might have been lost.
(b) Detection limits/Repeatability. The coefficient of variation for repeated measurements was found to be ~8% (mostly due to biological variability). With our current experimental design the instrument detector can report data with accuracy as low as ~300. For example, changes from 55 fluorescence units (FU) to 110 FU cannot be considered "2 fold increase" because values lie within the noise error of the detector. On the contrary, data from 1000 to 2000 are significant.
(c) Comparability between phosphoproteins. In the xMAP sandwich assays used to collect the data for this challenge, the fluorescence measured for one phosphoprotein is not directly comparable to that of another phosphoprotein. For example, the same readout of 1000 in AKT and ERK signal does not imply that the concentration of AKT and ERK are the same. The reason is that even at the same concentration, the amount of light detected for different phosphoproteins depends on the affinity of the antibodies to the phosphoproteins.
(d) Comparability between training and test sets. The lysate concentration used for the measurements of the training data set (contained in the file SignalingNetworkChallenge_TrainingData.csv) was different from the lysate concentration used for the test data set. This was done to keep the measurement values within the linear range of the detector. Therefore, even for the same phosphoprotein and under the same conditions, the measurement in the training and test data sets could be different. This is why we give the value of the measurement at t=0 in the file DREAM4_TeamName_SignalingNetworkPredictions_Test.csv, as these values could, in principle, be different from the values at t=0 for similar conditions in the training set. Therefore, the predictions at t=30 min have to take into account the baseline value at t=0 of the test set rather than equivalent measurements in the training set. For clarifications on this important aspect of the data, please feel free to contact the DREAM organizers or the data providers.
Challenge participants will submit three files:
(1) Predictions of the 7 phosphoprotein activities under the various perturbations in the test set. These predictions should be submitted within the template file:
provided with the data. At submission, replace TeamName with the name of your team, and the entries containing the text "PREDICT" with your numerical predictions.
(2) A list of edges of the network underlying your predictions. (The model used to produce your prediction must be interpretable as a network.) Submit this list as the file
replacing TeamName with the name of your team. Your network must be submitted as a tab delimited list of node pairs, which represent edges in the network. Only edges supported by your model should be included in the submitted edge list, and the order of the list is inessential. Edges such as tgfa→ erk12 and igf1→ hsp27, for example, should be encoded as:
tgfa \tab erk12 igf1 \tab hsp27 ...
Identify the nodes using ONLY the following node labels: tgfa, igf1, tnfa, il1a, akt, jnk12, erk12, ikb, hsp27, mek12, p38, pi3k, ikk
These are the colored nodes in Figure 1. Your network submission may not have self-loops or node labels other than those provided above. (See section Network compression for submission for additional information.)
(3) A one to two page write-up explaining how the predictions are produced from the network. The write-up helps enforce the purpose of the challenge: to develop a predictive network model. This write-up can contain pseudo-code describing the algorithm used. Submit the write-up as the file
replacing TeamName with the name of your team and the file extension (ext) with your choice of txt, doc, rtf, or pdf.
Network compression for submission
Only some of the nodes in the pathway map of Figure 1 are measured or manipulated in the HepG2 cell line data. However, a model may contain a representation of additional proteins that are not measured or manipulated in the assays (latent variables). To facilitate scoring and comparison of models from different teams, we ask that the network edges be reported using only the nodes that we provide for the explicit purpose of submission of the network. For example, if your model has a pathway A-->B-->C, but B is a latent variable, then this pathway should be "compressed" and reported as A-->C for the purpose of submission of the network.
The submissions will be scored by the prediction error in the test set and the parsimony of the submitted network. More specifically the prediction cost function will be scored as a sum of squared errors over all the predictions. The teams that have a prediction error below a threshold (determined by a low p-value) will be further evaluated according to the number of edges in the submitted network. Of the most significant predictive models, the team with the sparsest network will be considered a best performer.
We realize that "cheating teams" might use a prediction model that is not associated with any network interpretation, and submit a network without edges. We strongly discourage this, as the idea of this challenge is to explore the possibility of using predictive models that are biologically interpretable, and could lead to the formulation of new hypotheses.
 Saez-Rodriguez J, Alexopoulos L, Epperlein J, Samaga R, Lauffenburger DA, Klamt, S and Sorger PK (2009) "Discrete logic modeling as a means to link protein signaling networks with functional analysis of mammalian signal transduction." Mol Syst Biol in press.
 Alexopoulos L, Saez-Rodriguez J, Cosgrove B, Lauffenburger DA, Sorger PK. Net- works reconstructed from cell response data reveal profound differences in signaling by Toll-like receptors and NF-κB in normal and transformed human hepatocytes. Submitted.
 Saez-Rodriguez J, Goldsipe A, Muhlich J, Alexopoulos LG, Millard B, Lauffenburger DA, Sorger PK. Flexible informatics for linking experimental data to mathematical models via DataRail, Bioinformatics. 2008 Mar 15;24(6):840-7. (http://code.google.com/p/sbpipeline/wiki/DataRail).
The challenge was generously provided before publication by Julio Saez-Rodriguez, Leonidas Alexopoulos*, and Peter Sorger, from the Department of Systems Biology, Harvard Medical School and Biological Engineering Department, M.I.T. The challenge has been designed in collaboration with Robert Prill and Gustavo Stolovitzky from the IBM T.J. Watson Research Center in New York.
*Present Address: Department of Mechanical Engineering, National Technical University of Athens
Don't hesitate to post a question in the DREAM discussion board if you need any clarification on this challenge.