The DREAM5 Network Inference Challenge

DREAM5, Challenge 4

1. Synopsis

The goal of this Network Inference Challenge is to reverse engineer gene regulatory networks from gene expression datasets. Participants are given four microarray compendia and are challenged to infer the structure of the underlying transcriptional regulatory networks.

Three of the four compendia were obtained from microorganisms, some of which are pathogens of clinical relevance. The fourth compendium is based on an in-silico (i.e., simulated) network. Each compendium consists of hundreds of microarray experiments, which include a wide range of genetic, drug, and environmental perturbations (or in the in-silico network case, simulations thereof). Network predictions will be evaluated on a subset of known interactions for each organism, or on the known network for the in-silico case.

2. The challenge

Participants are given four microarray compendia. In addition to the gene expression data, we also include a number of descriptive features for each microarray experiment (e.g., temporal information if the experiment is part of a time series, or the deleted gene if it’s a gene deletion experiment). Participants are further given a list of candidate transcription factors (TFs) for each compendium. In the case of the data arising from actual gene expression, some information has been anonymized such that the identity of the network, of the TFs and of the genes remains unknown to the participants.

We ask for predictions of the complete (genome-scale) transcriptional regulatory network for each compendium. In order to participate in the challenge, predictions for all four networks must be submitted (there are no sub-challenges).

The following table summarizes the number of TFs, the number of genes, and the number of microarray chips for each network:

Network # Transc Factors # Genes # Chips
Network 1 (in-silico) 195 1643 805
Network 2 99 2810 160
Network 3 334 4511 805
Network 4 333 5950 536

Network 1 is the in-silico network, Networks 2-4 correspond to the real microarray compendia obtained from microorganisms.

3. The Datasets

In this section, we describe the datasets that are given to the participants for each of the four networks. A substantial part of this data is unpublished and may not yet be used for publication without permission of the owners (see Section 8 for details).

All datasets are located in the file (click here to go to the download site):

For each network (Network1,..., Network4) there are three types of files, which contain the expression data, meta information for each microarray experiment, and a list of transcription factors, respectively:

where i ∈ {1,2,3,4}. The files are given in tab-separated value format (tsv). We describe the content of each of these files below.

3.1 Gene expression data

The files Networki_expression_data.tsv contain the matrix of gene expression values for Network i. Each row corresponds to a microarray chip, and each column to a gene. In other words, element (i, j) is the expression value of gene j in chip i of the compendium. The first line of the file gives the (anonymized) label of the gene for every column.

The expression data has been uniformly normalized for each compendium so that values are comparable across experiments.

For convenience, the exact same expression data will be posted on our website (gnw.sourceforge.net) in two alternative formats: (1) With genes in rows and experiments in columns, ready to be loaded with a biological data visualization tool; (2) With multiple replicates of the same experiment averaged (see below).

3.2 Chip features

The files Networki_chip_features.tsv contain meta information for each microarray chip for Network i. The information is presented as a matrix, where rows correspond to chips and columns to descriptive features. Row k gives the features for row k of the file Networki_expression_data.tsv (the k’th microarray chip of the compendium). There are eight columns, each corresponding to one of the following eight features:

We will now explain these features using an illustrative example. The table below contains the descriptions for thirteen microarrays, which were obtained from five different experiments. Microarrays that are from the same experiment were done in the exact same experimental setting (same experimenter, strain, growth medium, growth phase, etc).

#Experiment Perturbations PerturbationLevels Treatment DeletedGenes OverexpressedGenes Time Repeat
1 1 NA NA NA NA NA NA 1
2 1 NA NA NA NA NA NA 2
3 2 NA NA NA NA NA NA 1
4 2 P1 0.5 NA NA NA NA 1
5 2 P1 1.0 NA NA NA NA 1
6 3 NA NA NA NA NA 0 1
7 3 NA NA NA NA NA 30 1
8 3 NA NA NA NA NA 60 1
9 3 NA NA NA G5 NA 30 1
10 3 NA NA NA G5 NA 60 1
11 4 NA NA NA G5,G8 NA NA 1
12 5 P2,P3 NA NA NA G4 NA 1
13 5 P2,P3 NA 1 NA G4 NA 1

(The grey, leftmost column, counts the row numbers after the header and is not part of the actual file.)

Chips 1 and 2 (first and second row) are replicates of experiment 1, as indicated by the repeat numbers in the righmost column. There have been no drug or genetic perturbations in this experiment.

Chips 3, 4, and 5 were all obtained using the same experimental setting (experiment 2). Chips 4 and 5 measure the response of the organism to perturbation P1 at levels 0.5 and 1.0, respectively. Chip 3 is the control without the perturbation. Perturbations in the compendia are often drugs, but may also represent environmental variables such as glucose concentration or acidity. The dosage/level is not specified for all perturbations. Note that the same perturbation is often applied in several experiments of the compendium.

Chips 6–10 are part of a time-series (experiment 3). Chip 6 is the initial time point at t=0. Chips 9 and 10 show the expression profile at time t=30 and t=60 after knockout of gene G5. Chips 7 and 8 are the corresponding controls without knockout of G5.

If two genes are perturbed simultaneously, as in the double knockout of genes G5 and G8 Chip 11 (experiment 4), they are listed separated by a comma. The same format is used if two perturbations are applied simultaneously (experiment 5).

Drug and genetic perturbations are sometimes combined. In experiment 5, two drugs are applied (P2 and P3) and one gene (G4) is overexpressed all in the same experiment.

The treatment column can be ignored for most purposes, it’s used in only a handful of cases. (Rarely, the same experiment has been performed twice in a different manner. The treatment column is used to distinguish these experiments. For example, the same drugs (P2 and P3) may have been applied simultaneously in experiment 12 and sequentially in experiment 13.)

Note, experiments that are not part of a time series (time is ‘NA’) are often done at a time when the perturbation is assumed to have unfolded its full effect (for the in silico network we give the steady state of the perturbation).

3.3 Transcription Factors

The files Networki_transcription_factors.tsv list the genes of the network that are potential TFs for network i. Only genes that are part of this list should be included as regulators in the submitted network predictions (see below). Naturally, a TF may regulate another TF gene (TFs can also be targets in the network).

Note that our criteria for including candidate TFs were permissive. Thus, the lists probably contain incorrectly annotated/predicted TFs that are in truth not regulatory proteins. The scoring, however, will be done on the known transcription factors.

4. Submission information

Note: To participate in the challenge you need to submit predictions for all four networks and a writeup explaining your methods.

Network predictions

Submitted networks must be directed and unsigned. Self-interactions (auto-regulatory loops) are ignored in the evaluation (they will be removed from the gold standard networks).

For each network, submit a ranked list of no more than 100,000 regulatory link predictions ordered according to the confidence you assign to the predictions, from the most reliable (first row) to the least reliable (last row) prediction. You can submit less than 100,000 edges. Links (pairs of nodes) that are not part of the submitted list are considered to appear randomly ordered at the end of the list.

In your submission, use a tab-separated column format as in the example below:

A \tab B \tab X

where A and B are two different genes (no self-interactions). Use the labels given in the header of the file Networki_expression_data.tsv to identify genes (G1, G2, etc). Links are directed: the gene in the first column regulates the gene in the second column. (If both A regulates B and B regulates A, then both lines should be included.) Gene A must be a TF listed in the file Networki_transcription_factors.tsv of that network (gene B may be any gene of the network, TF or non-TF). X is a score between 0 and 1 that indicates the confidence level you assign to the prediction. (E.g., X = 1 if gene A is deemed to regulate gene B with highest confidence and X = 0 if A is deemed not to directly regulate B). Save the file as text, and name it:

DREAM5_NetworkInference_TeamName_Networki.txt

where TeamName is the name of the team with which you registered for the challenge, and i is 1, 2, 3, or 4.

Write-up

Finally we request that each participating team submits a short write-up (around one to three pages) explaining the methods used to arrive at their predictions of the phenotypes. This write-up, which is mandatory for submission, can contain pseudo-code, workflows, and explanations of the concepts. Submit the write-up as the file

DREAM5_NetworkInference_TeamName_Writeup.ext

replacing "TeamName" with the name of your team and the file extension (ext) with your choice of txt, doc, rtf, or pdf.

5. Scoring metrics

We will score the results using the area under the precision versus recall curve for the whole set of link predictions for a network. For the first k predictions (ranked by score, and for predictions with the same score, taken in the order they were submitted in the prediction files), precision is defined as the fraction of correct predictions to k, and recall is the proportion of correct predictions out of all the possible true connections. Other metrics such as the area under the ROC curve will also be evaluated. Teams will be ranked according to their overall performance over the four networks of a challenge.

For a detailed discussion of the submission format and scoring metrics, refer to Prill et al., PLoS ONE, 5:e9202, (2010) and Marbach et al., PNAS, 107:6286-91, (2010).

6. Some notes on the in silico dataset

The in-silico network and dataset was generated using GeneNetWeaver (GNW) version 3.0, which will be released after the DREAM5 conference. The framework is very similar to the previous edition of the challenge (the DREAM4 in-silico network challenge), except that for DREAM5 we simulate the exact same number and types of experiments as available in one of the microarray compendia extracted from actual measurements.

The in-silico data is presented in the same format as the microarray data, i.e., in the form of an expression matrix (Networki_expression_data.tsv) and a list of features for every experiment (Networki_experiment_features.tsv). Below, we briefly describe how the in-silico networks and datasets were generated.

Network structures. The topology of the in-silico network is based on known transcriptional regulatory networks of model organisms, as described in Marbach et al., J Comput Biol, 16:229-39, (2009).

Dynamical model. The dynamics on the networks were simulated using a detailed kinetic model of gene regulation. Both independent and synergistic gene regulation occur in the networks. Both transcription and translation are modeled (the provided gene expression datasets correspond to the mRNA concentration levels, protein concentrations are not given). For more information see Marbach et al., PNAS, 107:6286-91, (2010).

Noise. The simulations are based on stochastic differential equations (Langevin equations) to model internal noise in the dynamics of the networks. In addition, we add measurement noise to the generated gene expression datasets. We use an existing model of noise observed in microarrays, which is very similar to a mix of normal and lognormal noise.

Simulated Experiments. Next we explain how the different types of experiments of the compendia were simulated.

• Experiments. Different experiments were modeled by applying multifactorial perturbations to the network. Multifactorial perturbations were done by slightly increasing or decreasing the basal activation of all genes of the network simultaneously by different random amounts. A multifactorial perturbation may thus be interpreted as a variation of the network steady state. The exact same multifactorial perturbation was used for all chips of a given experiment.
• Drug perturbations. Drug perturbations (second column of the chip features) were also modeled by increasing or decreasing the basal activation of several genes simultaneously. In contrast to the multifactorial perturbations described above, which affect all genes, drug perturbations were assumed to affect only a small number of direct targets. The increase or decrease in activity of these genes depends on the level of the perturbation (third column of the experiment features). The same perturbation was used for all experiments where a given drug was applied. Drug perturbations were applied in addition to the multifactorial perturbation associated with the corresponding experiment.
• Genetic perturbations. Gene deletions and overexpression were simulated by reducing/increasing the transcription rate of the corresponding genes. Similar to drug perturbations, genetic perturbations were applied in addition to the multifactorial perturbation of the corresponding experiment.
• Time series. Time series experiments were simulated by applying the multifactorial perturbation of corresponding experiment at t=0. If in addition a drug or genetic perturbation is specified, this perturbation was also applied at t=0.
• Repeats. Repeats were obtained by running the stochastic simulation of the same experiment multiple times. The perturbations of repeats are identical, but differences in expression occur due to intrinsic noise and measurement noise.

For all experiments that are not part of a time series (time=NA), we provide the (noisy) steady-state of the network for the corresponding experiment/perturbation.

Note that the in-silico data is on a linear scale, whereas the actual microarray data is on a logarithmic scale (log2). The in silico model produces data that lends itself well to a linear scale (it does not vary over several orders of magnitude), so we did not log-transform it.

7. Authors

The identity of the data producers will be disclosed after the submission deadline.


8. Using the data in publications

Data from this challenge can be freely used. Please cite the following paper:

Wisdom of crowds for robust gene network inference. Marbach D*, Costello JC*, Küffner R*, Vega NM, Prill RJ, Camacho DM, Allison KR, The DREAM5 Consortium, Kellis M, Collins JJ, Stolovitzky G. Nature Methods, 9(8):796-804, 2012.
[PubMed] [pdf] [Supplement] [Companion site] [From the cover] [MIT news feature]

All datasets, gold standards, evaluation scripts, gene names, network predictions (individual teams and community), etc., are included in the supplementary data of the paper (please contact Daniel Marbach if you don't have access):
http://www.nature.com/nmeth/journal/v9/n8/full/nmeth.2016.html#/supplementary-information

9. Data Download

Download Data (Registration Required).

All datasets, gold standards, etc., are now also available for download as supplementary data of the "Wisdom of crowds" paper (see previous section)

10. Questions and Feedback

Don't hesitate to post a question in the DREAM Discussion board or directly contact Daniel Marbach (dmarbach@mit.edu) or Gustavo Stolovitzky(gustavo@us.ibm.com).

Retrieved from "http://wiki.c2b2.columbia.edu/dream/index.php/D5c4"

This page has been accessed 22,007 times. This page was last modified 10:23, 26 September 2013.

x
Find
Browse
The DREAM Project
Community portal
Current events
Recent changes
Random page
Help
Donations
Edit
Edit this page
Editing help
This page
Discuss this page
Post a comment
Printable version
Context
Page history
What links here
Related changes
My pages
Create an account or log in
Special pages
New pages
File list
Statistics
Bug reports
More...