01-overview.Rmd

# Introduction

This handbook describes the process of running a Cell Painting experiment.
While the code here will describe doing so in the context of running [Distributed-CellProfiler](https://github.com/CellProfiler/Distributed-CellProfiler) on AWS on images generated by a PerkinElmer microscope, 
then collating the data with [cytominer-database](https://github.com/cytomining/cytominer-database) and analyzing it with [pycytominer](https://github.com/cytomining/pycytominer), the basic procedure for running a Cell Painting experiment is the same no matter the microscope or the processing platform. 
Briefly, the steps for any and every platform are:

## Collect your software 

For the specific use case here, this involves 

- [Distributed-CellProfiler](https://github.com/CellProfiler/Distributed-CellProfiler)
- [pe2loaddata](https://github.com/broadinstitute/pe2loaddata)
- [cytominer-database](https://github.com/cytomining/cytominer-database)
- [pycytominer](https://github.com/cytomining/pycytominer)

along with their dependencies. 
Almost certainly, you will need a locally-to-your-images installed version of GUI CellProfiler that matches the version you want to run on your cluster - locally might mean on a local machine or a VM.

## Collect your pipelines 

You will minimally require these pipelines

- `illum` (illumination correction)
- `analysis` (segmentation and feature extraction)

But you may also want pipelines for

- Z projection
- QC
- assay development

## Determine how to get your image lists to CellProfiler 

CellProfiler needs to understand image sets, aka for each field-of-view that was captured how many channels were there, what would you like to name the channels were there, and what are the file names corresponding to each channel.
If you are using a Phenix Opera or Operetta, you can use the `pe2loaddata` program to generate a CSV that contains a list of image sets in an automated fashion, which can be passed to CellProfiler in via the LoadData module.
Otherwise, you have a couple of different options:

1. You can create a similar CSV using a script that you write yourself that handles the files from your microscope and makes a similar CSV. 
Minimally, you need a `FileName` and `PathName` column for each channel (ie `FileName_OrigDNA`), and Metadata columns for each piece of metadata CellProfiler needs (ie `Metadata_Plate`, `Metadata_Well`, and `Metadata_Site`)
1. You can use a local copy of the files CellProfiler will be running, configure the 4 input modules of CellProfiler to create your image sets, then export CSVs using CellProfiler's "Export Image Set Listing" option, and feed those into the pipelines to be run on your cluster.
1. Alter all of your pipelines to, rather than the LoadData CSV module, use the 4 input modules of CellProfiler and add, configure, and run each pipeline with the CreateBatchFiles module and use batch files in your cluster environment.

These options are ordered in terms of most-scripting-proficiency-needed to least-scripting-proficiency-needed as well as least-CellProfiler-proficiency-needed to most-CellProfiler-proficiency-needed.

## Execute your CellProfiler pipelines

### (Optional) Z projection

If your images were taken with multiple planes, you will need to Z-project them. 
All subsequent steps should be run on the projected images.

### (Optional) QC

You may want to run a quality control pipeline to determine your imaging plate quality.  
You can choose to run this locally or on your cluster compute environment.
You will need to evaluate the results of this pipeline somehow, in CellProfiler-Analyst, KNIME, SpotFire, etc. 
You may run illumination correction and assay development steps in the meantime, but should hold analysis steps until the results are evaluated.

### Illumination correction

You need to run a pipeline that is grouped by plate and creates an illumination correction function.  
Since it is grouped by plate, you don't need very many CPUs to run this, but it will take 6-24 hours depending on settings and image size.  
Assay development and analysis require this step to complete.

### (Optional) Assay Development

If desired, you can run a pipeline that executes on one image per well and carries out your segmentation but not measurement steps and makes (an) image(s) that you can use to evaluate the quality of the segmentation (either individually or by stitching them together first). 
This is not required but allows you to ensure that your segmentation parameters look reasonable across the variety of phenotypes present in your data.  
If being run, the final step should be held until this step can be evaluated. 

## Analysis

This pipeline segments the cells and measures the whole images and cells, and creates output CSVs (or can dump to a MySQL host if configured).  
This is typically run on each image site in parallel and thus can be sped up by using a large number of CPUs.

## Aggregate your data

Since the analysis is run in parallel, unless using a MySQL host you will have a number of sets of CSVs that need to be turned into a single file per plate.  
This is currently done with the `cytominer-database` program.

## Create and manipulate per-well profiles. 

The final step is to create per-well profiles, annotate them with metadata, and do steps such as plate normalization and feature selection.  
These are accomplished via a "profiling recipe" using `pycytominer`.