diff --git a/docs/_config.yml b/docs/_config.yml index 877d6186..898b6f77 100644 --- a/docs/_config.yml +++ b/docs/_config.yml @@ -1,11 +1,31 @@ -theme: jekyll-theme-cayman +theme: jekyll-theme-minimal show_downloads: false -baseurl: /cellxgene +baseurl: "" + +logo: cellxgene-logo.svg nav: - - title: Getting Started - url: getting-started.html - - title: Data - url: data.html - - title: FAQ - url: faq.html +- title: Installation + url: posts/install +- title: Gallery + url: posts/gallery +- title: Demo datasets + url: posts/demo-data +- title: Preparing your data + url: posts/prepare +- title: Launching cellxgene + url: posts/launch +- title: Hosting cellxgene + url: posts/hosted +- title: Annotating data + url: posts/annotations +- title: Methods + url: posts/methods +- title: Troubleshooting + url: posts/troubleshooting +- title: Roadmap + url: posts/roadmap +- title: Contributing (ideas or code) + url: posts/contribute +- title: Contact & finding help + url: posts/contact diff --git a/docs/_layouts/default.html b/docs/_layouts/default.html index ffdf1c86..f0e6b50b 100644 --- a/docs/_layouts/default.html +++ b/docs/_layouts/default.html @@ -1,50 +1,54 @@ - - {% if site.google_analytics %} - - - {% endif %} + + {% seo %} - - - - - + + +
+
+ cellxgene - -
+
{{ content }} - -
+
+ -
+
+ + {% if site.google_analytics %} + + {% endif %} diff --git a/docs/assets/css/style.scss b/docs/assets/css/style.scss index 5742f930..073436a2 100644 --- a/docs/assets/css/style.scss +++ b/docs/assets/css/style.scss @@ -2,3 +2,11 @@ --- @import "{{ site.theme }}"; +section { width: 500px; float: right; padding-bottom: 50px; padding-top:15px} +a { color: #9E2364; text-decoration: none; } + +a:hover, a:focus { color: #9E2364; font-weight: bold; } + +a small { font-size: 11px; color: #9E2364; margin-top: -0.3em; display: block; } + +a:hover small { color: #9E2364; } diff --git a/docs/cellxgene-logo.svg b/docs/cellxgene-logo.svg index 1ff58a4e..f8b3d369 100644 --- a/docs/cellxgene-logo.svg +++ b/docs/cellxgene-logo.svg @@ -1,52 +1,23 @@ - - - - - - - + + + + + + + + + + + + + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + diff --git a/docs/data.md b/docs/data.md deleted file mode 100644 index 5fded59c..00000000 --- a/docs/data.md +++ /dev/null @@ -1,108 +0,0 @@ ---- -layout: default -title: data -description: Data ---- - -# Using `cellxgene prepare` - -#### What is `cellxgene prepare`? - -`prepare` offers an easy command line interface (CLI) to preliminarily wrangle your data into the required format for previewing it with `cellxgene`. - -#### What is `cellxgene prepare` _not_? - -`cellxgene prepare` is not meant as a way to formally process or analyze your data. It's simply a utility for quickly wrangling your data into cellxgene-compatible format and computing a "vanilla" embedding so you can try out `cellxgene` and get a general sense of a dataset. - -#### How do I install `cellxgene prepare`? - -The `cellxgene prepare` command is an optional install that you can install alongside `cellxgene launch` by running - -``` -pip install cellxgene[prepare] -``` - -#### What input formats does it accept? - -Currently, we accept `h5ad` and `loom` files, as well as `10x` directories, and are hoping to accept more formats in the future. - -While we'd like to support quick conversion from seurat and bioconductor, these packages don't currently output a python-parseable intermediate file type. In the meantime, you might check out the [converters](https://satijalab.org/seurat/v3.0/conversion_vignette.html) that are under early development. - -#### What can `cellxgene prepare` do? - -`prepare` uses scanpy to: - -- Handle simple data normalization (from a [recipe](https://www.pydoc.io/pypi/scanpy-0.2.3/autoapi/preprocessing/recipes/index.html)) -- Do basic preprocessing to run PCA and compute the neighbor graph -- Infer clusters -- Reduce dimensionality to generate embeddings. - You can control which steps to run and their methods (when applicable), via the CLI. The CLI also includes options for computing QC metrics, enforcing matrix sparcity, specifying index names, and plotting output. - -**To see a full list of available arguments and options, run `cellxgene prepare --help`.** - -#### How do I use `cellxgene prepare`? - -As a quick example, let's construct a command to use `prepare` to take a raw expression matrix and generate a processed `h5ad` ready to visualize with cellxgene. - -We'll start off using the raw data from the pbmc3k dataset. This dataset is described [here](https://icb-scanpy.readthedocs-hosted.com/en/stable/api/scanpy.datasets.pbmc3k.html), and is available as part of the scanpy API. For this example, we'll assume this raw data is stored in a file called `pbmc3k-raw.h5ad`. - -Our `prepare` compose our command looks like this: - -``` -cellxgene prepare pbmc3k-raw.h5ad \ - --run-qc \ # (A) - --recipe seurat \ # (B) - --layout tsne --layout umap \ # (C) - --output pbmc3k-prepared.h5ad # (D) -``` - -Let's look at what `prepare` is doing to our data, and how each step relates to the command above. You can see a walkthrough of what's going on under the hood for this example in [this notebook](https://github.com/chanzuckerberg/cellxgene-vignettes/blob/master/dataset-processing/pbmc3k-prepare-example.ipynb). - -**1 - Compute quality control metrics and store this in our `AnnData` object for later inspection (A)** -**2 - Normalize the expression matrix using a basic preprocessing recipe (B)** -**3 - Do some preprocessing to run PCA and compute the neighbor graph (auto)** -**4 - Infer clusters with the Louvain algorithm and store these labels to visualize later (auto)** -**5 - Compute and store umap and tsne embeddings (C)** -**6 - Write results to file (D)** - -# Example datasets to use with cellxgene - -**To download and use these datasets, run:** -`curl -O [URL]` -`unzip [filename.zip]` -`cellxgene launch [filename.h5ad] --open` - -### Peripheral blood mononuclear cells - -Healthy human PBMCs (10X). - -- Source: [10X genomics](https://support.10xgenomics.com/single-cell-gene-expression/datasets/1.1.0/pbmc3k) -- Cells: 2,638 -- File size: 19MB -- [Raw data](http://cf.10xgenomics.com/samples/cell-exp/1.1.0/pbmc3k/pbmc3k_filtered_gene_bc_matrices.tar.gz) -- [Processing](https://github.com/chanzuckerberg/cellxgene-vignettes/blob/master/dataset-processing/pbmc3k-processing.ipynb) -- Download: `curl -O https://cellxgene-example-data.czi.technology/pbmc3k.h5ad.zip` - -### Tabula muris - -20 organs and tissues from healthy mice (Smart-Seq2). -Rich metadata and annotations. - -- Source: [bioRxiv, CZBiohub](https://www.biorxiv.org/content/10.1101/237446v2) -- Cells: 45,423 -- File size: 174MB -- [Raw data](https://figshare.com/projects/Tabula_Muris_Transcriptomic_characterization_of_20_organs_and_tissues_from_Mus_musculus_at_single_cell_resolution/27733) -- [Processing](https://github.com/chanzuckerberg/cellxgene-vignettes/blob/master/dataset-processing/tabula-muris-processing.ipynb) -- Download: `curl -O https://cellxgene-example-data.czi.technology/tabula-muris.h5ad.zip` - -### Tabula muris senis - -22 organs and tissues from healthy mice at ages 3mo, 18mo, 21mo, and 24mo (Smart-Seq2). -Rich metadata and annotations. - -- Source: [bioRxiv, CZBiohub](https://www.biorxiv.org/content/10.1101/661728v1) -- Cells: 81,478 -- File size: 3.9GB -- Raw data [geo link coming soon!] -- [Processing](https://www.biorxiv.org/content/10.1101/661728v1) -- Download: `curl -O https://cellxgene-example-data.czi.technology/tabula-muris-senis.h5ad.zip` diff --git a/docs/faq.md b/docs/faq.md deleted file mode 100644 index d401e96d..00000000 --- a/docs/faq.md +++ /dev/null @@ -1,130 +0,0 @@ ---- -layout: default -title: FAQ -description: Frequently Asked Questions ---- - -# Data formatting - -#### What file formats can I use with _cellxgene_? - -Currently, you can go straight into `cellxgene launch` with your own analyzed data in h5ad format, after you have performed dimenstionality reduction (tsne, umap) and clustering (louvain). - -If your data is in a different format, and/or you still need to perform dimensionality reduction and clustering, `cellxgene` can do that for you with the `prepare` command. `cellxgene prepare` runs `scanpy` under the hood and can read in any format that is currently supported by `scanpy` (including mtx, loom, and more listed [here](https://scanpy.readthedocs.io/en/latest/api/index.html#reading)). - -To add `cellxgene prepare` to your cellxgene installation run `pip install cellxgene[prepare]`. - -The output of `cellxgene prepare` is a h5ad file with your computed clusters and tsne/umap projections that can be used in `cellxgene launch`. - -#### I have a directory of 10X-Genomics data with _mtx_ files and I've never used _scanpy_, can I use _cellxgene_? - -Yep! This should only take a couple steps. We'll assume your data is in a folder called `data/` and you've successfully installed `cellxgene` with the `prepare` packages as described above. Just run - -``` -cellxgene prepare data/ --output=data-processed.h5ad --embedding=umap -``` - -Depending on the size of the dataset, this may take some time. Once it's done, call - -``` -cellxgene launch data-processed.h5ad --embedding=umap --open -``` - -And your web browser should open with an interactive view of your data. - -#### I have extra metadata that I want to add to my dataset - -Currently this is not supported directly, but you should be able to do this yourself using `scanpy`. For example, this [notebook](https://github.com/falexwolf/fun-analyses/blob/master/tabula_muris/tabula_muris.ipynb) shows adding the contents of a `csv` file with metadata to an `anndata` object. For now, you could do this manually on your data in the same way and then save out the result before loading into `cellxgene`. - -#### What part of the _anndata_ objects does cellxgene pull in for visualization? - -- `.obs` and `.var` annotations are use to extract metadata for filtering -- `.X` is used to display expression (histograms, scatterplot & colorscale) and to compute differential expression -- `.obsm` is used for embedding(s). If an embedding has more than two components, the first two will be used for visualization. - -#### I have a BIG dataset - how can I make cellxgene run as fast as possible? - -If your dataset requires gigabytes of disk space, you may need to select an appropriate storage format in order to effectively utilize `cellxgene`. Tips and tricks: - -- `cellxgene` is optimized for columnar data access. For large datasets, format the expression matrix (`.X`) as either a [SciPy CSC sparse matrix](https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csc_matrix.html) or a dense Numpy array (whichever creates a smaller `h5ad` file). If you are using `cellxgene prepare`, include the `--sparse` flag to ensure `.X` is formatted as a CSC sparse matrix (by default, `.X` will be a dense matrix). -- `cellxgene` start time is directly proportional to `h5ad` file size and the speed of your file system. Expect that large (eg, million cell) datasets will take minutes to load, even on relatively fast computers with a high performance local hard drive. Once loaded, exploring metadata should still be quick. -- If your dataset size exceeds the size of memory (RAM) on the host computer, differential expression calculations will be extremely slow (or fail, if you run out of virtual memory). - -# Algorithms - -#### How are you computing and sorting differential expression results? - -We use a [Welch's _t_-test](https://en.wikipedia.org/wiki/Welch%27s_t-test) implementation including the same variance overestimation correction as used in `scanpy`. We sort the `tscore` to identify the top N genes, and then filter to remove any that fall below a cutoff log fold change value, which can help remove spurious test results. The default threshold is `0.01` and can be changed using the option `--diffexp-lfc-cutoff`. - -# Problems, errors, & bugs - -#### How do I create a Python environment for _cellxgene_? - -If you use conda and want to create a [conda environment](https://conda.io/docs/user-guide/tasks/manage-environments.html) for _cellxgene_ you can use the following commands - -``` -conda create --yes -n cellxgene python=3.7 -conda activate cellxgene -pip install cellxgene -``` - -Or you can create a virtual environment by using - -``` -ENV_NAME=cellxgene -python3.7 -m venv ${ENV_NAME} -source ${ENV_NAME}/bin/activate -pip install cellxgene -``` - -#### I ran _prepare_ and I'm getting results that look unexpected - -You might want to try running one of the preprocessing recipes included with `scanpy` (read more about them [here](https://scanpy.readthedocs.io/en/latest/api/index.html#recipes)). You can specify this with the `--recipe` option, such as - -``` -cellxgene prepare data/ --output=data-processed.h5ad --recipe=zheng17 -``` - -It should be easy to run `prepare` then call `cellxgene launch` a few times with different settings to explore different behaviors. We may explore adding other preprocessing options in the future. - -#### I tried to `pip install cellxgene` and got a weird error I don't understand - -This may happen, especially as we work out bugs in our installation process! Please create a new [Github issue](https://github.com/chanzuckerberg/cellxgene/issues), explain what you did, and include all the error messages you saw. It'd also be super helpful if you call `pip freeze` and include the full output alongside your issue. - -#### I'm following the developer instructions and get an error about "missing files and directories” when trying to build the client - -This is likely because you do not have node and npm installed, we recommend using [nvm](https://github.com/creationix/nvm) if you're new to using these tools. - -# Data access - -#### Can I use a _s3:_ or _gs:_ URL with `cellxgene launch`? - -Yes. Support for S3 and GCS is not enabled by default. If you wish to directly access S3 or GFS, install one or both of the following packages using `pip`: - -- [s3fs](https://s3fs.readthedocs.io/en/latest/) for S3 support -- [gcsfs](https://gcsfs.readthedocs.io/en/latest/) for GCS support - -For example: - -``` -pip install s3fs -cellxgene launch s3://mybucket.s3-us-west-2.amazonaws.com/mydata.h5ad -``` - -#### What does the command line option `--backed` do? - -The `--backed` option instructs `cellxgene launch` to read the H5AD file in "backed" mode (for more information, see the -[anndata.read_h5ad() documentation](https://anndata.readthedocs.io/en/latest/anndata.read_h5ad.html#anndata.read_h5ad)). - -By default, cellxgene will read the entire H5AD will be into memory at startup, improving application speed and performance. -Very large datasets may not fit in memory. The "--backed" mode will read the file incrementally, reducing memory -use, and for large files, improving startup speed. _However_, this option will also significantly slow -down access to gene expression histograms, and may render differential expression calculations too slow -to use (see `--disable-diffexp` for an option to disable this feature). - -#### What does the command line option `--disable-diffexp` do? - -The `--disable-diffexp` option will disable and hide the `Compute Differential Expression` feature. -For large datasets, or datasets loaded with the `--backed` option, computing differential expression may -be extremely slow or use excessive reources on the host computer (eg, memory thrasing). -Disabling the feature will ensure that the end-user does not accidentally initiate this computation. diff --git a/docs/getting-started.md b/docs/getting-started.md deleted file mode 100644 index 9bee50d0..00000000 --- a/docs/getting-started.md +++ /dev/null @@ -1,122 +0,0 @@ -## getting started - -You'll need **python 3.6+** and **Google Chrome**. -The web UI is tested on OSX and Windows using Chrome, and the python CLI is tested on OSX and Ubuntu (via WSL/Windows). It should work on other platforms, but if you run into trouble let us know. - -To install run - -``` -pip install cellxgene -``` - -To start exploring a dataset call - -``` -cellxgene launch dataset.h5ad --open -``` - -If you want an example dataset download [this file](https://github.com/chanzuckerberg/cellxgene/raw/master/example-dataset/pbmc3k.h5ad) and then call - -``` -cellxgene launch pbmc3k.h5ad --open -``` - -You can also directly specify URLs as a data source, and the data will be downloaded during launch - -``` -cellxgene launch https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/example-dataset/pbmc3k.h5ad -``` - -On Mac OS and Ubuntu, you should see your web browser open with the following - - - -**Note**: automatic opening of the browser with the `--open` flag only works on OS X, on other platforms you'll need to directly point to the provided link in your browser. - -There are several options available, such as: - -- `--embedding` to restrict available emdeddings in the UI, eg, `tsne`, `umap`, `diffmap`, `phate`, `draw_graph_fa`, or `draw_graph_fr` -- `--title` to show a title on the explorer -- `--open` to automatically open the web browser after launching (OS X only) - -To see all options call - -``` -cellxgene launch --help -``` - -There is an additional subcommand called `cellxgene prepare` that takes an existing dataset in one of several formats and applies minimal preprocessing and reformatting so that `launch` can use it (see [the next section](#data-formatting) for more info on `prepare`). - -## data formatting - -### requirements - -The `launch` command assumes that the data is stored in the `.h5ad` format from the [`anndata`](https://anndata.readthedocs.io/en/latest/index.html) library. It also assumes that certain computations have already been performed. Briefly, the `.h5ad` format wraps a two-dimensional `ndarray` and stores additional metadata as "annotations" for either observations (referred to as `obs` and `obsm`) or variables (`var` and `varm`). `cellxgene launch` makes the following assumptions about your data (we recommend loading and inspecting your data using `scanpy` to validate these assumptions) - -- an `obs` field has a unique identifier for every cell (you can specify which field to use with the `--obs-names` option, by default it will use the value of `data.obs_names`) -- a `var` field has a unique identifier for every gene (you can specify which field to use with the `--var-names` option, by default it will use the value of `data.var_names`) -- an `obsm` field contains the two-dimensional coordinates for the embedding that you want to render (e.g. `X_umap` for the `umap` embedding) -- any additional `obs` fields will be rendered as per-cell continuous or categorical metadata by the app (e.g. `louvain` cluster assignments) - -### prepare - -The `prepare` command is included to help you format your data. It uses `scanpy` under the hood. This is especially useful if you are starting with raw unanalyzed data and are unfamiliar with `scanpy`. - -To install `cellxgene prepare` alongside `cellxgene launch` - -``` -pip install cellxgene[prepare] -``` - -If the aforementioned optional package installation fails, you can also install these packages directly: - -``` -pip install scanpy>=1.3.7 python-igraph louvain>=0.6 -``` - -To prepare from an existing `.h5ad` file use - -``` -cellxgene prepare dataset.h5ad --output=dataset-processed.h5ad -``` - -This will load the input data, perform PCA and nearest neighbor calculations, compute `umap` and `tsne` embeddings and `louvain` cluster assignments, and save the results in a new file called `dataset-processed.h5ad` that can be loaded using `cellxgene launch`. Data can be loaded from several formats, including `.h5ad` `.loom` and a `10-Genomics-formatted` `mtx` directory. Several options are available, including running one of the preprocessing `recipes` included with `scanpy`, which include steps like cell filtering and gene selection. To learn more about the `recipes` please see the `scanpy` [documentation](https://scanpy.readthedocs.io/en/latest/api/index.html#recipes). - -Depending on the options chosen, `prepare` can take a long time to run (a few minutes for datasets with 10-100k cells, up to an hour or more for datasets with >100k cells). If you want `prepare` to run faster we recommend using the `sparse` option and only computing the embedding for `umap`, using a call like this - -``` -cellxgene prepare dataset.h5ad --output=dataset-processed.h5ad --embedding=umap --sparse -``` - -To see all options call - -``` -cellxgene prepare --help -``` - -## conda and virtual environments - -If you use conda and want to create a conda environment for `cellxgene` you can use the following commands - -``` -conda create --yes -n cellxgene python=3.7 -conda activate cellxgene -pip install cellxgene -``` - -Or you can create a virtual environment by using - -``` -ENV_NAME=cellxgene -python3.7 -m venv ${ENV_NAME} -source ${ENV_NAME}/bin/activate -pip install cellxgene -``` - -## docker - -We have included a dockerfile to conveniently run cellxgene from docker. - -1. Build the image `docker build . -t cellxgene` -2. Run the container and mount data `docker run -v "$PWD/example-dataset/:/data/" -p 5005:5005 cellxgene launch --host 0.0.0.0 data/pbmc3k.h5ad` - - You will need to use --host 0.0.0.0 to have the container listen to incoming requests from the browser diff --git a/docs/cellxgene-demo-1.gif b/docs/images/cellxgene-demo-1.gif similarity index 100% rename from docs/cellxgene-demo-1.gif rename to docs/images/cellxgene-demo-1.gif diff --git a/docs/cellxgene-demo-2.gif b/docs/images/cellxgene-demo-2.gif similarity index 100% rename from docs/cellxgene-demo-2.gif rename to docs/images/cellxgene-demo-2.gif diff --git a/docs/cellxgene-demo-3.gif b/docs/images/cellxgene-demo-3.gif similarity index 100% rename from docs/cellxgene-demo-3.gif rename to docs/images/cellxgene-demo-3.gif diff --git a/docs/cellxgene-demo.gif b/docs/images/cellxgene-demo.gif similarity index 100% rename from docs/cellxgene-demo.gif rename to docs/images/cellxgene-demo.gif diff --git a/docs/cellxgene-opening-screenshot.png b/docs/images/cellxgene-opening-screenshot.png similarity index 100% rename from docs/cellxgene-opening-screenshot.png rename to docs/images/cellxgene-opening-screenshot.png diff --git a/docs/customGene.gif b/docs/images/customGene.gif similarity index 100% rename from docs/customGene.gif rename to docs/images/customGene.gif diff --git a/docs/diffexp.gif b/docs/images/diffexp.gif similarity index 100% rename from docs/diffexp.gif rename to docs/images/diffexp.gif diff --git a/docs/index.md b/docs/index.md index 1e029c7c..a80dbeb4 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,17 +1,11 @@ - - -_an interactive explorer for single-cell transcriptomics data_ +--- +title: Index +subtitle: Index +layout: default +--- +# Quick start Whether you need to visualize one thousand cells or one million, cellxgene helps you gain insight into your single-cell data. -## features - -#### flexible selections, coloring, and differential expression of your selected sets of cells - - -#### single-gene analyses (e.g. expression analysis) - - -## quick start To install cellxgene you need Python 3.6+. We recommend [installing cellxgene into a conda or virtual environment.](/faq.html#how-do-i-create-a-python-environment-for-cellxgene) @@ -23,19 +17,20 @@ pip install cellxgene Download an example [anndata](https://anndata.readthedocs.io/en/latest/) file ``` bash -curl -o pbmc3k.h5ad https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/example-dataset/pbmc3k.h5ad +curl -o tabula-muris.h5ad https://cellxgene-example-data.czi.technology/tabula-muris.h5ad.zip +unzip tabula-muris.h5ad.zip ``` Launch cellxgene ``` bash -cellxgene launch pbmc3k.h5ad --open +cellxgene launch tabula-muris.h5ad --open ``` -To explore more datasets already formatted for cellxgene, see [Data](data) or -visit [Getting Started](getting-started) to learn more about formatting your own +To explore more datasets already formatted for cellxgene, check out the [Demo data](demo-data) or +see [Preparing your data](prepare) to learn more about formatting your own data for cellxgene. -## getting help +# Getting help We'd love to hear from you! diff --git a/docs/posts/annotations.md b/docs/posts/annotations.md new file mode 100644 index 00000000..e60f6295 --- /dev/null +++ b/docs/posts/annotations.md @@ -0,0 +1,55 @@ +--- +layout: default +title: annotations +description: Creating annotations +--- + +# Creating annotations in cellxgene + +We are _piloting_ a new feature in cellxgene that enables users to create and edit categorical annotations within the app. + +## Quick start for annotations + +You can enable this experimental feature like so: + +`cellxgene launch mydata.h5ad --experimental-label-file myannotations.csv` + +Any annotations you create in the application will be autosaved in `myannotations.csv`. + +## Data management + +To preserve data provenance, **`cellxgene` does not alter the input h5ad file**. + +Rather, newly-created annotations are saved in the specified CSV file. If the specified file already exists, the previously-created annotations will be loaded as mutable (changeable) values and the CSV will be updated (overwritten) with any edits made. This allows you to create annotations and then come back later to continue working on them. + +Once you're finished with your annotations, you should finalize and preserve your work by merging your `csv` into your main `h5ad` file. + +You can do so like this: +``` +import pandas as pd +import scanpy as sc + +new_annotations = pd.read_csv('myannotations.csv', + comment='#', + dtype='category', + index_col=0) +anndata = sc.read('mydata.h5ad') +anndata.obs = anndata.obs.join(new_annotations) +``` + +## FAQ + +### How do I know my annotations are saved? + +`cellxgene` autosaves any changes made to your annotations every 3 seconds. + +### What about creating continuous annotations? + +Continuous metadata is important! However, these values (e.g., pseudotime) are the result of statistical analyses that are beyond cellxgene's visualization- and exploration-focused scope. We do, of course, provide visualization of continuous metadata values computed elsewhere and stored in `anndata.obs`. + +### I keep getting weird index errors when trying to join my annotations to my anndata?? +This is most likely because the h5ad file you are working with is not the original file used to generate the annotations! We recommend merging new annotations in on a regular basis for this reason. + +### I have feedback and ideas for you! + +Wonderful! This is a very new and complex feature; we would LOVE to [hear your feedback](contact) :) diff --git a/docs/posts/contact.md b/docs/posts/contact.md new file mode 100644 index 00000000..37389381 --- /dev/null +++ b/docs/posts/contact.md @@ -0,0 +1,26 @@ +--- +layout: default +title: Contact +description: Contact +--- +# Get in touch + +We'd love to hear from you! + +For questions, suggestions, or accolades, [join the `#cellxgene-users` channel on the CZI Science Slack](https://join-cellxgene-users.herokuapp.com/) and say "hi!". + +For any errors, [report bugs on Github](https://github.com/chanzuckerberg/cellxgene/issues). + +# Team + +The current core team: + +- Colin Megill, frontend & product design +- Bruce Martin, software engineer +- Sidney Bell, computational biologist +- Lia Prins, designer +- Severiano Badajoz, software engineer + +We would also like to gratefully acknowledge contributions from past core team members: + +- Charlotte Weaver, software engineer diff --git a/docs/posts/contribute.md b/docs/posts/contribute.md new file mode 100644 index 00000000..02b71f13 --- /dev/null +++ b/docs/posts/contribute.md @@ -0,0 +1,70 @@ +# Code of conduct +We warmly welcome contributions from the community! + +To ensure a welcoming experience for our entire community, this project adheres to the Contributor Covenant +[code of conduct](https://github.com/chanzuckerberg/.github/tree/master/CODE_OF_CONDUCT.md). +By participating, you are expected to uphold this code. Please report unacceptable behavior +to opensource@chanzuckerberg.com. + +If you have any questions about any of this stuff, just ask! :) + +# Contributing ideas and issues + +We'd love to hear from you! Please submit any bug reports and feature requests through [Github issues](https://github.com/chanzuckerberg/cellxgene/issues). + +# Direct contributions +## Getting started + +If you are interested in working on `cellxgene` development, you'll need to use git to make a copy of the [project repository](https://www.youtube.com/watch?v=A-4WltCTVms&list=PLe6EXFvnTV7-_41SpakZoTIYCgX4aMTdU&index=2&t=0s) and share your changes. +If you're new to git, we recommend [GitKraken](https://www.gitkraken.com/) for an intuitive interface. + +Please submit any direct contributions by [forking the repository](https://www.youtube.com/watch?v=Lb4yvfrX_7I&list=PLe6EXFvnTV7-_41SpakZoTIYCgX4aMTdU&index=3&t=9s), creating a branch, and [submitting a Pull Request](https://www.youtube.com/watch?v=2VX1ISk9XH8&list=PLe6EXFvnTV7-_41SpakZoTIYCgX4aMTdU&index=9&t=0s). + +First, you'll need the following installed on your machine + +- python 3.6+ +- node and npm (we recommend using [nvm](https://github.com/creationix/nvm) if this is your first time with node) + +Then clone the project + +``` +git clone https://github.com/chanzuckerberg/cellxgene.git +``` + +This is enough to get you started with editing documentation. If you'd like to contribute code: + +Build the client web assets by calling `make` from inside the `cellxgene` folder + +``` +make +``` + +Install all requirements (we recommend doing this inside a [virtual environment](install)) + +``` +pip install -e . +``` + +You can start the app while developing either by calling `cellxgene` or by calling `python -m server`. We recommend using the `--debug` flag to see more output, which you can include when reporting bugs. + +If you have any questions about developing or contributing, come hang out with us by joining the [CZI Science Slack](https://join-cellxgene-users.herokuapp.com/) and posting in the `#cellxgene-dev` channel. + +## Contributing code + +This project has made a few key design choices: + +- The front-end is built with [`regl`](https://github.com/regl-project/regl) (a webgl library), [`react`](https://reactjs.org/), [`redux`](https://redux.js.org/), [`d3`](https://github.com/d3/d3), and [`blueprint`](https://blueprintjs.com/docs/#core) to handle rendering large numbers of cells with lots of complex interactivity +- The app is designed with a client-server model that can support a range of existing analysis packages for Python-based backend computational tasks (currently built for [scanpy](https://github.com/theislab/scanpy)) +- The client uses fast cross-filtering to handle selections and comparisons across subsets of data + +Depending on your background and interests, you might want to contribute to the frontend, or backend, or both! + +Please submit any direct contributions via a Pull Request. It'd be great for PRs to include test cases and documentation updates where relevant, though we know the core test suite is itself still a work in progress. + +## Contributing documentation + +The documentation is written in [markdown](https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet), and lives in the directory `cellxgene/docs/posts`. You can directly edit or add to these files and submit a Pull Request as described above. + +To preview your changes on your local machine, you'll need to install Jekyll and Ruby using [these instructions](https://jekyllrb.com/docs/installation/) (you don't have to know how to program in Ruby, just install it). + +You can then preview your changes by running `cellxgene/docs$ bundle exec jekyll serve` and navigating to the url indicated in the terminal. diff --git a/docs/posts/demo-data.md b/docs/posts/demo-data.md new file mode 100644 index 00000000..07d7de62 --- /dev/null +++ b/docs/posts/demo-data.md @@ -0,0 +1,43 @@ +--- +layout: default +title: demo-data +description: Demo datasets +--- + +# Example datasets +**To download and use these datasets, run:** +`curl -O [URL]` +`unzip [filename.zip]` +`cellxgene launch [filename.h5ad] --open` + +### Peripheral blood mononuclear cells +Healthy human PBMCs (10X). + +- Source: [10X genomics](https://support.10xgenomics.com/single-cell-gene-expression/datasets/1.1.0/pbmc3k) +- Cells: 2,638 +- File size: 19MB +- [Raw data](http://cf.10xgenomics.com/samples/cell-exp/1.1.0/pbmc3k/pbmc3k_filtered_gene_bc_matrices.tar.gz) +- [Processing](https://github.com/chanzuckerberg/cellxgene-vignettes/blob/master/dataset-processing/pbmc3k-processing.ipynb) +- Download: `curl -O https://cellxgene-example-data.czi.technology/pbmc3k.h5ad.zip` + +### Tabula muris +20 organs and tissues from healthy mice (Smart-Seq2). +Rich metadata and annotations. + +- Source: [bioRxiv, CZBiohub](https://www.biorxiv.org/content/10.1101/237446v2) +- Cells: 45,423 +- File size: 174MB +- [Raw data](https://figshare.com/projects/Tabula_Muris_Transcriptomic_characterization_of_20_organs_and_tissues_from_Mus_musculus_at_single_cell_resolution/27733) +- [Processing](https://github.com/chanzuckerberg/cellxgene-vignettes/blob/master/dataset-processing/tabula-muris-processing.ipynb) +- Download: `curl -O https://cellxgene-example-data.czi.technology/tabula-muris.h5ad.zip` + +### Tabula muris senis +22 organs and tissues from healthy mice at ages 3mo, 18mo, 21mo, and 24mo (Smart-Seq2). +Rich metadata and annotations. + +- Source: [bioRxiv, CZBiohub](https://www.biorxiv.org/content/10.1101/661728v1) +- Cells: 81,478 +- File size: 3.9GB +- Raw data [geo link coming soon!] +- [Processing](https://www.biorxiv.org/content/10.1101/661728v1) +- Download: `curl -O https://cellxgene-example-data.czi.technology/tabula-muris-senis.h5ad.zip` diff --git a/docs/posts/gallery.md b/docs/posts/gallery.md new file mode 100644 index 00000000..fadb303d --- /dev/null +++ b/docs/posts/gallery.md @@ -0,0 +1,25 @@ +--- +title: Gallery +subtitle: Features & examples +layout: default +--- + +# Features + +### Flexible selections, coloring, and differential expression of your selected sets of cells + + +### Single-gene analyses (e.g. expression analysis) + + +# Interactive demos + +Several groups have independently [deployed](hosted) various versions of cellxgene to the web. +Check out the cool data that our users are using cellxgene to explore! + +### [Kidney cell atlas](https://www.kidneycellatlas.org/) +### [Tabula muris senis](https://tabula-muris-senis.ds.czbiohub.org/) +### [Hemocytes](https://hemocytes.cellgeni.sanger.ac.uk/) +### [Melanoma](https://melanoma.cellgeni.sanger.ac.uk/) + +_Want us to link to your dataset here? [Just send us a note!](contact)_ diff --git a/docs/posts/hosted.md b/docs/posts/hosted.md new file mode 100644 index 00000000..16944038 --- /dev/null +++ b/docs/posts/hosted.md @@ -0,0 +1,64 @@ +# Hosting cellxgene on the web + +Cellxgene is intended to be used by researchers on their local machines. However, we recognize that sharing and exploring data on the web is important. We're exploring how we could better support this in the future, and [would welcome your input](https://github.com/chanzuckerberg/cellxgene/issues/875)! + +In the meantime, you can see examples of how other groups have approached this [here](gallery). While we don't officially support web deployment, we've offered some guidance below on one way to deploy cellxgene to the web. + +# Deploying cellxgene with Heroku + +## Quickstart + +Clicking on the following button will forward you to Heroku to begin the deployment process: + + + Deploy + + +If not already logged in to Heroku, there you will be prompted to log in or sign up for an account. + +Once logged in you will be sent to the setup page. Here you can set some of the basic settings for the app: + +### Default settings + +- `App name`: the unique name for your deployment +- This will also serve as the default URL (e.g. https://cellxgene.herokapp.com/) +- `App owner`: Who will own this app. Either you personally or an organization/team +- `Region`: Location of the server where the app will be deployed (EU or US) + +### Configuration + +- `DATASET`: A _publicly_ accessible URL pointing to a .h5ad file to view +- This defaults to pbm3k.h5ad + +After filling out the settings and pressing the `Deploy app` button Heroku will begin building your deployment. This process will take a few minutes, but once completed you will have a personal free hosted version of cellxgene! + +## What is Heroku? + +Heroku is a quick and easy way to host applications on the cloud. + +A Heroku deployment of cellxgene means that the app is not running on your local machine. Instead, the app is installed, configured, and ran on the Heroku servers (read: cloud). + +On Heroku's servers, applications run on a [dyno](https://www.heroku.com/dynos) which are Heroku's implementation and abstraction of containers. + +Heroku is one of many options available for hosting instances of cellxgene on the web. +Some other options include: Amazon Web Services, Google Cloud Platform, Digital Ocean, and Microsoft Azure. + +## Why use Heroku to deploy cellxgene? + +What Heroku enables is a quick, non-technical method of setting up a cellxgene instance. No command line knowledge needed. This also allows machines to access the instance via the internet, so sharing a visualized dataset is as simple as sharing a link. + +Because cellxgene currently heavily relies on its Python backend for providing the viewer with the necessary data and tooling, it is currently not possible to host cellxgene as a static webpage. + +This is a good option if you want to quickly deploy an instance of cellxgene to the web. Heroku deployments are free for small datasets up to around 250MBs in size. See below regarding larger datasets. + +## When should I not deploy with Heroku? + +- The default free dyno offered by Heroku is limited in memory to 512 MBs + - The amount of memory needed for the dyno is roughly the same size as the h5ad file + - Heroku offers tiered paid dynos. More can be found [here](https://www.heroku.com/pricing) + - Note that this can get _very_ expensive for larger datasets (\$25+ a month) +- On the free dyno, after 30 minutes of inactivity, Heroku will put your app into a hibernation mode. On the next access, Heroku will need time to boot the dyno back online. +- Having multiple simultaneous users requires more memory. This means that the free container size is easily overwhelmed by multiple users, even with small datasets; this can be addressed by purchasing a larger container size +- For this facilitated Heroku deployment to work, your dataset must be hosted on a publicly accessible URL +- By default, Heroku publically shares your instance to anyone with the URL. + - There are many ways of securing your instance. One quick and simple way is by installing [wwwhisper](https://elements.heroku.com/addons/wwwhisper), a Heroku addon diff --git a/docs/posts/index.html b/docs/posts/index.html new file mode 100644 index 00000000..13cce65b --- /dev/null +++ b/docs/posts/index.html @@ -0,0 +1,109 @@ + + + + + + + + +Index | cellxgene + + + + + + + + + + + + + + + +
+
+ cellxgene + +

An interactive explorer for single-cell transcriptomics data

+

+ + Quick start
+ + Gallery
+ + Installation
+ + Demo datasets
+ + Preparing your data
+ + Launching cellxgene
+ + Hosting cellxgene
+ + Contributing (ideas or code)
+ + Methods
+ + FAQ
+ + Roadmap
+ + Contact & finding help
+ + + + Code + +

+
+ +
+

Quick start

+ +

Whether you need to visualize one thousand cells or one million, cellxgene helps you gain insight into your single-cell data.

+ +

To install cellxgene you need Python 3.6+. We recommend installing cellxgene into a conda or virtual environment.

+ +

Install the package.

+
pip install cellxgene
+
+ +

Download an example anndata file

+ +
curl -o tabula-muris.h5ad https://cellxgene-example-data.czi.technology/tabula-muris.h5ad.zip
+unzip tabula-muris.h5ad.zip
+
+ +

Launch cellxgene

+
cellxgene launch tabula-muris.h5ad --open
+
+ +

To explore more datasets already formatted for cellxgene, check out the Demo data or +see Preparing your data to learn more about formatting your own +data for cellxgene.

+ +

Getting help

+ +

We’d love to hear from you!

+ +

For questions, suggestions, or accolades, join the #cellxgene-users channel on the CZI Science Slack and say “hi!”.

+ +

For any errors, report bugs on Github.

+ +
+ +
+ + + + diff --git a/docs/posts/install.md b/docs/posts/install.md new file mode 100644 index 00000000..0f0c892c --- /dev/null +++ b/docs/posts/install.md @@ -0,0 +1,68 @@ +--- +title: Install +subtitle: Install +layout: default +--- +# Installing cellxgene + +Cellxgene has two parts: +* [`cellxgene`](launch) is the main explorer application, which takes an already-processed `h5ad` file as input. This is installed by default. +* [`cellxgene prepare`](prepare) provides auxiliary functionality for preparing your dataset. This is *not* installed by default. + +## Requirements +You'll need **python 3.6+** and **Google Chrome**. +The web UI is tested on OSX and Windows using Chrome, and the python CLI is tested on OSX and Ubuntu (via WSL/Windows). +It should work on other platforms, but if you run into trouble let us know. + +## Basic install +To install `cellxgene` alone, run: +`pip install cellxgene` + +To install `cellxgene` and the optional `cellxgene prepare`, run: +`pip install cellxgene[prepare]` + +_Note: if the aforementioned optional `prepare` package installation fails, you can also install these packages directly:_ +``` +pip install scanpy>=1.3.7 python-igraph louvain>=0.6 +``` + +## Using a conda environment +To install `cellxgene` alone, run: +``` +conda create --yes -n cellxgene python=3.7 +conda activate cellxgene +pip install cellxgene +``` + +To install `cellxgene` and the optional `cellxgene prepare`, run: +``` +conda create --yes -n cellxgene python=3.7 +conda activate cellxgene +pip install cellxgene[prepare] +``` + +## Using a virtual environment +To install `cellxgene` alone, run: +``` +ENV_NAME=cellxgene +python3.7 -m venv ${ENV_NAME} +source ${ENV_NAME}/bin/activate +pip install cellxgene +``` + +To install `cellxgene` and `cellxgene prepare`, run: +``` +ENV_NAME=cellxgene +python3.7 -m venv ${ENV_NAME} +source ${ENV_NAME}/bin/activate +pip install cellxgene[prepare] +``` + +## Using docker +Build the image +`docker build . -t cellxgene` + +Run the container and mount data +`docker run -v "$PWD/example-dataset/:/data/" -p 5005:5005 cellxgene launch --host 0.0.0.0 data/pbmc3k.h5ad` + +You will need to use `--host 0.0.0.0` to have the container listen to incoming requests from the browser diff --git a/docs/posts/launch.md b/docs/posts/launch.md new file mode 100644 index 00000000..98dd5dcb --- /dev/null +++ b/docs/posts/launch.md @@ -0,0 +1,71 @@ +--- +layout: default +title: demo-data +description: Demo datasets +--- + +# Launching cellxgene with your dataset + +## Quickstart +Once you've [prepared your data](prepare) for cellxgene, you can launch the app using +``` +cellgene launch mydataset.h5ad --open +``` +On Mac OS and Ubuntu, you should see your web browser open with the following + + + +**Note**: automatic opening of the browser with the `--open` flag only works on OS X, on other platforms you'll need to directly point to the provided link in your browser. + +## Launching from a URL + +You can also launch from a URL directly like this: +``` +cellxgene launch https://github.com/chanzuckerberg/cellxgene/blob/master/example-dataset/pbmc3k.h5ad +``` +Support for S3 and GCS is not enabled by default. If you wish to directly access S3 or GFS, install one or both of the following packages using `pip`: + +- [s3fs](https://s3fs.readthedocs.io/en/latest/) for S3 support +- [gcsfs](https://gcsfs.readthedocs.io/en/latest/) for GCS support + +For example: +``` +pip install s3fs +cellxgene launch s3://mybucket.s3-us-west-2.amazonaws.com/mydata.h5ad +``` + +## Options for cellxgene `launch` + +**For the most up-to-date and comprehensive list of options, run `cellxgene launch --help`** + +`--open` automatically opens the web browser after launching (OS X only). + +`--experimental-label-file` enables an experimental feature to allow users to create new categorical annotations in the application. These newly-created annotations are saved in the specified CSV file. If the specified file already exists, the previously-contained annotations will be loaded as mutable (changeable) values and the CSV will be updated (overwritten) with any edits made. If the file does not exist, it will be created. See [here](annotations) for more details. + +`--diffexp-lfc-cutoff` as explained [here](methods), genes are only returned in differential expression if the effect size is above the specified threshold for log fold change. Defaults to 0.01. + +`--disable-diffexp` will disable and hide the `Compute Differential Expression` feature. +For large datasets, or datasets loaded with the `--backed` option, computing differential expression may +be extremely slow or use excessive reources on the host computer (eg, memory thrasing). +Disabling the feature will ensure that the end-user does not accidentally initiate this computation. + +`--backed` option instructs `cellxgene launch` to read the H5AD file in "backed" mode (for more information, see the +[anndata.read_h5ad() documentation](https://anndata.readthedocs.io/en/latest/anndata.read_h5ad.html#anndata.read_h5ad)). + +By default, cellxgene will read the entire H5AD will be into memory at startup, improving application speed and performance. +Very large datasets may not fit in memory. The "--backed" mode will read the file incrementally, reducing memory +use, and for large files, improving startup speed. _However_, this option will also significantly slow +down access to gene expression histograms, and may render differential expression calculations too slow +to use (see `--disable-diffexp` for an option to disable this feature). + +`--embedding` restricts which embeddings will be available in the viewer. By default, all embeddings specified in `anndata.obsm['X_name']` will be loaded; if you have many embeddings, you may wish to restrict this list for a speedier launch. + +`--title` adds a title to the viewer. Defaults to file name. + +`--about` adds a link where users can go to find more infomation about the dataset. Requires `https`. + +`--obs-names` allows you to specify which column in `anndata.obs` to use as `anndata.obs.index`. + +`--var-names` allows you to specify which column in `anndata.var` to use as `anndata.var.index`. + +`--max-category-items` omits categorical metadata fields that contain more than N _distinct_ values. Defaults to 1000. diff --git a/docs/posts/methods.md b/docs/posts/methods.md new file mode 100644 index 00000000..907e6f8d --- /dev/null +++ b/docs/posts/methods.md @@ -0,0 +1,21 @@ +# Methods + +## Data structure: _anndata_ fields used for visualization + +### Expression data + +Gene expression values are pulled from `anndata.X`. These feed into the histograms, scatterplot, colorscale, and differential expression calculations. We're [working on ways](https://github.com/chanzuckerberg/cellxgene/issues/689) to incorporate `anndata.raw` and other `anndata.layers`! + +### Metadata + +Categorical (e.g., cluster labels) and continuous (e.g., pseudotime) metadata are pulled from `anndata.obs`. Any column added here will be available for visualization in cellxgene. You can also [create new categorical annotations](annotations) within the application. + +### Embeddings + +Cellxgene looks for embeddings (e.g., tSNE, UMAP, PCA, spatial coordinates) in `anndata.obsm`. These fields must follow the scanpy convention of starting with `X_`, e.g., `anndata.obsm['X_umap']`. If an embedding has more than two components, the first two will be used for visualization. + +## Differential expression + +We're actively working on how to improve differential expression within the app. + +Currently, we use a [Welch's _t_-test](https://en.wikipedia.org/wiki/Welch%27s_t-test) implementation, including the same variance overestimation correction as used in `scanpy`. We sort the `tscore` to identify the top 15 genes, and then filter to remove any that fall below a cutoff log fold change value, which can help remove spurious test results. The default threshold is `0.01` and can be changed using the option `--diffexp-lfc-cutoff`. diff --git a/docs/posts/prepare.md b/docs/posts/prepare.md new file mode 100644 index 00000000..45cc8a75 --- /dev/null +++ b/docs/posts/prepare.md @@ -0,0 +1,104 @@ +--- +layout: default +title: prepare +description: Preparing your data +--- +# Data format requirements + +If your data is in `h5ad` file (from the [`anndata`](https://anndata.readthedocs.io/en/latest/index.html) library) and meets the following requirements, you can go straight to `cellxgene launch`: + +- Expression values (raw or normalized) in `anndata.X` +- At least one embedding (e.g., tSNE, UMAP) in `anndata.obsm`, specified with the prefix `X_` (e.g., by default scanpy stores UMAP coordinates in `anndata.obsm['X_umap']`) +- A unique identifier for every cell is available in an `anndata.obs` field (you can specify this with the `--obs-names` option) +- A unique identifier for every gene is available in an `anndata.var` field (you can specify which field to use with the `--var-names` option) + +#### What about R objects from seurat / bioconductor!?** +We hear you! We'd also love to be able to ingest these files directly. This isn't currently possible, but in the meantime, you can use one of [these handy adapters](https://satijalab.org/seurat/v3.0/conversion_vignette.html) to convert to `h5ad`. + +#### Can I use data hosted on the web somewhere? +Yes! You can launch from a URL instead of a filepath. The same data format requirements apply. Please see [here](launch) for more details. + +# Using cellxgene `prepare` + +If your data is in a different format, and/or you still need to perform dimensionality reduction and/or clustering, `cellxgene` can do that for you with the `prepare` command. + +## What is cellxgene `prepare`? + +`cellxgene prepare` offers an easy command line interface (CLI) to preliminarily wrangle your data into the required format for previewing it with `cellxgene`. It runs `scanpy` under the hood and can read in any format that is currently supported by `scanpy` (including mtx, loom, and more listed [here](https://scanpy.readthedocs.io/en/latest/api/index.html#reading)). + +`prepare` uses scanpy to: + +- Handle simple data normalization (from a [recipe](https://www.pydoc.io/pypi/scanpy-0.2.3/autoapi/preprocessing/recipes/index.html)) +- Do basic preprocessing to run PCA and compute the neighbor graph +- Infer clusters +- Reduce dimensionality to generate embeddings. + +You can control which steps to run and their methods (when applicable), via the CLI. The CLI also includes options for computing QC metrics, enforcing matrix sparcity, specifying index names, and plotting output. + +## What is cellxgene `prepare` _not_? + +`cellxgene prepare` is not meant as a way to formally process or analyze your data. It's simply a utility for quickly wrangling your data into cellxgene-compatible format and computing a "vanilla" embedding so you can try out `cellxgene` and get a general sense of a dataset. + +## Quickstart for `cellxgene prepare` +To add `cellxgene prepare` to your [cellxgene installation](install), run +`pip install cellxgene[prepare]` + +Then run `prepare` on your data with: +``` +cellxgene prepare dataset.h5ad --output=dataset-processed.h5ad +``` + +This will load the input data, perform PCA and nearest neighbor calculations, compute `umap` and `tsne` embeddings and `louvain` cluster assignments, and save the results in a new file called `dataset-processed.h5ad` that can be loaded using `cellxgene launch`. + +## Example usage + +As a quick example, let's construct a command to use `prepare` to take a raw expression matrix and generate a processed `h5ad` ready to visualize with cellxgene. + +We'll start off using the raw data from the pbmc3k dataset. This dataset is described [here](https://icb-scanpy.readthedocs-hosted.com/en/stable/api/scanpy.datasets.pbmc3k.html), and is available as part of the scanpy package. For this example, we'll assume this raw data is stored in a file called `pbmc3k-raw.h5ad`. + +Our `prepare` compose our command looks like this: + +``` +cellxgene prepare pbmc3k-raw.h5ad \ + --run-qc \ # (A) + --recipe seurat \ # (B) + --layout tsne --layout umap \ # (C) + --output pbmc3k-prepared.h5ad # (D) +``` + +Let's look at what `prepare` is doing to our data, and how each step relates to the command above. You can see a walkthrough of what's going on under the hood for this example in [this notebook](https://github.com/chanzuckerberg/cellxgene-vignettes/blob/master/dataset-processing/pbmc3k-prepare-example.ipynb). + +**(A) - Compute quality control metrics and store this in our `AnnData` object for later inspection** +**(B) - Normalize the expression matrix using a basic preprocessing recipe** +**(auto) - Do some preprocessing to run PCA and compute the neighbor graph** +**(auto) - Infer clusters with the Louvain algorithm and store these labels to visualize later** +**(C) - Compute and store umap and tsne embeddings** +**(D) - Write results to file** + +## Options for cellxgene `prepare` + +**For the most up-to-date and comprehensive list of options, run `cellxgene prepare --help`** + +`--embedding` controls which dimensionality reduction algorithm is applies to your data. +Options are `umap` and/or `tsne`. Defaults to both. + +`--recipe` controls which normalization steps to apply to your data, based on one of the preprocessing `recipes` included with `scanpy`. +These recipes include steps like cell filtering and gene selection; see the `scanpy` [documentation](https://scanpy.readthedocs.io/en/latest/api/index.html#recipes) for more details. +Options are `none`, `seurat`, or `zheng17`. Defaults to `none`. + +`--sparse` is a flag determines whether to enforce a sparse matrix. For large datasets, `prepare` can take a long time to run (a few minutes for datasets with 10-100k cells, up to an hour or more for datasets with >100k cells). If you want `prepare` to run faster we recommend using the `sparse` option. +If this flag is not included, default is `False` + +`--run-qc / --skip-qc` is a flag that determines whether to calculate QC metrics (saved to `anndata.obs` and `anndata.var`). See the `scanpy` [documentation](https://scanpy.readthedocs.io/en/stable/api/scanpy.pp.calculate_qc_metrics.html) for details. +Default is `run-qc` + +`--make-obs-names-unique` / `--make-var-names-unique` are flags that determine whether to rename `obs` (cell) / `var` (gene) names, respectively, to be unique. +Default is `True` + +`--set-obs-names` controls which field in `anndata.obs` (cell metadata) is used as the _index_ for cells (e.g., a cell ID column). +Default is `anndata.obs.names` + +`--set-var-names` controls which field in `anndata.var` (gene metadata) is used as the _index_ for genes. +Default is `anndata.var.names` + +`--output` and `--overwrite` control where the processed data is saved. diff --git a/docs/posts/roadmap.md b/docs/posts/roadmap.md new file mode 100644 index 00000000..0ee97520 --- /dev/null +++ b/docs/posts/roadmap.md @@ -0,0 +1,52 @@ +--- +layout: default +title: roadmap +description: Roadmap +--- + +# Roadmap + +cellxgene makes it easier for biologists to collaboratively explore and understand their single-cell RNA-seq data. +In the near term, we are focused on continuing to enable fast, interactive exploration of single-cell data, supporting collaborative workflows in single-cell analysis, and improving user support. +If you have questions or feedback about this roadmap, please submit an issue on GitHub. +Please note: this roadmap is subject to change. + +Last updated: June 25, 2019 + +## Fast, interactive exploration of single-cell data + +### Exposing Relationships Between Metadata and Data +Biologists need to understand how variables (stored in metadata) are associated with one another and how they relate to changes in gene expression. +Building upon visualization features that reveal categorical metadata relationships (cluster occupancy) and gene expression relationships (scatterplot), we plan to add exploratory visualization components that enable investigation of relationships between metadata and gene expression. +See [issue #616](https://github.com/chanzuckerberg/cellxgene/issues/616) for more details. + +### Contextualizing Genes +While exploring a transcriptomics dataset, scientists need to understand the biological context of genes. +This context may be provided by user-defined gene metadata or publicly available gene databases. +We plan to support augmenting gene names with additional information that is useful to biologists. +See [issue #96](https://github.com/chanzuckerberg/cellxgene/issues/96) for more detail. + +## Support collaborative workflows in single-cell analysis + +### Manual Annotations +cellxgene offers exploratory visualizations that are critical for manual annotation workflows, especially in collaborative environments. +We plan to support manually annotating cells with labels (i.e., cell type or QC flags), and their easy export for downstream analysis. +See [issue #524](https://github.com/chanzuckerberg/cellxgene/issues/524) for more details. + +### Simple Click to Launch [Paused] + +Many biologists prefer not to interact with the command line and need an OS-native experience when using cellxgene. +We plan to implement a point-and-click installation and launch experience so that users can easily load data into cellxgene. +See [issue #687](https://github.com/chanzuckerberg/cellxgene/issues/687) for details. + +### Python API [Paused] +For computational biologists, saving h5ad files then loading them into cellxgene is a point of friction. +We plan to support importing cellxgene as a Python package so that users can launch cellxgene directly from an interactive environment (such as Jupyter, IPython, or Spyder), and pass data to and from the cellxgene UI. + +## Improving user support + +### Improved documentation +cellxgene has some specific expectations about how data is stored. +We want to ensure that new users can get started easily and learn how to use cellxgene with their own data. +We plan to improve documentation on getting started, installation, data, and contributing. +See [issue #533](https://github.com/chanzuckerberg/cellxgene/issues/533) for more details. diff --git a/docs/posts/troubleshooting.md b/docs/posts/troubleshooting.md new file mode 100644 index 00000000..e2483fa4 --- /dev/null +++ b/docs/posts/troubleshooting.md @@ -0,0 +1,22 @@ +--- +layout: default +title: Troubleshooting +description: Troubleshooting +--- +# Troubleshooting tips & tricks + +#### I tried to `pip install cellxgene` and got a weird error I don't understand + +This may happen, especially as we work out bugs in our installation process! Please create a new [Github issue](https://github.com/chanzuckerberg/cellxgene/issues), explain what you did, and include all the error messages you saw. It'd also be super helpful if you call `pip freeze` and include the full output alongside your issue. + +#### I have a BIG dataset - how can I make cellxgene run as fast as possible? + +If your dataset requires gigabytes of disk space, you may need to select an appropriate storage format in order to effectively utilize `cellxgene`. Tips and tricks: + +- `cellxgene` is optimized for columnar data access. For large datasets, format the expression matrix (`.X`) as either a [SciPy CSC sparse matrix](https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csc_matrix.html) or a dense Numpy array (whichever creates a smaller `h5ad` file). If you are using `cellxgene prepare`, include the `--sparse` flag to ensure `.X` is formatted as a CSC sparse matrix (by default, `.X` will be a dense matrix). +- `cellxgene` start time is directly proportional to `h5ad` file size and the speed of your file system. Expect that large (eg, million cell) datasets will take minutes to load, even on relatively fast computers with a high performance local hard drive. Once loaded, exploring metadata should still be quick. +- If your dataset size exceeds the size of memory (RAM) on the host computer, differential expression calculations will be extremely slow (or fail, if you run out of virtual memory). In this case, we recommend running with the `--disable-diffexp` flag. + +#### I'm following the developer instructions and get an error about "missing files and directories” when trying to build the client + +This is likely because you do not have node and npm installed, we recommend using [nvm](https://github.com/creationix/nvm) if you're new to using these tools.