mirror of
https://github.com/Novartis/cellxgene-gateway.git
synced 2026-10-02 02:48:11 +08:00
george-hall-ucl
81c8ce4219
This adds the flag `GATEWAY_ENABLE_GENE_SETS` to enable support for gene sets. To simplify implementation, activating this flag also activates `GATEWAY_ENABLE_ANNOTATIONS`. The gene sets are saved in a file that has the same name as the annotations `csv` but with `_gene_sets` appended to the file name (before the extension). This file is hidden in filecrawler, and the gene sets are loaded when the associated annotations file is loaded. If the annotations file is missing, then an Exception is raised. I have updated one unit test to make it expect `--disable-gene-sets-save` in the default case (i.e. if `GATEWAY_ENABLE_ANNOTATIONS = 0`). All units tests pass. I have updated the README to document `GATEWAY_ENABLE_GENE_SETS`.
200 lines
7.6 KiB
Markdown
200 lines
7.6 KiB
Markdown
# Overview
|
||
|
||
Cellxgene Gateway allows you to use the Cellxgene Server provided by the Chan Zuckerberg Institute (https://github.com/chanzuckerberg/cellxgene) with multiple datasets. It displays an index of available h5ad (anndata) files. When a user clicks on a file name, it launches a Cellxgene Server instance that loads that particular data file and once it is available proxies requests to that server.
|
||
|
||
[](https://codecov.io/gh/Novartis/cellxgene-gateway) [](https://pypi.org/project/cellxgene-gateway/) [](https://pypistats.org/packages/cellxgene-gateway)
|
||
|
||
# Running locally
|
||
|
||
## Prequisites
|
||
|
||
1. This project requires python 3.6 or higher. Please check your version with
|
||
|
||
```bash
|
||
$ python --version
|
||
```
|
||
|
||
2. It is also a good idea to set up a venv
|
||
|
||
```bash
|
||
python -m venv .cellxgene-gateway
|
||
source .cellxgene-gateway/bin/activate # type `deactivate` to deactivate the venv
|
||
```
|
||
|
||
## Install cellxgene-gateway
|
||
|
||
### Option 1: Pip Install from Github
|
||
|
||
```bash
|
||
pip install git+https://github.com/Novartis/cellxgene-gateway
|
||
```
|
||
Note: you may need to downgrade h5py with `pip install h5py==2.9.0` due to an [issue](https://github.com/theislab/scanpy/issues/832) in a dependency.
|
||
### Option 2: Install from PyPI
|
||
|
||
```bash
|
||
pip install cellxgene-gateway
|
||
```
|
||
|
||
## Running cellxgene gateway
|
||
|
||
1. Prepare a folder with .h5ad files, for example
|
||
|
||
```bash
|
||
mkdir ../cellxgene_data
|
||
wget https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/example-dataset/pbmc3k.h5ad -O ../cellxgene_data/pbmc3k.h5ad
|
||
```
|
||
|
||
|
||
2. Set your environment variables correctly:
|
||
|
||
```bash
|
||
export CELLXGENE_DATA=../cellxgene_data # change this directory if you put data in a different place.
|
||
export CELLXGENE_LOCATION=`which cellxgene`
|
||
```
|
||
|
||
3. Now, execute the cellxgene gateway:
|
||
|
||
```bash
|
||
cellxgene-gateway
|
||
```
|
||
|
||
Here's what the environment variables mean:
|
||
|
||
* `CELLXGENE_LOCATION` - the location of the cellxgene executable, e.g. `~/anaconda2/envs/cellxgene/bin/cellxgene`
|
||
|
||
At least one of the following is required:
|
||
* `CELLXGENE_DATA` - a directory that can contain subdirectories with `.h5ad` data files, *without* trailing slash, e.g. `/mnt/cellxgene_data`
|
||
* `CELLXGENE_BUCKET` - an s3 bucket that can contain keys with `.h5ad` data files, e.g. `my-cellxgene-data-bucket`
|
||
Cellxgene Gateway is designed to make it easy to add additional data sources, please see the source code for gateway.py and the ItemSource interface in items/item_source.py
|
||
|
||
Optional environment variables:
|
||
* `CELLXGENE_ARGS` - catch-all variable that can be used to pass additional command line args to cellxgene server
|
||
* `EXTERNAL_HOST` - the hostname and port from the perspective of the web browser, typically `localhost:5005` if running locally. Defaults to "localhost:{GATEWAY_PORT}"
|
||
* `EXTERNAL_PROTOCOL` - typically http when running locally, can be https when deployed if the gateway is behind a load balancer or reverse proxy that performs https termination. Default value "http"
|
||
* `GATEWAY_IP` - ip addess of instance gateway is running on, mostly used to display SSH instructions. Defaults to `socket.gethostbyname(socket.gethostname())`
|
||
* `GATEWAY_PORT` - local port that the gateway should bind to, defaults to 5005
|
||
* `GATEWAY_EXPIRE_SECONDS` - time in seconds that a cellxgene process will remain idle before being terminated. Defaults to 3600 (one hour)
|
||
* `GATEWAY_EXTRA_SCRIPTS` - JSON array of script paths, will be embedded into each page and forwarded with `--scripts` to cellxgene server
|
||
* `GATEWAY_ENABLE_ANNOTATIONS` - Set to `true` or to `1` to enable cellxgene annotations.
|
||
* `GATEWAY_ENABLE_GENE_SETS` - Set to `true` or to `1` to enable cellxgene gene sets. Also enables `GATEWAY_ENABLE_ANNOTATIONS`.
|
||
* `GATEWAY_ENABLE_BACKED_MODE` - Set to `true` or to `1` to load AnnData in file-backed mode. This saves memory and speeds up launch time but may reduce overall performance.
|
||
* `GATEWAY_LOG_LEVEL` - default is `INFO`. set to `DEBUG` to increase logging and to `WARNING` to decrease logging.
|
||
* `S3_ENABLE_LISTINGS_CACHE` - Set to `true` or to `1` to cache listings of S3 folders for performance. If the cache becomes stale, set `filecrawl.html?refresh=true` query parameter to refresh the cache.
|
||
|
||
If any of the following optional variables are set, [ProxyFix](https://werkzeug.palletsprojects.com/en/1.0.x/middleware/proxy_fix/) will be used.
|
||
* `PROXY_FIX_FOR` - Number of upstream proxies setting X-Forwarded-For
|
||
* `PROXY_FIX_PROTO` - Number of upstream proxies setting X-Forwarded-Proto
|
||
* `PROXY_FIX_HOST` - Number of upstream proxies setting X-Forwarded-Host
|
||
* `PROXY_FIX_PORT` - Number of upstream proxies setting X-Forwarded-Port
|
||
* `PROXY_FIX_PREFIX` - Number of upstream proxies setting X-Forwarded-Prefix
|
||
|
||
The defaults should be fine if you set up a venv and cellxgene_data folder as above.
|
||
|
||
## Running cellxgene-gateway with Docker
|
||
|
||
First, build Docker image:
|
||
|
||
```bash
|
||
docker build -t cellxgene-gateway .
|
||
```
|
||
|
||
Then, cellxgene-gateway can be launched as such:
|
||
|
||
```bash
|
||
docker run -it --rm \
|
||
-v <local_data_dir>:/cellxgene-data \
|
||
-p 5005:5005 \
|
||
cellxgene-gateway
|
||
```
|
||
|
||
Additional environment variables can be provided with the `-e` parameter:
|
||
|
||
```bash
|
||
docker run -it --rm \
|
||
-v <local_data_dir>:/cellxgene-data \
|
||
-e GATEWAY_PORT=8080 \
|
||
-p 8080:8080 \
|
||
cellxgene-gateway
|
||
```
|
||
|
||
# Customization
|
||
|
||
The current paradigm for customization is to modify files during a build or deployment phase:
|
||
|
||
* To modify CSS or JS on particular gateway pages, overwrite or append to the templates
|
||
* To add script tags such as for user analytics to all pages, set GATEWAY_EXTRA_SCRIPTS
|
||
* these scripts will also be run on the pages served by cellxgene server via the --scripts parameter
|
||
* See https://github.com/chanzuckerberg/cellxgene/pull/680 for details on --scripts parameter
|
||
|
||
Currently we use a bash script that copies the gateway to a "build" directory before modifying templates with sed and the like. There is probably a better way.
|
||
|
||
# Development
|
||
|
||
We’re actively developing. Please see the "future work" section of the [wiki](https://github.com/Novartis/cellxgene-gateway/wiki#future-work). If you’re interested in being a contributor please reach out to [@alokito](https://github.com/alokito).
|
||
|
||
## Developer Install
|
||
|
||
If you want to develop the code, you will need to clone the repo. Make sure you have the prequesite listed above, then:
|
||
|
||
1. Clone the repo
|
||
|
||
```bash
|
||
git clone https://github.com/Novartis/cellxgene-gateway.git
|
||
cd cellxgene-gateway
|
||
```
|
||
|
||
2. Install requirements with
|
||
|
||
```bash
|
||
pip install -r requirements.txt
|
||
```
|
||
|
||
3. Install the gateway in developer mode
|
||
|
||
```bash
|
||
python setup.py develop
|
||
```
|
||
|
||
For convenience, the code repo includes a `run.sh.example` shell script to run the gateway.
|
||
|
||
4. Install pre-commit hooks
|
||
|
||
```bash
|
||
conda install -c conda-forge pre-commit
|
||
pre-commit install
|
||
```
|
||
|
||
|
||
## Running Tests
|
||
|
||
[](https://travis-ci.org/Novartis/cellxgene-gateway)
|
||
|
||
```bash
|
||
python -m unittest discover tests
|
||
```
|
||
|
||
## Code Coverage
|
||
```bash
|
||
coverage run -m unittest discover tests
|
||
coverage html
|
||
```
|
||
|
||
## Running Linters
|
||
|
||
pip install isort flake8 black
|
||
|
||
```bash
|
||
isort -rc . # rc means recursive, and was deprecated in dev version of isort
|
||
black .
|
||
```
|
||
|
||
# Getting Help
|
||
|
||
If you need help for any reason, please make a github ticket. One of the contributors should help you out.
|
||
|
||
# Contributors
|
||
|
||
* Niket Patel - https://github.com/NiketPatel9
|
||
* Alok Saldanha - https://github.com/alokito
|
||
* Yohann Potier - https://github.com/ypotier
|