mirror of
https://github.com/Novartis/cellxgene-gateway.git
synced 2026-10-11 07:20:56 +08:00
Initial Home page
Alokito
+110
@@ -0,0 +1,110 @@
|
|||||||
|
# Overview
|
||||||
|
|
||||||
|
The [Cellxgene project](https://github.com/chanzuckerberg/cellxgene) from the Chan Zuckberg Institute allows rich visualization of single cell RNA seq data. However, it is limited to visualizing a single dataset at a time. This repo contains Cellxgene Gateway, a small python/flask app that allows you to host an unlimited number of datasets on a single server. It dynamically launches instances of cellxgene gateway, and spins them down after a period of inactivity.
|
||||||
|
|
||||||
|
# Gateway Concept
|
||||||
|
|
||||||
|
The Gateway mediates between the incoming request, which always passes through a fixed domain name and port, and multiple cellxgene servers (one per dataset) that are either running in separate processes on a single server (currently implemented) or on an external docker container (potential improvement).
|
||||||
|
|
||||||
|
The role of the cellxgene gateway is
|
||||||
|
* translate incoming requests that mention the dns name and port of the ALB into requests for the cellxgene server running in the VPC.
|
||||||
|
* It must preserve the incoming accept header.
|
||||||
|
* translate outgoing responses that mention the cellxgene basepath into responses that mention the gateway base path.
|
||||||
|
* It must preserve the outgoing content-type header.
|
||||||
|
* It must translate the protocol for external resources to be consistent with the gateway basepath to avoid mixed content errors
|
||||||
|
|
||||||
|
# Class Structure of Gateway
|
||||||
|
|
||||||
|
# Spawn Process Implementation
|
||||||
|
|
||||||
|
The basic idea here is to write a [http://flask.pocoo.org/](Flask) app that receives all requests for cellxgene.server. It will fork a process running cellxgene for each dataset, and keep track of which processes are running by creating a file in `/tmp/cellxgene-instances`. Although this approach will not scale beyond a few concurrent datasets, it is easier to implement than the docker container approach and has significant overlap, so it is a reasonably first step.
|
||||||
|
|
||||||
|
## Tracking Instances
|
||||||
|
|
||||||
|
Instances are tracked in the pid_list array within the PidCache class. Each element of the array has the following structure:
|
||||||
|
|
||||||
|
{
|
||||||
|
dataset: '/username/hpc/P11Merged_noCC',
|
||||||
|
pid: 31245,
|
||||||
|
port: 8101,
|
||||||
|
last_access: <unix timestamp>
|
||||||
|
}
|
||||||
|
|
||||||
|
Where
|
||||||
|
|
||||||
|
* `dataset` is a path fragment identifying dataset. This is used to identify when we have an existing cellxgene process for a dataset.
|
||||||
|
* `pid` is the process id of the spawned process running cellxgene. This is used to kill the process when the dataset has been idle for too long.
|
||||||
|
* `port` is the port that the spawned process is listening on. This is used for the forwarded http requests.
|
||||||
|
* `last_access` is the last time a request for any resource was served for the dataset
|
||||||
|
|
||||||
|
The actual data files are located at cellxgene_efs + dataset.
|
||||||
|
|
||||||
|
## Execution Logic
|
||||||
|
|
||||||
|
1. If there is no path provided,
|
||||||
|
1. provide an index of the available datasets with links that include the path.
|
||||||
|
2. the index can be produced by iterating over all the directories/files similar to SPRING
|
||||||
|
2. Else verify the path provided corresponds to an existing dataset
|
||||||
|
1. Typical full url would be https://spring.server/username/hpc/P11Merged_noCC
|
||||||
|
2. If there is no matching dataset, return a "not found" message
|
||||||
|
3. Load all of the files from /tmp/cellxgene-instances
|
||||||
|
4. If there is no file with matching dataset
|
||||||
|
1. Grab lock for dataset with e.g. shared_lock = rwlock.SharedLock('/username/hpc/P11Merged_noCC')
|
||||||
|
2. select an unused port. We can simply start from 8100 and increment until we find an unused port.
|
||||||
|
3. Spawn cellxgene to display that dataset on the specified port
|
||||||
|
4. Write <pid>.txt
|
||||||
|
5. Release the dataset lock
|
||||||
|
5. Forward request to cellxgene, send response back to client
|
||||||
|
|
||||||
|
## Request Forwarding
|
||||||
|
|
||||||
|
### Browser Side
|
||||||
|
|
||||||
|
Here is the general form of requests that the browser will send to the flask app, which we have called "cellxgene gateway":
|
||||||
|
|
||||||
|
https://cellxgene.server/view/<dataset><subpath>
|
||||||
|
// example:
|
||||||
|
https://cellxgene.server/view/username/pbmc3k.h5ad/api/v0.2
|
||||||
|
|
||||||
|
> When running the gateway locally on port 5000, the browser side would be
|
||||||
|
> http://localhost:5000/view/username/pbmc3k.h5ad/api/v0.2
|
||||||
|
|
||||||
|
In both cases, we will call the part before the subpath (including the dataset) the "gateway basepath". Note two things
|
||||||
|
|
||||||
|
1. The gateway basepath NEVER ends in a slash.
|
||||||
|
2. The subpath ALWAYS begins with a slash.
|
||||||
|
3. The dataset will always match the path to a dataset on disk, relative to the dataset root directory.
|
||||||
|
|
||||||
|
### Flask Side
|
||||||
|
|
||||||
|
1. get the port from the dataset name, in the example 'username/pbmc3k.h5ad'
|
||||||
|
2. Make a request to the cellxgene server at http://localhost:<port><subpath>. We will call the part without the subpath the "cellxgene basepath"
|
||||||
|
1. Make sure to copy all request headers from the incoming "gateway request" to the outgoing "cellxgene request"
|
||||||
|
3. Convert the "cellxgene response" that comes back into a "gateway response" that can be sent to the browser
|
||||||
|
1. Replace all instances of the cellxgene basepath with the gateway basepath
|
||||||
|
2. copy the relevant response headers, including "Content-Type" from the cellxgene response to the flask response
|
||||||
|
4. return flask response to the browser
|
||||||
|
|
||||||
|
### Example Path Parts
|
||||||
|
|
||||||
|
| Part | Example |
|
||||||
|
| ------------- | ------------- |
|
||||||
|
| gateway url | https://cellxgene.server/view/username/pbmc3k.h5ad/api/v0.2 |
|
||||||
|
| gateway basepath | https://cellxgene.server/view/username/pbmc3k.h5ad |
|
||||||
|
| gateway host | cellxgene.server (localhost:5005 when running locally) |
|
||||||
|
| path | /username/pbmc3k.h5ad/api/v0.2 |
|
||||||
|
| dataset | username/pbmc3k.h5ad |
|
||||||
|
| subpath | /api/v0.2 |
|
||||||
|
| cellxgene basepath | http://localhost:8000 |
|
||||||
|
| cellxgene url | http://localhost:8000/api/v0.2 |
|
||||||
|
|
||||||
|
# Spawn Docker Containers Implementation
|
||||||
|
|
||||||
|
In theory, to support Docker we need the following changes:
|
||||||
|
|
||||||
|
* Instead of spawning a cellxgene process, we need to launch a docker container that can mount EFS and display the dataset
|
||||||
|
* The <pid>.txt files should become docker information files named after some container identifier provided by AWS
|
||||||
|
* The files should be stored on the cellxgene EFS (shared filesystem) instead of in /tmp.
|
||||||
|
|
||||||
|
I'll let you know how it goes in practice if we ever get to it 😄 .
|
||||||
|
|
||||||
Reference in New Issue
Block a user