mirror of
https://github.com/chanzuckerberg/cellxgene.git
synced 2026-10-01 21:18:12 +08:00
Restv2 feature branch merge to master (#284)
Move to new REST v0.2 communication between front and back-end. This is a first cut implementation which is functional, but will need follow-up enhancements for performance, error checking, etc. Protocol spec is in docs directory. * Add filtering via indexing * Using new filter specs Indexing working * Added filtering by annotation value * factor out common methods * Documentation * create enum for axis (obs/var) * Better description for filter's return * Add boolean to enumerated types * Augmented enum for scanpy axis * Create schema for annotations Based on datatype within scanpy/anndata + tests * remove obsolete schema parse script * Update rest api to remove old routes and add schema route * Separate development requirements * Warning for unsupported datatypes * include -r requirements.txt in dev * Merged downcast warnings * Fixed bug where names were NaNs Needed to include the index too when creating the series * Add config endpoint * Generate app features from CLI selections * Move features to driver * Add tests for schema * Clearer version wording * python3 version of super * version from engine to package level * move features to driver * Revise layout function to match the new spec * GET for layout/obs * PUT Layout (#211) * PUT Layout * Csweaver/annotations (#212) * Update scanpy engine to support the rest v0.2 annotation requests * GET endpoint for obs annotations + tests * Documentation * Test annotations in scanpy engine * Description for annotation-keys param * annotation->annotations * clarified return for annotations * Use URL query list for annotations fields * parse_filter parses v0.2 GET filters (#215) * parse_filter parses v0.2 GET filters * Don't allow index filters from query params * Better variable conversion * Parse filter improvements - uses default dict - renamed filter -> query_filter * Cleanup Tasks (#216) * Add test_api back into travis build * Do custom JSON encoding the correct way * Run cellxgene server in test setup * Cleanup new tests too * Option to bind to all interfaces (#225) app.run("0.0.0.0") instead of app.run("127.0.0.1") binds to all interfaces. Note: There are comments on the internet that says that the flask server is not up to the task of production serving. I don't think that such scalability concerns apply here, but I was able to get cellxgene working with twistd relatively easily, and we could switch to that if there are scalability concerns. Test plan: browsed to <ip>:5005/api/v0.2/config on a different host. * Add filtering via indexing * Using new filter specs Indexing working * Added filtering by annotation value * factor out common methods * Documentation * create enum for axis (obs/var) * Better description for filter's return * Add boolean to enumerated types * Augmented enum for scanpy axis * Create schema for annotations Based on datatype within scanpy/anndata + tests * remove obsolete schema parse script * Update rest api to remove old routes and add schema route * Separate development requirements * Warning for unsupported datatypes * include -r requirements.txt in dev * Merged downcast warnings * Fixed bug where names were NaNs Needed to include the index too when creating the series * Add config endpoint * Generate app features from CLI selections * Move features to driver * Add tests for schema * Clearer version wording * python3 version of super * version from engine to package level * move features to driver * Revise layout function to match the new spec * GET for layout/obs * PUT Layout (#211) * PUT Layout * Csweaver/annotations (#212) * Update scanpy engine to support the rest v0.2 annotation requests * GET endpoint for obs annotations + tests * Documentation * Test annotations in scanpy engine * Description for annotation-keys param * annotation->annotations * clarified return for annotations * Use URL query list for annotations fields * parse_filter parses v0.2 GET filters (#215) * parse_filter parses v0.2 GET filters * Don't allow index filters from query params * Better variable conversion * Parse filter improvements - uses default dict - renamed filter -> query_filter * Cleanup Tasks (#216) * Add test_api back into travis build * Do custom JSON encoding the correct way * Run cellxgene server in test setup * Cleanup new tests too * Option to bind to all interfaces (#225) app.run("0.0.0.0") instead of app.run("127.0.0.1") binds to all interfaces. Note: There are comments on the internet that says that the flask server is not up to the task of production serving. I don't think that such scalability concerns apply here, but I was able to get cellxgene working with twistd relatively easily, and we could switch to that if there are scalability concerns. Test plan: browsed to <ip>:5005/api/v0.2/config on a different host. * Fix merge errors - import warnings was improperly deleted - scanpy engine tests were totally wrong * Fix merge error with driver * PUT /annotations (#235) * Add query param for annotation name * fix descriptions, eliminate else clause * first cut at initial data load on rest 0.2 api * Annotation var (#248) * Fix bug strings are always objects in pandas * Add axis to annotation method * Add /annotation/var to REST api * Csweaver/expressiondata (#242) * Refactor expression method for REST v2 * Add message to QueryStringError * Fix range filters * Add GET route for /data * /data PUT route * rename expression to data_frame * clarification of error * Improve accept type handling * support all schema types for 0.2 REST API * remove REST 0.1 code; connect var annotations loading * config reducer; use config to set data set title; remove obsolete templating code for data set title * REST 0.2 expression conversion support * partial port of expression to REST 0.2 * diffexp (#273) * Add diffexp method to scanpy and test * Minor tweaks to diffexp Get a minimal working version to unblock FE development * Fixing things git deleted * cleanup print statements * Add index test * additional, partial REST 0.2 bring up of diffexp * Ignore unstructured annotations for data (#275) This is a temp hack, need to figure out how to include data.uns if there is only one gene * diffexp REST 0.2 port finish * ignore unstructured annotaitons on all routes except layout * correctly use varDataCache; maintain state during world rebuild * correct varDataCache use * temporarily disable all memoization * refinements to expression data caching * clear cell sets upon regraph/reset * update version of REST to 0.2 * Travis build fixes - comment out cache import - fix duplicate test name * Remove dependency from travis * clarify semantics of config variables * move generic action helpers into util
This commit is contained in:
Bruce Martin
+12
-3
@@ -9,11 +9,13 @@ from flask_cors import CORS
|
||||
from flask_restful_swagger_2 import get_swagger_blueprint
|
||||
|
||||
from .rest_api.rest import get_api_resources
|
||||
from .util.utils import Float32JSONEncoder
|
||||
from .web import webapp
|
||||
|
||||
REACTIVE_LIMIT = 1_000_000
|
||||
|
||||
app = Flask(__name__, static_folder="web/static")
|
||||
app.json_encoder = Float32JSONEncoder
|
||||
cache = Cache(app, config={"CACHE_TYPE": "simple", "CACHE_DEFAULT_TIMEOUT": 860000})
|
||||
Compress(app)
|
||||
CORS(app)
|
||||
@@ -54,9 +56,12 @@ def run_scanpy(args):
|
||||
)
|
||||
|
||||
from .scanpy_engine.scanpy_engine import ScanpyEngine
|
||||
app.data = ScanpyEngine(args.data_directory, schema="data_schema.json",
|
||||
graph_method=args.layout, diffexp_method=args.diffexp)
|
||||
app.run(host="127.0.0.1", debug=True, port=args.port)
|
||||
app.data = ScanpyEngine(args.data_directory, layout_method=args.layout, diffexp_method=args.diffexp)
|
||||
if args.bind_all:
|
||||
host = "0.0.0.0"
|
||||
else:
|
||||
host = "127.0.0.1"
|
||||
app.run(host=host, debug=True, port=args.port)
|
||||
|
||||
|
||||
def main():
|
||||
@@ -64,6 +69,10 @@ def main():
|
||||
parser.add_argument("--title", "-t", help="Title to display -- if this is omitted the title will be the name "
|
||||
"of the directory from the data_directory arg")
|
||||
parser.add_argument("--port", help="Port to run server on.", type=int, default=5005)
|
||||
parser.add_argument(
|
||||
"--bind-all",
|
||||
help="Bind to all interfaces (this makes the server accessible beyond this computer)",
|
||||
action="store_true")
|
||||
subparsers = parser.add_subparsers(dest="cellxgene_command")
|
||||
try:
|
||||
from .scanpy_engine.scanpy_engine import ScanpyEngine
|
||||
|
||||
+51
-30
@@ -2,18 +2,38 @@ from abc import ABCMeta, abstractmethod
|
||||
|
||||
|
||||
class CXGDriver(metaclass=ABCMeta):
|
||||
def __init__(self, data, schema=None, graph_method=None, diffexp_method=None):
|
||||
|
||||
def __init__(self, data, layout_method=None, diffexp_method=None):
|
||||
self.data = self._load_data(data)
|
||||
self.layout_method = layout_method
|
||||
self.diffexp_method = diffexp_method
|
||||
self.cluster = None
|
||||
|
||||
@property
|
||||
def features(self):
|
||||
features = {
|
||||
"cluster": {"available": False},
|
||||
"layout": {
|
||||
"obs": {"available": False},
|
||||
"var": {"available": False},
|
||||
},
|
||||
"diffexp": {"available": False}
|
||||
}
|
||||
# TODO - Interactive limit should be generated from the actual available methods see GH issue #94
|
||||
if self.layout_method:
|
||||
# TODO handle "var" when gene layout becomes available
|
||||
features["layout"]["obs"] = {"available": True, "interactiveLimit": 15000}
|
||||
if self.diffexp_method:
|
||||
features["diffexp"] = {"available": True, "interactiveLimit": 5000}
|
||||
if self.cluster:
|
||||
features["cluster"] = {"available": True, "interactiveLimit": 45000}
|
||||
return features
|
||||
|
||||
@staticmethod
|
||||
@abstractmethod
|
||||
def _load_data(data):
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def _load_or_infer_schema(data):
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def cells(self):
|
||||
pass
|
||||
@@ -23,33 +43,32 @@ class CXGDriver(metaclass=ABCMeta):
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def filter_cells(self, filter):
|
||||
def filter_dataframe(self, filter):
|
||||
"""
|
||||
Filter cells from data and return a subset of the data
|
||||
A filter is a dictionary where the key is a metadatata category
|
||||
Value is dictionary
|
||||
value_type: int, float, string
|
||||
variable_type: continuous, categorical
|
||||
query: filter value, for categorical [val1, val2], for continuous {min: x, max:y}
|
||||
Filters are combined with the and operator
|
||||
:param filter:
|
||||
:return: filtered dataframe
|
||||
Filter cells from data and return a subset of the data. They can operate on both obs and var dimension with
|
||||
indexing and filtering by annotation value. Filters are combined with the and operator.
|
||||
See REST specs for info on filter format:
|
||||
https://docs.google.com/document/d/1Fxjp1SKtCk7l8QP9-7KAjGXL0eldi_qEnNT0NmlGzXI/edit#heading=h.8qc9q57amldx
|
||||
|
||||
:param filter: dictionary with filter parames
|
||||
:return: View into scanpy object with cells/genes filtered
|
||||
"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def metadata(self, df, fields=None):
|
||||
def annotation(self, df, axis, fields=None):
|
||||
"""
|
||||
Gets metadata key:value for each cells
|
||||
Gets annotation value for each observation
|
||||
|
||||
:param axis:
|
||||
:param df: from filter_cells, dataframe
|
||||
:param fields: list of keys for metadata to return, returns all metadata values if not set.
|
||||
:return: list of metadata values
|
||||
:param fields: list of keys for annotation to return, returns all annotation values if not set.
|
||||
:return: dict: names - list of fields in order, data - list of lists or metadata [idx, val1, val2...]
|
||||
"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def create_graph(self, df):
|
||||
def layout(self, df):
|
||||
"""
|
||||
Computes a n-d layout for cells through dimensionality reduction.
|
||||
:param df: from filter_cells, dataframe
|
||||
@@ -58,24 +77,26 @@ class CXGDriver(metaclass=ABCMeta):
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def diffexp(self, df1, df2):
|
||||
def diffexp(self, df1, df2, genes):
|
||||
"""
|
||||
Computes the top differentially expressed genes between two clusters
|
||||
:param df1: from filter_cells, dataframe containing first set of cells
|
||||
:param df2: from filter_cells, dataframe containing second set of cells
|
||||
:return: top genes, stats and expression values for top genes
|
||||
Computes the top differentially expressed variables between two observation sets. If dataframes
|
||||
contain a subset of variables, then statistics for all variables will be returned, otherwise
|
||||
only the top N vars will be returned.
|
||||
:param df1: from filter_cells, dataframe containing first set of observations
|
||||
:param df2: from filter_cells, dataframe containing second set of observations
|
||||
:param topN: Limit results to top N (Top var mode only)
|
||||
:return: top genes, stats and expression values for variables
|
||||
"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def expression(self, df):
|
||||
def data_frame(self, df):
|
||||
"""
|
||||
Retrieves expression for each gene for cells in data frame
|
||||
:param df:
|
||||
:param df: from filter_cells, dataframe
|
||||
:return: {
|
||||
"genes": list of genes,
|
||||
"cells": list of cells and expression list,
|
||||
"nonzero_gene_count": number of nonzero genes
|
||||
"var": list of variable ids,
|
||||
"obs": [cellid, var1 expression, var2 expression, ...],
|
||||
}
|
||||
"""
|
||||
pass
|
||||
|
||||
+492
-388
@@ -1,435 +1,539 @@
|
||||
from http import HTTPStatus
|
||||
import pkg_resources
|
||||
|
||||
from flask import (
|
||||
Blueprint, request, current_app
|
||||
Blueprint, current_app, jsonify, make_response, request
|
||||
)
|
||||
from flask_restful_swagger_2 import Api, swagger, Resource
|
||||
from werkzeug.datastructures import ImmutableMultiDict
|
||||
|
||||
from server.app.util.utils import make_payload
|
||||
from server.app.util.filter import parse_filter
|
||||
from server.app.util.constants import Axis, DiffExpMode
|
||||
from server.app.util.filter import parse_filter, QueryStringError
|
||||
from server.app.util.models import FilterModel
|
||||
|
||||
|
||||
class InitializeAPI(Resource):
|
||||
class SchemaAPI(Resource):
|
||||
@swagger.doc({
|
||||
"summary": "get metadata schema, ranges for values, and cell count to initialize cellxgene app",
|
||||
"summary": "get schema for dataframe and annotations",
|
||||
"tags": ["initialize"],
|
||||
"parameters": [],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "initialization data for UI",
|
||||
"description": "schema",
|
||||
"examples": {
|
||||
"application/json": {
|
||||
"data": {
|
||||
"cellcount": 3589,
|
||||
"options": {
|
||||
"Sample.type": {
|
||||
"options": {
|
||||
"Glioblastoma": 3589
|
||||
}
|
||||
},
|
||||
"Selection": {
|
||||
"options": {
|
||||
"Astrocytes(HEPACAM)": 714,
|
||||
"Endothelial(BSC)": 123,
|
||||
"Microglia(CD45)": 1108,
|
||||
"Neurons(Thy1)": 685,
|
||||
"Oligodendrocytes(GC)": 294,
|
||||
"Unpanned": 665
|
||||
}
|
||||
},
|
||||
"Splice_sites_AT.AC": {
|
||||
"range": {
|
||||
"max": 1025,
|
||||
"min": 152
|
||||
}
|
||||
},
|
||||
"Splice_sites_Annotated": {
|
||||
"range": {
|
||||
"max": 1075869,
|
||||
"min": 26
|
||||
}
|
||||
}
|
||||
"schema": {
|
||||
"dataframe": {
|
||||
"nObs": 383,
|
||||
"nVar": 19944,
|
||||
"type": "float32"
|
||||
},
|
||||
"schema": {
|
||||
"CellName": {
|
||||
"displayname": "Name",
|
||||
"type": "string",
|
||||
"variabletype": "categorical"
|
||||
},
|
||||
"Class": {
|
||||
"displayname": "Class",
|
||||
"type": "string",
|
||||
"variabletype": "categorical"
|
||||
},
|
||||
"ERCC_reads": {
|
||||
"displayname": "ERCC Reads",
|
||||
"type": "int",
|
||||
"variabletype": "continuous"
|
||||
},
|
||||
"ERCC_to_non_ERCC": {
|
||||
"displayname": "ERCC:Non-ERCC",
|
||||
"type": "float",
|
||||
"variabletype": "continuous"
|
||||
},
|
||||
"Genes_detected": {
|
||||
"displayname": "Genes Detected",
|
||||
"type": "int",
|
||||
"variabletype": "continuous"
|
||||
}
|
||||
},
|
||||
"genes": ["1/2-SBSRNA4", "A1BG", "A1BG-AS1"]
|
||||
|
||||
},
|
||||
"status": {
|
||||
"error": False,
|
||||
"errormessage": ""
|
||||
"annotations": {
|
||||
"obs": [
|
||||
{"name": "name", "type": "string"},
|
||||
{"name": "tissue_type", "type": "string"},
|
||||
{"name": "num_reads", "type": "int32"},
|
||||
{"name": "sample_name", "type": "string"},
|
||||
{
|
||||
"name": "clusters",
|
||||
"type": "categorical",
|
||||
"categories": [99, 1, "unknown cluster"]
|
||||
},
|
||||
{"name": "QScore", "type": "float32"}
|
||||
],
|
||||
"var": [
|
||||
{"name": "name", "type": "string"},
|
||||
{"name": "gene", "type": "string"}
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
})
|
||||
def get(self):
|
||||
from server.app.app import REACTIVE_LIMIT
|
||||
return make_payload({
|
||||
"schema": current_app.data.schema,
|
||||
"cellcount": current_app.data.cell_count,
|
||||
"reactivelimit": REACTIVE_LIMIT,
|
||||
"genes": current_app.data.genes(),
|
||||
"ranges": current_app.data.metadata_ranges(),
|
||||
|
||||
})
|
||||
return make_response(jsonify({"schema": current_app.data.schema}), 200)
|
||||
|
||||
|
||||
class CellsAPI(Resource):
|
||||
class ConfigAPI(Resource):
|
||||
@swagger.doc({
|
||||
"summary": "filter based on metadata fields to get a subset cells, expression data, and metadata",
|
||||
"tags": ["cells"],
|
||||
"description": "Cells takes query parameters defined in the schema retrieved from the /initialize enpoint. "
|
||||
"<br>For categorical metadata keys filter based on `key=value` <br>"
|
||||
" For continuous metadata keys filter by `key=min,max`<br> Either value "
|
||||
"can be replaced by a \*. To have only a minimum value `key=min,\*` To have only a maximum "
|
||||
"value `key=\*,max` <br>Graph data (if retrieved) is normalized"
|
||||
" To only retrieve cells that don't have a value for the key filter by `key`",
|
||||
"summary": "Configuration information to assist in front-end adaptation"
|
||||
" to underlying engine, available functionality, interactive time limits, etc",
|
||||
"tags": ["initialize"],
|
||||
"parameters": [],
|
||||
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "initialization data for UI",
|
||||
"description": "schema",
|
||||
"examples": {
|
||||
"application/json": {
|
||||
"data": {
|
||||
"badmetadatacount": 0,
|
||||
"cellcount": 0,
|
||||
"cellids": ["..."],
|
||||
"metadata": [
|
||||
"config": {
|
||||
"features": [
|
||||
{"method": "POST", "path": "/cluster/", "available": False},
|
||||
{
|
||||
"CellName": "1001000173.G8",
|
||||
"Class": "Neoplastic",
|
||||
"Cluster_2d": "11",
|
||||
"Cluster_2d_color": "#8C564B",
|
||||
"Cluster_CNV": "1",
|
||||
"Cluster_CNV_color": "#1F77B4",
|
||||
"ERCC_reads": "152104",
|
||||
"ERCC_to_non_ERCC": "0.562454470489481",
|
||||
"Genes_detected": "1962",
|
||||
"Location": "Tumor",
|
||||
"Location.color": "#FF7F0E",
|
||||
"Multimapping_reads_percent": "2.67",
|
||||
"Neoplastic": "Neoplastic",
|
||||
"Non_ERCC_reads": "270429",
|
||||
"Sample.name": "BT_S2",
|
||||
"Sample.name.color": "#AEC7E8",
|
||||
"Sample.type": "Glioblastoma",
|
||||
"Sample.type.color": "#1F77B4",
|
||||
"Selection": "Unpanned",
|
||||
"Selection.color": "#98DF8A",
|
||||
"Splice_sites_AT.AC": "102",
|
||||
"Splice_sites_Annotated": "122397",
|
||||
"Splice_sites_GC.AG": "761",
|
||||
"Splice_sites_GT.AG": "125741",
|
||||
"Splice_sites_non_canonical": "56",
|
||||
"Splice_sites_total": "126660",
|
||||
"Total_reads": "1741039",
|
||||
"Unique_reads": "1400382",
|
||||
"Unique_reads_percent": "80.43",
|
||||
"Unmapped_mismatch": "2.15",
|
||||
"Unmapped_other": "0.18",
|
||||
"Unmapped_short": "14.56",
|
||||
"housekeeping_cluster": "2",
|
||||
"housekeeping_cluster_color": "#AEC7E8",
|
||||
"recluster_myeloid": "NA",
|
||||
"recluster_myeloid_color": "NA"
|
||||
"method": "POST",
|
||||
"path": "/layout/obs",
|
||||
"available": True,
|
||||
"interactiveLimit": 10000
|
||||
},
|
||||
],
|
||||
"reactive": True,
|
||||
"graph": [
|
||||
[
|
||||
"1001000173.G8",
|
||||
0.93836,
|
||||
0.28623
|
||||
],
|
||||
|
||||
[
|
||||
"1001000173.D4",
|
||||
0.1662,
|
||||
0.79438
|
||||
]
|
||||
{"method": "POST", "path": "/layout/var", "available": False}
|
||||
|
||||
],
|
||||
"status": {
|
||||
"error": False,
|
||||
"errormessage": ""
|
||||
}
|
||||
|
||||
},
|
||||
}
|
||||
},
|
||||
},
|
||||
|
||||
"400": {
|
||||
"description": "bad query params",
|
||||
}
|
||||
}
|
||||
})
|
||||
def get(self):
|
||||
payload = {
|
||||
"metadata": [],
|
||||
"cellcount": 0,
|
||||
"graph": [],
|
||||
"ranges": {},
|
||||
}
|
||||
# get query params
|
||||
cells_filter = parse_filter(request.args, current_app.data.schema)
|
||||
filtered_data = current_app.data.filter_cells(cells_filter)
|
||||
payload["metadata"] = current_app.data.metadata(filtered_data)
|
||||
payload["ranges"] = current_app.data.metadata_ranges(filtered_data)
|
||||
payload["graph"] = current_app.data.create_graph(filtered_data)
|
||||
payload["cellcount"] = current_app.data.cell_count
|
||||
return make_payload(payload)
|
||||
|
||||
|
||||
class ExpressionAPI(Resource):
|
||||
@swagger.doc({
|
||||
"summary": "Json with gene list and expression data by cell, limited to first 40 cells",
|
||||
"tags": ["expression"],
|
||||
"parameters": [
|
||||
{
|
||||
"name": "include_unexpressed_genes",
|
||||
"description": "Include genes that have 0 expression across all cells in set",
|
||||
"in": "path",
|
||||
"type": "bool",
|
||||
}
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "Json for heatmap",
|
||||
"examples": {
|
||||
"application/json": {
|
||||
"data": {
|
||||
"cells": [
|
||||
{
|
||||
"cellname": "1/2-SBSRNA4",
|
||||
"e": [0, 0, 214, 0, 0]
|
||||
},
|
||||
],
|
||||
"genes": [
|
||||
"1001000173.G8",
|
||||
"1001000173.D4",
|
||||
"1001000173.B4",
|
||||
"1001000173.A2",
|
||||
"1001000173.E2"
|
||||
],
|
||||
"nonzero_gene_count": 2857
|
||||
},
|
||||
"status": {
|
||||
"error": False,
|
||||
"errormessage": ""
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
def get(self):
|
||||
expression_data = current_app.data.expression()
|
||||
return make_payload(expression_data)
|
||||
|
||||
@swagger.doc({
|
||||
"summary": "Json with gene list and expression data by cell",
|
||||
"tags": ["expression"],
|
||||
"parameters": [
|
||||
{
|
||||
"name": "body",
|
||||
"in": "body",
|
||||
"schema": {
|
||||
"example": {
|
||||
"celllist": ["1001000173.G8", "1001000173.D4"],
|
||||
"genelist": ["1/2-SBSRNA4", "A1BG", "A1BG-AS1", "A1CF", "A2LD1", "A2M", "A2ML1", "A2MP1",
|
||||
"A4GALT"],
|
||||
"include_unexpressed_genes": True,
|
||||
}
|
||||
|
||||
}
|
||||
},
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "Json for expressiondata",
|
||||
"examples": {
|
||||
"application/json": {
|
||||
"data": {
|
||||
"cells": [
|
||||
{
|
||||
"cellname": "1001000173.D4",
|
||||
"e": [0, 0]
|
||||
},
|
||||
{
|
||||
"cellname": "1001000173.G8",
|
||||
"e": [0, 0]
|
||||
}
|
||||
],
|
||||
"genes": [
|
||||
"ABCD4",
|
||||
"ZWINT"
|
||||
],
|
||||
"nonzero_gene_count": 2857
|
||||
},
|
||||
"status": {
|
||||
"error": False,
|
||||
"errormessage": ""
|
||||
}
|
||||
|
||||
}
|
||||
}
|
||||
},
|
||||
"400": {
|
||||
"description": "Required parameter missing/incorrect",
|
||||
}
|
||||
}
|
||||
})
|
||||
def post(self):
|
||||
args = request.get_json()
|
||||
cell_list = args.get("celllist", [])
|
||||
gene_list = args.get("genelist", [])
|
||||
if not cell_list and not gene_list:
|
||||
return make_payload([], "must include celllist and/or genelist parameter", 400)
|
||||
|
||||
expression_data = current_app.data.expression(cell_list, gene_list)
|
||||
|
||||
if cell_list and len(expression_data["cells"]) < len(cell_list):
|
||||
return make_payload([], "Some cell ids not available", 400)
|
||||
if gene_list and len(expression_data["genes"]) < len(gene_list):
|
||||
return make_payload([], "Some genes not available", 400)
|
||||
|
||||
return make_payload(expression_data)
|
||||
|
||||
|
||||
class DifferentialExpressionAPI(Resource):
|
||||
@swagger.doc({
|
||||
"summary": "Get the top expressed genes for two cell sets. Calculated using t-test",
|
||||
"tags": ["expression"],
|
||||
"parameters": [
|
||||
{
|
||||
"name": "body",
|
||||
"in": "body",
|
||||
"schema": {
|
||||
"example": {
|
||||
"celllist1": ["1001000176.C12", "1001000176.C7", "1001000177.F11"],
|
||||
"celllist2": ["1001000012.D2", "1001000017.F10", "1001000033.C3", "1001000229.D4"],
|
||||
"num_genes": 5,
|
||||
"pval": 0.000001,
|
||||
},
|
||||
}
|
||||
}
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "top expressed genes for cellset1, cellset2",
|
||||
"examples": {
|
||||
"application/json": {
|
||||
"data": {
|
||||
"celllist1": {
|
||||
"ave_diff": [
|
||||
432.0132935431362,
|
||||
12470.5623982637,
|
||||
957.0246880086814
|
||||
],
|
||||
"mean_expression_cellset1": [
|
||||
438.6185567010309,
|
||||
13315.536082474227,
|
||||
1076.5773195876288
|
||||
],
|
||||
"mean_expression_cellset2": [
|
||||
6.605263157894737,
|
||||
844.9736842105264,
|
||||
119.55263157894737
|
||||
],
|
||||
"pval": [
|
||||
3.8906598089944563e-35,
|
||||
1.9086226376018916e-25,
|
||||
7.847480544069826e-21
|
||||
],
|
||||
"topgenes": [
|
||||
"TMSB10",
|
||||
"FTL",
|
||||
"TMSB4X"
|
||||
]
|
||||
"displayNames": {
|
||||
"engine": "ScanPy version 1.33",
|
||||
"dataset": "/home/joe/mouse/blorth.csv"
|
||||
},
|
||||
"celllist2": {
|
||||
"ave_diff": [
|
||||
-6860.599158979924,
|
||||
-519.1314432989691,
|
||||
-10278.328269126423
|
||||
],
|
||||
"mean_expression_cellset1": [
|
||||
2.8350515463917527,
|
||||
0.6185567010309279,
|
||||
23.09278350515464
|
||||
],
|
||||
"mean_expression_cellset2": [
|
||||
6863.434210526316,
|
||||
519.75,
|
||||
10301.421052631578
|
||||
],
|
||||
"pval": [
|
||||
4.662891833748732e-44,
|
||||
3.6278087029927103e-37,
|
||||
8.396825170618402e-35
|
||||
],
|
||||
"topgenes": [
|
||||
"SPARCL1",
|
||||
"C1orf61",
|
||||
"CLU"
|
||||
]
|
||||
}
|
||||
},
|
||||
"status": {
|
||||
"error": False,
|
||||
"errormessage": ""
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
def get(self):
|
||||
config = {
|
||||
"config": {
|
||||
"features": [
|
||||
{"method": "POST", "path": "/cluster/", **current_app.data.features["cluster"]},
|
||||
{"method": "POST", "path": "/layout/obs", **current_app.data.features["layout"]["obs"]},
|
||||
{"method": "POST", "path": "/layout/var", **current_app.data.features["layout"]["var"]},
|
||||
{"method": "POST", "path": "/diffexp/", **current_app.data.features["diffexp"]},
|
||||
],
|
||||
"displayNames": {
|
||||
"engine": f"cellxgene Scanpy engine version {pkg_resources.get_distribution('cellxgene').version}",
|
||||
"dataset": current_app.config["DATASET_TITLE"]
|
||||
}
|
||||
}
|
||||
}
|
||||
return make_response(jsonify(config), 200)
|
||||
|
||||
|
||||
class LayoutObsAPI(Resource):
|
||||
@swagger.doc({
|
||||
"summary": "Get the default layout for all observations.",
|
||||
"tags": ["layout"],
|
||||
"parameters": [],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "layout",
|
||||
"examples": {
|
||||
"application/json": {
|
||||
"layout": {
|
||||
"ndims": 2,
|
||||
"coordinates": [
|
||||
[0, 0.284483, 0.983744],
|
||||
[1, 0.038844, 0.739444]
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
def get(self):
|
||||
return make_response((jsonify({"layout": current_app.data.layout(current_app.data.data)})))
|
||||
|
||||
@swagger.doc({
|
||||
"summary": "Observation layout for filtered subset.",
|
||||
"tags": ["layout"],
|
||||
"parameters": [
|
||||
{
|
||||
"name": "filter",
|
||||
"description": "Complex Filter",
|
||||
"in": "body",
|
||||
"schema": FilterModel
|
||||
}
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "layout",
|
||||
"examples": {
|
||||
"application/json": {
|
||||
"layout": {
|
||||
"ndims": 2,
|
||||
"coordinates": [
|
||||
[0, 0.284483, 0.983744],
|
||||
[1, 0.038844, 0.739444]
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
def put(self):
|
||||
df = current_app.data.filter_dataframe(request.get_json()["filter"])
|
||||
return make_response((jsonify({"layout": current_app.data.layout(df)})))
|
||||
|
||||
|
||||
class AnnotationsObsAPI(Resource):
|
||||
@swagger.doc({
|
||||
"summary": "Fetch annotations (metadata) for all observations.",
|
||||
"tags": ["annotations"],
|
||||
"parameters": [{
|
||||
"in": "query",
|
||||
"name": "annotation-name",
|
||||
"type": "string",
|
||||
"description": "list of 1 or more annotation names"
|
||||
}],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "annotations",
|
||||
"examples": {
|
||||
"application/json": {
|
||||
"names": [
|
||||
"tissue_type", "sex", "num_reads", "clusters"
|
||||
],
|
||||
"data": [
|
||||
[0, "lung", "F", 39844, 99],
|
||||
[1, "heart", "M", 83, 1],
|
||||
[49, "spleen", None, 2, "unknown cluster"],
|
||||
|
||||
]
|
||||
}
|
||||
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
def get(self):
|
||||
fields = request.args.getlist("annotation-name", None)
|
||||
try:
|
||||
annotation_response = current_app.data.annotation(current_app.data.data, "obs", fields)
|
||||
except KeyError:
|
||||
return make_response(f"Error bad key in {fields}", 404)
|
||||
return make_response(jsonify(annotation_response))
|
||||
|
||||
@swagger.doc({
|
||||
"summary": "Fetch annotations (metadata) for filtered subset of observations.",
|
||||
"tags": ["annotations"],
|
||||
"parameters": [
|
||||
{
|
||||
"in": "query",
|
||||
"name": "annotation-name",
|
||||
"type": "string",
|
||||
"description": "list of 1 or more annotation names"
|
||||
},
|
||||
{
|
||||
"name": "filter",
|
||||
"description": "Complex Filter",
|
||||
"in": "body",
|
||||
"schema": FilterModel
|
||||
}
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "annotations",
|
||||
"examples": {
|
||||
"application/json": {
|
||||
"names": [
|
||||
"tissue_type", "sex", "num_reads", "clusters"
|
||||
],
|
||||
"data": [
|
||||
[0, "lung", "F", 39844, 99],
|
||||
[1, "heart", "M", 83, 1],
|
||||
[49, "spleen", None, 2, "unknown cluster"],
|
||||
|
||||
]
|
||||
}
|
||||
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
def put(self):
|
||||
fields = request.args.getlist("annotation-name", None)
|
||||
df = current_app.data.filter_dataframe(request.get_json()["filter"], include_uns=False)
|
||||
try:
|
||||
annotation_response = current_app.data.annotation(df, "obs", fields)
|
||||
except KeyError:
|
||||
return make_response(f"Error bad key in {fields}", 404)
|
||||
return make_response(jsonify(annotation_response))
|
||||
|
||||
|
||||
class AnnotationsVarAPI(Resource):
|
||||
@swagger.doc({
|
||||
"summary": "Fetch annotations (metadata) for all variables.",
|
||||
"tags": ["annotations"],
|
||||
"parameters": [{
|
||||
"in": "query",
|
||||
"name": "annotation-name",
|
||||
"type": "string",
|
||||
"description": "list of 1 or more annotation names"
|
||||
}],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "annotations",
|
||||
"examples": {
|
||||
"application/json": {
|
||||
"names": [
|
||||
"name", "category"
|
||||
],
|
||||
"data": [
|
||||
[0, "ATAD3C", 1],
|
||||
[1, "RER1", None],
|
||||
[49, "S100B", 6]
|
||||
]
|
||||
}
|
||||
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
def get(self):
|
||||
fields = request.args.getlist("annotation-name", None)
|
||||
try:
|
||||
annotation_response = current_app.data.annotation(current_app.data.data, "var", fields)
|
||||
except KeyError:
|
||||
return make_response(f"Error bad key in {fields}", 404)
|
||||
return make_response(jsonify(annotation_response))
|
||||
|
||||
@swagger.doc({
|
||||
"summary": "Fetch annotations (metadata) for filtered subset of variables.",
|
||||
"tags": ["annotations"],
|
||||
"parameters": [
|
||||
{
|
||||
"in": "query",
|
||||
"name": "annotation-name",
|
||||
"type": "string",
|
||||
"description": "list of 1 or more annotation names"
|
||||
},
|
||||
{
|
||||
"name": "filter",
|
||||
"description": "Complex Filter",
|
||||
"in": "body",
|
||||
"schema": FilterModel
|
||||
}
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "annotations",
|
||||
"examples": {
|
||||
"application/json": {
|
||||
"names": [
|
||||
"name", "category"
|
||||
],
|
||||
"data": [
|
||||
[0, "ATAD3C", 1],
|
||||
[1, "RER1", None],
|
||||
[49, "S100B", 6]
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
def put(self):
|
||||
fields = request.args.getlist("annotation-name", None)
|
||||
df = current_app.data.filter_dataframe(request.get_json()["filter"], include_uns=False)
|
||||
try:
|
||||
annotation_response = current_app.data.annotation(df, "var", fields)
|
||||
except KeyError:
|
||||
return make_response(f"Error bad key in {fields}", 404)
|
||||
return make_response(jsonify(annotation_response))
|
||||
|
||||
|
||||
class DiffExpObsAPI(Resource):
|
||||
@swagger.doc({
|
||||
"summary": "Generate differential expression (DE) statistics for two specified subsets of data, "
|
||||
"as indicated by the two provided observation complex filters",
|
||||
"tags": ["diffexp"],
|
||||
# TODO sort out params
|
||||
# "parameters": [
|
||||
# # {
|
||||
# # "in": "body",
|
||||
# # "name": "mode",
|
||||
# # "type": "string",
|
||||
# # "required": True,
|
||||
# # "description": "topN or varFilter"
|
||||
# # },
|
||||
# {
|
||||
# "in": "query",
|
||||
# "name": "count",
|
||||
# "type": "int32",
|
||||
# "description": "TopN mode: how many vars to return"
|
||||
# },
|
||||
# {
|
||||
# "in": "body",
|
||||
# "name": "varFilter",
|
||||
# "schema": FilterModel,
|
||||
# "description": "varFilter: Complex filter, only var for which vars to return"
|
||||
# },
|
||||
# {
|
||||
# "in": "body",
|
||||
# "name": "set1",
|
||||
# "schema": FilterModel,
|
||||
# "required": True,
|
||||
# "description": "Complex filter, only obs - observations in set1"
|
||||
# },
|
||||
# {
|
||||
# "in": "body",
|
||||
# "name": "set2",
|
||||
# "schema": FilterModel,
|
||||
# "description": "Complex filter, only obs - observations in set2. If not included, inverse of set1."
|
||||
# },
|
||||
# ],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "Statistics are encoded as an array of arrays, with fields ordered as: "
|
||||
"varIndex, avgDiff, pVal, pValAdj, set1AvgExp, set2AvgExp",
|
||||
"examples": {
|
||||
"application/json": [
|
||||
[328, -2.569489, 2.655706e-63, 3.642036e-57, 383.393, 583.9],
|
||||
[1250, -2.569489, 2.655706e-63, 3.642036e-57, 383.393, 583.9],
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
def post(self):
|
||||
args = request.get_json()
|
||||
cell_list_1 = args.get("celllist1", [])
|
||||
cell_list_2 = args.get("celllist2", [])
|
||||
num_genes = args.get("num_genes", 7)
|
||||
pval = args.get("pval", 0.5)
|
||||
if not (cell_list_1 and cell_list_2):
|
||||
return make_payload([],
|
||||
"must include celllist1 and celllist2 parameters",
|
||||
400)
|
||||
data = current_app.data.diffexp(cell_list_1, cell_list_2, pval, num_genes)
|
||||
return make_payload(data)
|
||||
# confirm mode is present and legal
|
||||
try:
|
||||
mode = DiffExpMode(args["mode"])
|
||||
except KeyError:
|
||||
return make_response("Error: mode is required", 400)
|
||||
except ValueError:
|
||||
return make_response(f"Error: invalid mode option {args['mode']}", 400)
|
||||
# Validate filters
|
||||
if mode == DiffExpMode.VAR_FILTER:
|
||||
if "varFilter" not in args:
|
||||
return make_response("varFilter is required when mode is set to varFilter ", 400)
|
||||
if Axis.OBS in args["varFilter"]["filter"]:
|
||||
return make_response("Obs filter not allowed in varFilter", 400)
|
||||
if "set1" not in args:
|
||||
return make_response("set1 is required.", 400)
|
||||
if Axis.VAR in args["set1"]["filter"]:
|
||||
return make_response("Var filter not allowed for set1", 400)
|
||||
# set2
|
||||
if "set2" not in args:
|
||||
return make_response("Set2 as inverse of set1 is not implemented", 501)
|
||||
if Axis.VAR in args["set2"]["filter"]:
|
||||
return make_response("Var filter not allowed for set2", 400)
|
||||
set1_filter = args["set1"]["filter"]
|
||||
set2_filter = args.get("set2", {"filter": {}})["filter"]
|
||||
if "varFilter" in args:
|
||||
set1_filter[Axis.VAR] = args["varFilter"]["filter"][Axis.VAR]
|
||||
set2_filter[Axis.VAR] = args["varFilter"]["filter"][Axis.VAR]
|
||||
df1 = current_app.data.filter_dataframe(set1_filter, include_uns=False)
|
||||
# TODO inverse
|
||||
df2 = current_app.data.filter_dataframe(set2_filter, include_uns=False)
|
||||
# exceeds size limit
|
||||
if df1.shape[0] + df2.shape[0] > current_app.data.features["diffexp"]["interactiveLimit"]:
|
||||
return make_response("Non-interactive request", 403)
|
||||
# mode
|
||||
count = args.get("count", None)
|
||||
try:
|
||||
diffexp = current_app.data.diffexp(df1, df2, count)
|
||||
except ValueError as ve:
|
||||
return make_response(ve.message, 400)
|
||||
return make_response(jsonify(diffexp))
|
||||
|
||||
|
||||
class DataObsAPI(Resource):
|
||||
@swagger.doc({
|
||||
"summary": "Get data (expression values) from the dataframe.",
|
||||
"tags": ["data"],
|
||||
"parameters": [
|
||||
{
|
||||
"in": "query",
|
||||
"name": "filter",
|
||||
"type": "string",
|
||||
"description": "axis:key:value"
|
||||
},
|
||||
{
|
||||
"in": "query",
|
||||
"name": "accept-type",
|
||||
"type": "string",
|
||||
"description": "MIME type"
|
||||
},
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "expression",
|
||||
"examples": {
|
||||
"application/json": {
|
||||
"var": [0, 20000],
|
||||
"obs": [
|
||||
[1, 39483, 3902, 203, 0, 0, 28]
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"400": {
|
||||
"description": "Malformed filter"
|
||||
},
|
||||
"406": {
|
||||
"description": "Unacceptable MIME type"
|
||||
},
|
||||
}
|
||||
})
|
||||
def get(self):
|
||||
# request.args is immutable
|
||||
args = dict(request.args)
|
||||
accept_type = args.pop("accept-type", None)
|
||||
try:
|
||||
filter_ = parse_filter(ImmutableMultiDict(args), current_app.data.schema['annotations'])
|
||||
except QueryStringError as e:
|
||||
return make_response(e.message, HTTPStatus.BAD_REQUEST)
|
||||
df = current_app.data.filter_dataframe(filter_, include_uns=False)
|
||||
if accept_type and accept_type[0] == "application/json":
|
||||
return make_response((jsonify(current_app.data.data_frame(df))))
|
||||
# TODO support CSV
|
||||
else:
|
||||
return make_response(f"Unsupported accept-type: {accept_type}", HTTPStatus.NOT_ACCEPTABLE)
|
||||
|
||||
@swagger.doc({
|
||||
"summary": "Get data (expression values) from the dataframe.",
|
||||
"tags": ["data"],
|
||||
"parameters": [
|
||||
{
|
||||
'name': 'filter',
|
||||
'description': 'Complex Filter',
|
||||
'in': 'body',
|
||||
'schema': FilterModel
|
||||
}
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "expression",
|
||||
"examples": {
|
||||
"application/json": {
|
||||
"var": [0, 20000],
|
||||
"obs": [
|
||||
[1, 39483, 3902, 203, 0, 0, 28]
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"400": {
|
||||
"description": "Malformed filter"
|
||||
},
|
||||
"406": {
|
||||
"description": "Unacceptable MIME type"
|
||||
},
|
||||
}
|
||||
})
|
||||
def put(self):
|
||||
if not request.accept_mimetypes.best_match(["application/json", "text/csv"]):
|
||||
return make_response(f"Unsupported MIME type '{request.accept_mimetypes}'", HTTPStatus.NOT_ACCEPTABLE)
|
||||
# TODO catch error for bad filter
|
||||
df = current_app.data.filter_dataframe(request.get_json()["filter"], include_uns=False)
|
||||
if request.accept_mimetypes.best_match(['application/json']):
|
||||
return make_response((jsonify(current_app.data.data_frame(df))))
|
||||
# TODO support CSV
|
||||
else:
|
||||
return make_response(f"Unsupported MIME type '{request.accept_mimetypes}'", HTTPStatus.NOT_ACCEPTABLE)
|
||||
|
||||
|
||||
def get_api_resources():
|
||||
bp = Blueprint("api", __name__, url_prefix="/api/v0.1")
|
||||
bp = Blueprint("api", __name__, url_prefix="/api/v0.2")
|
||||
api = Api(bp, add_api_spec_resource=False)
|
||||
api.add_resource(InitializeAPI, "/initialize")
|
||||
api.add_resource(CellsAPI, "/cells")
|
||||
api.add_resource(ExpressionAPI, "/expression")
|
||||
api.add_resource(DifferentialExpressionAPI, "/diffexpression")
|
||||
api.add_resource(SchemaAPI, "/schema")
|
||||
api.add_resource(ConfigAPI, "/config")
|
||||
api.add_resource(LayoutObsAPI, "/layout/obs")
|
||||
api.add_resource(AnnotationsObsAPI, "/annotations/obs")
|
||||
api.add_resource(DiffExpObsAPI, "/diffexp/obs")
|
||||
api.add_resource(AnnotationsVarAPI, "/annotations/var")
|
||||
api.add_resource(DataObsAPI, "/data/obs")
|
||||
return api
|
||||
|
||||
@@ -1,36 +1,68 @@
|
||||
import os
|
||||
import warnings
|
||||
|
||||
import numpy as np
|
||||
from pandas import DataFrame, Series
|
||||
import scanpy.api as sc
|
||||
from scipy import stats
|
||||
|
||||
from server.app.app import cache
|
||||
# TODO fix memoization so that it correctly identifies the same request
|
||||
# from server.app.app import cache
|
||||
from server.app.driver.driver import CXGDriver
|
||||
from server.app.util.schema_parse import parse_schema
|
||||
from server.app.util.constants import Axis, DEFAULT_TOP_N, DiffExpMode
|
||||
|
||||
|
||||
class ScanpyEngine(CXGDriver):
|
||||
|
||||
def __init__(self, data, schema=None, graph_method="umap", diffexp_method="ttest"):
|
||||
self.data = self._load_data(data)
|
||||
self.schema = self._load_or_infer_schema(data, schema)
|
||||
self._set_cell_names()
|
||||
def __init__(self, data, layout_method=None, diffexp_method=None):
|
||||
super().__init__(data, layout_method=layout_method, diffexp_method=diffexp_method)
|
||||
self._validatate_data_types()
|
||||
self._add_mandatory_annotations()
|
||||
self.cell_count = self.data.shape[0]
|
||||
self.gene_count = self.data.shape[1]
|
||||
self.graph_method = graph_method
|
||||
self.diffexp_method = diffexp_method
|
||||
self._create_schema()
|
||||
self.layout(self.data)
|
||||
|
||||
def _set_cell_names(self):
|
||||
self.data.obs["cell_name"] = list(self.data.obs.index)
|
||||
def _create_schema(self):
|
||||
self.schema = {
|
||||
"dataframe": {
|
||||
"nObs": self.cell_count,
|
||||
"nVar": self.gene_count,
|
||||
"type": str(self.data.X.dtype)
|
||||
},
|
||||
"annotations": {
|
||||
"obs": [],
|
||||
"var": []
|
||||
}
|
||||
}
|
||||
for ax in Axis:
|
||||
curr_axis = getattr(self.data, str(ax))
|
||||
for ann in curr_axis:
|
||||
ann_schema = {"name": ann}
|
||||
data_kind = curr_axis[ann].dtype.kind
|
||||
if data_kind == 'f':
|
||||
ann_schema["type"] = "float32"
|
||||
elif data_kind in ['i', 'u']:
|
||||
ann_schema["type"] = "int32"
|
||||
elif data_kind == "?":
|
||||
ann_schema["type"] = "boolean"
|
||||
elif data_kind == "O" and curr_axis[ann].dtype == "object":
|
||||
ann_schema["type"] = "string"
|
||||
elif data_kind == "O" and curr_axis[ann].dtype == "category":
|
||||
ann_schema["type"] = "categorical"
|
||||
ann_schema["categories"] = curr_axis[ann].dtype.categories.tolist()
|
||||
else:
|
||||
raise TypeError(f"Annotations of type {curr_axis[ann].dtype} are unsupported by cellxgene.")
|
||||
self.schema["annotations"][ax].append(ann_schema)
|
||||
|
||||
@classmethod
|
||||
def add_to_parser(cls, subparsers, invocation_function):
|
||||
scanpy_group = subparsers.add_parser("scanpy", help="run cellxgene using the scanpy engine")
|
||||
# TODO these choices should be generated from the actual available methods
|
||||
# TODO these choices should be generated from the actual available methods see GH issue #94
|
||||
scanpy_group.add_argument("-l", "--layout", choices=["umap", "tsne"], default="umap",
|
||||
help="Algorithm to use for graph layout")
|
||||
scanpy_group.add_argument("-d", "--diffexp", choices=["ttest"], default="ttest",
|
||||
help="Algorithm to use to calculate differential expression")
|
||||
help="Algorithm to used to calculate differential expression")
|
||||
scanpy_group.add_argument("data_directory", metavar="dir", help="Directory containing data and schema file")
|
||||
scanpy_group.set_defaults(func=invocation_function)
|
||||
return scanpy_group
|
||||
@@ -39,206 +71,227 @@ class ScanpyEngine(CXGDriver):
|
||||
def _load_data(data):
|
||||
return sc.read(os.path.join(data, "data.h5ad"))
|
||||
|
||||
def _load_or_infer_schema(self, data, schema):
|
||||
if not os.path.isfile(os.path.join(data, schema)):
|
||||
# Initialize with cell name which is built off the index
|
||||
data_schema = {
|
||||
"CellName": {
|
||||
"type": "string",
|
||||
"variabletype": "categorical",
|
||||
"displayname": "Name",
|
||||
"include": True
|
||||
}
|
||||
}
|
||||
metadata_fields = list(self.data.obs)
|
||||
for m in metadata_fields:
|
||||
# Since there are many type of float/int in numpy datatypes the kind attribute of a datatype object
|
||||
# offers a decent insight into whether it can be lumped in with floats or ints, which is what we
|
||||
# care about here.
|
||||
data_kind = self.data.obs[m].dtype.kind
|
||||
variable_type = "categorical"
|
||||
data_type = "string"
|
||||
if data_kind == 'f':
|
||||
variable_type = "continuous"
|
||||
data_type = "float"
|
||||
elif data_kind in ['i', 'u']:
|
||||
data_type = "int"
|
||||
if self.data.obs[m].nunique() > 50:
|
||||
variable_type = "continuous"
|
||||
data_schema[m] = {
|
||||
"type": data_type,
|
||||
"variabletype": variable_type,
|
||||
"displayname": m,
|
||||
"include": True
|
||||
}
|
||||
else:
|
||||
data_schema = parse_schema(os.path.join(data, schema))
|
||||
return data_schema
|
||||
@staticmethod
|
||||
def _top_sort(values, sort_order, top_n=None):
|
||||
"""
|
||||
Sorts an iterable in sort order limited by top_n
|
||||
:param values: iterable of values to sort
|
||||
:param sort_order: ndarray order to sort in
|
||||
:param top_n: cutoff number to return
|
||||
:return: values sorted by sort_order limited by top_n
|
||||
"""
|
||||
return values[sort_order][:top_n]
|
||||
|
||||
def _add_mandatory_annotations(self):
|
||||
# ensure gene
|
||||
self.data.var["name"] = Series(list(self.data.var.index), dtype="unicode_", index=self.data.var.index)
|
||||
self.data.var.index = Series(list(range(self.data.var.shape[0])), dtype="category")
|
||||
# ensure cell name
|
||||
self.data.obs["name"] = Series(list(self.data.obs.index), dtype="unicode_", index=self.data.obs.index)
|
||||
self.data.obs.index = Series(list(range(self.data.obs.shape[0])), dtype="category")
|
||||
|
||||
def _validatate_data_types(self):
|
||||
if self.data.X.dtype != "float32":
|
||||
warnings.warn(f"Scanpy data matrix is in {self.data.X.dtype} format not float32. "
|
||||
f"Precision may be truncated.")
|
||||
for ax in Axis:
|
||||
curr_axis = getattr(self.data, str(ax))
|
||||
for ann in curr_axis:
|
||||
datatype = curr_axis[ann].dtype
|
||||
downcast_map = {'int64': 'int32',
|
||||
'uint32': 'int32',
|
||||
'uint64': 'int32',
|
||||
'float64': 'float32',
|
||||
}
|
||||
if datatype in downcast_map:
|
||||
warnings.warn(f"Scanpy annotation {ax}:{ann} is in unsupported format: {datatype}. "
|
||||
f"Data will be downcast to {downcast_map[datatype]}.")
|
||||
|
||||
def cells(self):
|
||||
return list(self.data.obs.index)
|
||||
return self.data.obs.index.tolist()
|
||||
|
||||
def genes(self):
|
||||
return self.data.var.index.tolist()
|
||||
|
||||
# Can't seem to cache a view of a dataframe, need to investigate why
|
||||
def filter_cells(self, filter):
|
||||
def filter_dataframe(self, filter, include_uns=True):
|
||||
"""
|
||||
Filter cells from data and return a subset of the data
|
||||
A filter is a dictionary where the key is a metadatata category
|
||||
Value is dictionary
|
||||
value_type: int, float, string
|
||||
variable_type: continuous, categorical
|
||||
query: filter value, for categorical [val1, val2], for continuous {min: x, max:y}
|
||||
Filters are combined with the and operator
|
||||
:param filter:
|
||||
:return: filtered dataframe
|
||||
Filter cells from data and return a subset of the data. They can operate on both obs and var dimension with
|
||||
indexing and filtering by annotation value. Filters are combined with the and operator.
|
||||
See REST specs for info on filter format:
|
||||
# TODO update this link to swagger when it's done
|
||||
https://docs.google.com/document/d/1Fxjp1SKtCk7l8QP9-7KAjGXL0eldi_qEnNT0NmlGzXI/edit#heading=h.8qc9q57amldx
|
||||
|
||||
:param filter: dictionary with filter parames
|
||||
:param include_uns: bool, include unstructured annotations
|
||||
:return: View into scanpy object with cells/genes filtered
|
||||
"""
|
||||
cell_idx = np.ones((self.cell_count,), dtype=bool)
|
||||
for key, value in filter.items():
|
||||
if value["variable_type"] == "categorical":
|
||||
key_idx = np.in1d(getattr(self.data.obs, key), value["query"])
|
||||
cell_idx = np.logical_and(cell_idx, key_idx)
|
||||
cells_idx = np.ones((self.cell_count,), dtype=bool)
|
||||
genes_idx = np.ones((self.gene_count,), dtype=bool)
|
||||
if Axis.OBS in filter:
|
||||
if "index" in filter["obs"]:
|
||||
cells_idx = self._filter_index(filter["obs"]["index"], cells_idx, Axis.OBS)
|
||||
if "annotation_value" in filter["obs"]:
|
||||
cells_idx = self._filter_annotation(filter["obs"]["annotation_value"], cells_idx, Axis.OBS)
|
||||
if Axis.VAR in filter:
|
||||
if "index" in filter["var"]:
|
||||
genes_idx = self._filter_index(filter["var"]["index"], genes_idx, Axis.VAR)
|
||||
if "annotation_value" in filter["var"]:
|
||||
genes_idx = self._filter_annotation(filter["var"]["annotation_value"], genes_idx, Axis.VAR)
|
||||
# Due to anndata issues we can't index into cells and genes at the same time
|
||||
cell_data = self.data[cells_idx, :]
|
||||
data = cell_data[:, genes_idx]
|
||||
# TODO: tmp hack to avoid problems with filter that is limited to single gene
|
||||
if include_uns:
|
||||
data.uns = cell_data.uns
|
||||
return data
|
||||
|
||||
def _filter_index(self, filter, index, axis):
|
||||
"""
|
||||
Filter data based on index. ex. [1, 3, [111:200]]
|
||||
:param filter: subset of filter dict for obs/var:index
|
||||
:param index: np logical vector containing true for passing false for failing filter
|
||||
:param axis: Axis
|
||||
:return: np logical vector for whether the data passes the filter
|
||||
"""
|
||||
if axis == Axis.OBS:
|
||||
count_ = self.cell_count
|
||||
elif axis == Axis.VAR:
|
||||
count_ = self.gene_count
|
||||
idx_filter = np.zeros((count_,), dtype=bool)
|
||||
for i in filter:
|
||||
if type(i) == list:
|
||||
idx_filter[i[0]:i[1]] = True
|
||||
else:
|
||||
min_ = value["query"]["min"]
|
||||
max_ = value["query"]["max"]
|
||||
idx_filter[i] = True
|
||||
return np.logical_and(index, idx_filter)
|
||||
|
||||
def _filter_annotation(self, filter, index, axis):
|
||||
"""
|
||||
Filter data based on annotation value
|
||||
:param filter: subset of filter dict for obs/var:annotation_value
|
||||
:param index: np logical vector containing true for passing false for failing filter
|
||||
:param axis: string obs or var
|
||||
:return: np logical vector for whether the data passes the filter
|
||||
"""
|
||||
d_axis = getattr(self.data, axis.value)
|
||||
for v in filter:
|
||||
if d_axis[v["name"]].dtype.name in ["boolean", "category", "object"]:
|
||||
key_idx = np.in1d(getattr(d_axis, v["name"]), v["values"])
|
||||
index = np.logical_and(index, key_idx)
|
||||
else:
|
||||
min_ = v.get("min", None)
|
||||
max_ = v.get("max", None)
|
||||
if min_ is not None:
|
||||
key_idx = np.array((getattr(self.data.obs, key) >= min_).data)
|
||||
cell_idx = np.logical_and(cell_idx, key_idx)
|
||||
key_idx = (getattr(d_axis, v["name"]) >= min_).ravel()
|
||||
index = np.logical_and(index, key_idx)
|
||||
if max_ is not None:
|
||||
key_idx = np.array((getattr(self.data.obs, key) <= max_).data)
|
||||
cell_idx = np.logical_and(cell_idx, key_idx)
|
||||
return self.data[cell_idx, :]
|
||||
key_idx = (getattr(d_axis, v["name"]) <= max_).ravel()
|
||||
index = np.logical_and(index, key_idx)
|
||||
return index
|
||||
|
||||
@cache.memoize()
|
||||
def metadata_ranges(self, df=None):
|
||||
metadata_ranges = {}
|
||||
if not df:
|
||||
df = self.data
|
||||
for field in self.schema:
|
||||
if self.schema[field]["variabletype"] == "categorical":
|
||||
group_by = field
|
||||
if group_by == "CellName":
|
||||
group_by = "cell_name"
|
||||
metadata_ranges[field] = {"options": df.obs.groupby(group_by).size().to_dict()}
|
||||
else:
|
||||
metadata_ranges[field] = {
|
||||
"range": {
|
||||
"min": df.obs[field].min(),
|
||||
"max": df.obs[field].max()
|
||||
}
|
||||
}
|
||||
return metadata_ranges
|
||||
|
||||
@cache.memoize()
|
||||
def metadata(self, df, fields=None):
|
||||
# @cache.memoize()
|
||||
def annotation(self, df, axis, fields=None):
|
||||
"""
|
||||
Gets metadata key:value for each cells
|
||||
Gets annotation value for each observation
|
||||
|
||||
:param axis:
|
||||
:param df: from filter_cells, dataframe
|
||||
:param fields: list of keys for metadata to return, returns all metadata values if not set.
|
||||
:return: list of metadata values
|
||||
:param fields: list of keys for annotation to return, returns all annotation values if not set.
|
||||
:return: dict: names - list of fields in order, data - list of lists or metadata
|
||||
[observation ids, val1, val2...]
|
||||
"""
|
||||
metadata = df.obs.to_dict(orient="records")
|
||||
for idx in range(len(metadata)):
|
||||
metadata[idx]["CellName"] = metadata[idx].pop("cell_name", None)
|
||||
return metadata
|
||||
df_axis = getattr(df, axis)
|
||||
if not fields:
|
||||
fields = df_axis.columns.tolist()
|
||||
annotations = DataFrame(df_axis[fields], index=df_axis.index)
|
||||
return {
|
||||
"names": fields,
|
||||
"data": annotations.reset_index().values.tolist()
|
||||
}
|
||||
|
||||
@cache.memoize()
|
||||
def create_graph(self, df):
|
||||
# @cache.memoize()
|
||||
def layout(self, df):
|
||||
"""
|
||||
Computes a n-d layout for cells through dimensionality reduction.
|
||||
:param df: from filter_cells, dataframe
|
||||
:return: [cellid, x, y]
|
||||
:return: [cellid, x, y, ...]
|
||||
"""
|
||||
getattr(sc.tl, self.graph_method)(df, random_state=123)
|
||||
graph = df.obsm["X_{graph_method}".format(graph_method=self.graph_method)]
|
||||
normalized_graph = (graph - graph.min()) / (graph.max() - graph.min())
|
||||
return np.hstack((df.obs["cell_name"].values.reshape(len(df.obs.index), 1), normalized_graph)).tolist()
|
||||
|
||||
@cache.memoize()
|
||||
def diffexp(self, cell_list_1, cell_list_2, pval, num_genes):
|
||||
"""
|
||||
Computes the top differentially expressed genes between two clusters
|
||||
:param df1: from filter_cells, dataframe containing first set of cells
|
||||
:param df2: from filter_cells, dataframe containing second set of cells
|
||||
:return: top genes, stats and expression values for top genes
|
||||
"""
|
||||
cells_idx_1 = np.in1d(self.data.obs["cell_name"], cell_list_1)
|
||||
cells_idx_2 = np.in1d(self.data.obs["cell_name"], cell_list_2)
|
||||
expression_1 = self.data.X[cells_idx_1, :]
|
||||
expression_2 = self.data.X[cells_idx_2, :]
|
||||
diff_exp = stats.ttest_ind(expression_1, expression_2)
|
||||
# TODO break this up into functions
|
||||
set1 = np.logical_and(diff_exp.pvalue < pval, diff_exp.statistic > 0)
|
||||
set2 = np.logical_and(diff_exp.pvalue < pval, diff_exp.statistic < 0)
|
||||
stat1 = diff_exp.statistic[set1]
|
||||
stat2 = diff_exp.statistic[set2]
|
||||
sort_set1 = np.argsort(stat1)[::-1]
|
||||
sort_set2 = np.argsort(stat2)
|
||||
pval1 = diff_exp.pvalue[set1][sort_set1]
|
||||
pval2 = diff_exp.pvalue[set2][sort_set2]
|
||||
mean_ex1_set1 = np.mean(expression_1[:, set1], axis=0)[sort_set1]
|
||||
mean_ex2_set1 = np.mean(expression_2[:, set1], axis=0)[sort_set1]
|
||||
mean_ex1_set2 = np.mean(expression_1[:, set2], axis=0)[sort_set2]
|
||||
mean_ex2_set2 = np.mean(expression_2[:, set2], axis=0)[sort_set2]
|
||||
mean_diff1 = mean_ex1_set1 - mean_ex2_set1
|
||||
mean_diff2 = mean_ex1_set2 - mean_ex2_set2
|
||||
genes_cellset_1 = self.data.var_names[set1][sort_set1]
|
||||
genes_cellset_2 = self.data.var_names[set2][sort_set2]
|
||||
# TODO Filtering cells is fine, but filtering genes does nothing because the neighbors are
|
||||
# calculated using the original vars (geneset) and this doesn’t get updated when you use less.
|
||||
# Need to recalculate neighbors (long) if user requests new layout filtered by var
|
||||
getattr(sc.tl, self.layout_method)(df, random_state=123)
|
||||
df_layout = df.obsm[f"X_{self.layout_method}"]
|
||||
normalized_layout = DataFrame((df_layout - df_layout.min()) / (df_layout.max() - df_layout.min()),
|
||||
index=df.obs.index)
|
||||
return {
|
||||
"celllist1": {
|
||||
"topgenes": genes_cellset_1.tolist()[:num_genes],
|
||||
"mean_expression_cellset1": mean_ex1_set1.tolist()[:num_genes],
|
||||
"mean_expression_cellset2": mean_ex2_set1.tolist()[:num_genes],
|
||||
"pval": pval1.tolist()[:num_genes],
|
||||
"ave_diff": mean_diff1.tolist()[:num_genes]
|
||||
},
|
||||
"celllist2": {
|
||||
"topgenes": genes_cellset_2.tolist()[:num_genes],
|
||||
"mean_expression_cellset1": mean_ex1_set2.tolist()[:num_genes],
|
||||
"mean_expression_cellset2": mean_ex2_set2.tolist()[:num_genes],
|
||||
"pval": pval2.tolist()[:num_genes],
|
||||
"ave_diff": mean_diff2.tolist()[:num_genes]
|
||||
},
|
||||
"ndims": normalized_layout.shape[1],
|
||||
# reset_index gets obs' id into output
|
||||
"coordinates": normalized_layout.reset_index().values.tolist()
|
||||
}
|
||||
|
||||
@cache.memoize()
|
||||
def expression(self, cells=None, genes=None):
|
||||
# @cache.memoize()
|
||||
def diffexp(self, df1, df2, top_n=None):
|
||||
"""
|
||||
Retrieves expression for each gene for cells in data frame
|
||||
:param df:
|
||||
Computes the top differentially expressed variables between two observation sets. If dataframes
|
||||
contain a subset of variables, then statistics for all variables will be returned, otherwise
|
||||
only the top N vars will be returned.
|
||||
:param df1: from filter_cells, dataframe containing first set of observations
|
||||
:param df2: from filter_cells, dataframe containing second set of observations
|
||||
:param topN: Limit results to top N (Top var mode only)
|
||||
:return: top genes, stats and expression values for variables
|
||||
"""
|
||||
# If not the same genes, test is wrong!
|
||||
if np.any(df1.var.index != df2.var.index):
|
||||
raise ValueError("Variables ares not the same in set1 and set2")
|
||||
|
||||
# If not all genes, they used a var filter
|
||||
if df1.var.shape[0] < self.gene_count:
|
||||
mode = DiffExpMode.VAR_FILTER
|
||||
if top_n:
|
||||
raise Warning("Top N was specified but will not be used in 'Var Filter' mode")
|
||||
else:
|
||||
mode = DiffExpMode.TOP_N
|
||||
if not top_n:
|
||||
top_n = DEFAULT_TOP_N
|
||||
|
||||
genes_idx = df1.var.index
|
||||
diffexp_result = stats.ttest_ind(df1.X, df2.X)
|
||||
pval = diffexp_result.pvalue
|
||||
bonferroni_pval = 1 - (1 - pval) ** self.gene_count
|
||||
ave_exp_set1 = np.mean(df1.X, axis=0)
|
||||
ave_exp_set2 = np.mean(df2.X, axis=0)
|
||||
ave_diff = ave_exp_set1 - ave_exp_set2
|
||||
if mode == DiffExpMode.TOP_N:
|
||||
sort_order = np.argsort(np.abs(diffexp_result.statistic))[::-1]
|
||||
# If top_n > length it will just return length
|
||||
genes = self._top_sort(genes_idx, sort_order, top_n)
|
||||
pval = self._top_sort(pval, sort_order, top_n)
|
||||
bonferroni_pval = self._top_sort(bonferroni_pval, sort_order, top_n)
|
||||
ave_exp_set1 = self._top_sort(ave_exp_set1, sort_order, top_n)
|
||||
ave_exp_set2 = self._top_sort(ave_exp_set2, sort_order, top_n)
|
||||
ave_diff = self._top_sort(ave_diff, sort_order, top_n)
|
||||
|
||||
# varIndex, avgDiff, pVal, pValAdj, set1AvgExp, set2AvgExp
|
||||
result = []
|
||||
for i in range(len(genes)):
|
||||
result.append([genes[i], ave_diff[i], pval[i], bonferroni_pval[i], ave_exp_set1[i], ave_exp_set2[i]])
|
||||
# Results need to be returned in var index order
|
||||
return sorted(result, key=lambda gene: gene[0])
|
||||
|
||||
# @cache.memoize()
|
||||
def data_frame(self, df):
|
||||
"""
|
||||
Retrieves data for each variable for observations in data frame
|
||||
:param df: from filter_cells, dataframe
|
||||
:return: {
|
||||
"genes": list of genes,
|
||||
"cells": list of cells and expression list,
|
||||
"nonzero_gene_count": number of nonzero genes
|
||||
"var": list of variable ids,
|
||||
"obs": [cellid, var1 expression, var2 expression, ...],
|
||||
}
|
||||
"""
|
||||
if cells:
|
||||
cells_idx = np.in1d(self.data.obs["cell_name"], cells)
|
||||
else:
|
||||
cells_idx = np.ones((self.cell_count,), dtype=bool)
|
||||
if genes:
|
||||
genes_idx = np.in1d(self.data.var_names, genes)
|
||||
else:
|
||||
genes_idx = np.ones((self.gene_count,), dtype=bool)
|
||||
index = np.ix_(cells_idx, genes_idx)
|
||||
expression = self.data.X[index]
|
||||
|
||||
if not genes:
|
||||
genes = self.data.var.index.tolist()
|
||||
if not cells:
|
||||
cells = self.data.obs["cell_name"].tolist()
|
||||
|
||||
cell_data = []
|
||||
for idx, cell in enumerate(cells):
|
||||
cell_data.append({
|
||||
"cellname": cell,
|
||||
"e": list(expression[idx]),
|
||||
})
|
||||
|
||||
var_index = df.var.index.tolist()
|
||||
expression = DataFrame(df.X, index=df.obs.index)
|
||||
return {
|
||||
"genes": genes,
|
||||
"cells": cell_data,
|
||||
"nonzero_gene_count": int(np.sum(expression.any(axis=0)))
|
||||
"var": var_index,
|
||||
"obs": expression.reset_index().values.tolist()
|
||||
}
|
||||
|
||||
@@ -0,0 +1,27 @@
|
||||
from enum import Enum
|
||||
|
||||
|
||||
DEFAULT_TOP_N = 10
|
||||
|
||||
|
||||
class AugmentedEnum(Enum):
|
||||
def __hash__(self):
|
||||
return self.value.__hash__()
|
||||
|
||||
def __eq__(self, other):
|
||||
if isinstance(other, type(self)) or isinstance(other, str):
|
||||
return self.value == other
|
||||
return False
|
||||
|
||||
def __str__(self) -> str:
|
||||
return self.value
|
||||
|
||||
|
||||
class Axis(AugmentedEnum):
|
||||
OBS = "obs"
|
||||
VAR = "var"
|
||||
|
||||
|
||||
class DiffExpMode(AugmentedEnum):
|
||||
TOP_N = "topN"
|
||||
VAR_FILTER = "varFilter"
|
||||
+60
-43
@@ -1,5 +1,16 @@
|
||||
import json
|
||||
from collections import defaultdict
|
||||
|
||||
from numpy import float32, int32
|
||||
|
||||
from server.app.util.constants import Axis
|
||||
|
||||
|
||||
class QueryStringError(Exception):
|
||||
pass
|
||||
|
||||
def __init__(self, key, message):
|
||||
self.key = key
|
||||
self.message = message
|
||||
|
||||
|
||||
def _convert_variable(datatype, variable):
|
||||
@@ -9,62 +20,68 @@ def _convert_variable(datatype, variable):
|
||||
:param datatype: type to convert to
|
||||
:param variable (string or None): value of variable
|
||||
:return: converted variable
|
||||
:raises: ValueError
|
||||
:raises: AssertionError
|
||||
"""
|
||||
try:
|
||||
if variable is None:
|
||||
return variable
|
||||
if datatype == "int":
|
||||
variable = int(variable)
|
||||
elif datatype == "float":
|
||||
variable = float(variable)
|
||||
assert datatype in ["boolean", "categorical", "float32", "int32", "string"]
|
||||
if variable is None:
|
||||
return variable
|
||||
except ValueError:
|
||||
raise
|
||||
if datatype == "int32":
|
||||
variable = int32(variable)
|
||||
elif datatype == "float32":
|
||||
variable = float32(variable)
|
||||
elif datatype == "boolean":
|
||||
variable = json.loads(variable)
|
||||
assert isinstance(variable, bool)
|
||||
return variable
|
||||
|
||||
|
||||
def parse_filter(filter, schema):
|
||||
def parse_filter(query_filter, schema):
|
||||
"""
|
||||
The filter comes in as arguments from a GET/POST request
|
||||
For categorical metadata keys filter based on key=value
|
||||
For continuous metadata keys filter by key=min,max
|
||||
Either value can be replaced by a * To have only a minimum value key=min, To have only a maximum value key=*,max
|
||||
The filter comes in as arguments from a GET request
|
||||
For categorical metadata keys filter based on axis:key=value
|
||||
For continuous metadata keys filter by axis:key=min,max
|
||||
Either value can be replaced by a * To have only a minimum
|
||||
value axis:key=min,* To have only a maximum value axis:key=*,max
|
||||
|
||||
They combine via AND so a cell's metadata would have to match every filter
|
||||
|
||||
The results is a matrix with the cells the pass the filter and at this point all the genes
|
||||
:param filter: flask's request.args
|
||||
:param query_filter: flask's request.args
|
||||
:param schema: dictionary schema
|
||||
:raises QueryStringError
|
||||
:return:
|
||||
"""
|
||||
query = {}
|
||||
for key in filter:
|
||||
value = filter.getlist(key)
|
||||
if key not in schema:
|
||||
raise QueryStringError("Error: key {} not in metadata schema".format(key))
|
||||
query[key] = {
|
||||
"variable_type": schema[key]["variabletype"],
|
||||
"value_type": schema[key]["type"]
|
||||
}
|
||||
if query[key]["variable_type"] == "categorical":
|
||||
query[key]["query"] = [_convert_variable(query[key]["value_type"], v) for v in value]
|
||||
elif query[key]["variable_type"] == "continuous":
|
||||
value = value[0]
|
||||
query = defaultdict(lambda: defaultdict(list))
|
||||
|
||||
for key in query_filter:
|
||||
axis, annotation = key.split(":", 1)
|
||||
try:
|
||||
Axis(axis)
|
||||
except ValueError:
|
||||
raise QueryStringError(key, f"Error: key {key} not in metadata schema")
|
||||
ann_filter = {"name": annotation}
|
||||
for ann in schema[axis]:
|
||||
if ann["name"] == annotation:
|
||||
dtype = ann["type"]
|
||||
break
|
||||
else:
|
||||
raise QueryStringError(key, f"Error: {annotation} not a valid annotation name")
|
||||
if dtype in ["string", "categorical", "boolean"]:
|
||||
ann_filter["values"] = [_convert_variable(dtype, i) for i in query_filter.getlist(key)]
|
||||
else:
|
||||
value = query_filter.get(key)
|
||||
try:
|
||||
min, max = value.split(",")
|
||||
min_, max_ = value.split(",")
|
||||
except ValueError:
|
||||
raise QueryStringError("Error: min,max format required for range for key {}, got {}".format(key, value))
|
||||
if min == "*":
|
||||
min = None
|
||||
if max == "*":
|
||||
max = None
|
||||
raise QueryStringError(key, f"Error: min,max format required for range for {annotation}, got {value}")
|
||||
if min_ == "*":
|
||||
min_ = None
|
||||
if max_ == "*":
|
||||
max_ = None
|
||||
try:
|
||||
query[key]["query"] = {
|
||||
"min": _convert_variable(query[key]["value_type"], min),
|
||||
"max": _convert_variable(query[key]["value_type"], max)
|
||||
}
|
||||
ann_filter["min"] = _convert_variable(dtype, min_)
|
||||
ann_filter["max"] = _convert_variable(dtype, max_)
|
||||
except ValueError:
|
||||
raise QueryStringError(
|
||||
"Error: expected type {} for key {}, got {}".format(query[key]["type"], key, value)
|
||||
)
|
||||
raise QueryStringError(key, f"Error: expected type {query[key]['type']} for key {key}, got {value}")
|
||||
query[axis]["annotation_value"].append(ann_filter)
|
||||
return query
|
||||
|
||||
@@ -0,0 +1,65 @@
|
||||
from flask_restful_swagger_2 import Schema
|
||||
|
||||
|
||||
class AnnotationModel(Schema):
|
||||
type = "object"
|
||||
description = "Filter by annotation key: value"
|
||||
properties = {
|
||||
"name": {
|
||||
"type": "string"
|
||||
},
|
||||
# TODO update to OpenAPI v3.0 when a library is available that supports it
|
||||
# Unfortunately 2.0 doesn't have a way to have a schema that accepts multiple types
|
||||
# Overloading the type key with a list seems to work ok and makes it to the page
|
||||
"values": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": ["float32", "string", "int32", "bool"]
|
||||
}
|
||||
},
|
||||
"min": {
|
||||
"type": ["int32", "float32"],
|
||||
},
|
||||
"max": {
|
||||
"type": ["int32", "float32"],
|
||||
}
|
||||
}
|
||||
required = ["name"]
|
||||
|
||||
|
||||
class IndexModel(Schema):
|
||||
type = "object"
|
||||
description = "Filter by index of observation/variable ex. [0, 5, 15]"
|
||||
properties = {
|
||||
"index": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"format": "int32",
|
||||
"type": "integer"
|
||||
}
|
||||
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
class AxisModel(Schema):
|
||||
type = "object"
|
||||
description = "Axis of data -- obs or var"
|
||||
properties = {
|
||||
"index": IndexModel,
|
||||
"annotation_value": AnnotationModel.array()
|
||||
}
|
||||
|
||||
|
||||
class FilterModel(Schema):
|
||||
type = "object"
|
||||
description = "Complex filter"
|
||||
properties = {
|
||||
"filter": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"obs": AxisModel,
|
||||
"var": AxisModel
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,7 +0,0 @@
|
||||
import json
|
||||
|
||||
|
||||
def parse_schema(filename):
|
||||
with open(filename) as fh:
|
||||
schema = json.load(fh)
|
||||
return schema
|
||||
@@ -1,7 +1,6 @@
|
||||
import json
|
||||
|
||||
from numpy import float32, integer
|
||||
from flask import make_response, jsonify, Response
|
||||
|
||||
|
||||
class Float32JSONEncoder(json.JSONEncoder):
|
||||
@@ -11,30 +10,3 @@ class Float32JSONEncoder(json.JSONEncoder):
|
||||
elif isinstance(obj, integer):
|
||||
return int(obj)
|
||||
return json.JSONEncoder.default(self, obj)
|
||||
|
||||
|
||||
def make_payload(data, errormessage="", errorcode=200):
|
||||
"""
|
||||
Creates JSON respons for requests
|
||||
:param data: json data
|
||||
:param errormessage: error message
|
||||
:param errorcode: http error code
|
||||
:return: flask json repsonse
|
||||
"""
|
||||
error = False
|
||||
if errormessage:
|
||||
error = True
|
||||
# Questionable
|
||||
data = json.loads(json.dumps(data, cls=Float32JSONEncoder))
|
||||
return make_response(jsonify({
|
||||
"data": data,
|
||||
"status": {
|
||||
"error": error,
|
||||
"errormessage": errormessage,
|
||||
}
|
||||
}), errorcode)
|
||||
|
||||
|
||||
def make_streaming_response(data_generator, errorcode=200, content_type="application/json"):
|
||||
# TODO headers
|
||||
return Response(data_generator, status=errorcode, content_type=content_type)
|
||||
|
||||
Reference in New Issue
Block a user