Files
cellxgene/server/app/driver/driver.py
Bruce Martin eeec842ad0 Restv2 feature branch merge to master (#284)
Move to new REST v0.2 communication between front and back-end.   This is a first cut implementation which is functional, but will need follow-up enhancements for performance, error checking, etc.    Protocol spec is in docs directory.

* Add filtering via indexing

* Using new filter specs

Indexing working

* Added filtering by annotation value

* factor out common methods

* Documentation

* create enum for axis (obs/var)

* Better description for filter's return

* Add boolean to enumerated types

* Augmented enum for scanpy axis

* Create schema for annotations

Based on datatype within scanpy/anndata
+ tests

* remove obsolete schema parse script

* Update rest api to remove old routes and add schema route

* Separate development requirements

* Warning for unsupported datatypes

* include -r requirements.txt in dev

* Merged downcast warnings

* Fixed bug where names were NaNs

Needed to include the index too when creating the series

* Add config endpoint

* Generate app features from CLI selections

* Move features to driver

* Add tests for schema

* Clearer version wording

* python3 version of super

* version from engine to package level

* move features to driver

* Revise layout function to match the new spec

* GET for layout/obs

* PUT Layout (#211)

* PUT Layout

* Csweaver/annotations (#212)


* Update scanpy engine to support the rest v0.2 annotation requests

* GET endpoint for obs annotations + tests

* Documentation

* Test annotations in scanpy engine

* Description for annotation-keys param

* annotation->annotations

* clarified return for annotations

* Use URL query list for annotations fields

* parse_filter parses v0.2 GET filters (#215)

* parse_filter parses v0.2 GET filters

* Don't allow index filters from query params

* Better variable conversion

* Parse filter improvements

- uses default dict
- renamed filter -> query_filter

* Cleanup Tasks (#216)

* Add test_api back into travis build

* Do custom JSON encoding the correct way

* Run cellxgene server in test setup

* Cleanup new tests too

* Option to bind to all interfaces (#225)

app.run("0.0.0.0") instead of app.run("127.0.0.1") binds to all interfaces.

Note: There are comments on the internet that says that the flask server is not up to the task of production serving.  I don't think that such scalability concerns apply here, but I was able to get cellxgene working with twistd relatively easily, and we could switch to that if there are scalability concerns.

Test plan: browsed to <ip>:5005/api/v0.2/config on a different host.

* Add filtering via indexing

* Using new filter specs

Indexing working

* Added filtering by annotation value

* factor out common methods

* Documentation

* create enum for axis (obs/var)

* Better description for filter's return

* Add boolean to enumerated types

* Augmented enum for scanpy axis

* Create schema for annotations

Based on datatype within scanpy/anndata
+ tests

* remove obsolete schema parse script

* Update rest api to remove old routes and add schema route

* Separate development requirements

* Warning for unsupported datatypes

* include -r requirements.txt in dev

* Merged downcast warnings

* Fixed bug where names were NaNs

Needed to include the index too when creating the series

* Add config endpoint

* Generate app features from CLI selections

* Move features to driver

* Add tests for schema

* Clearer version wording

* python3 version of super

* version from engine to package level

* move features to driver

* Revise layout function to match the new spec

* GET for layout/obs

* PUT Layout (#211)

* PUT Layout

* Csweaver/annotations (#212)


* Update scanpy engine to support the rest v0.2 annotation requests

* GET endpoint for obs annotations + tests

* Documentation

* Test annotations in scanpy engine

* Description for annotation-keys param

* annotation->annotations

* clarified return for annotations

* Use URL query list for annotations fields

* parse_filter parses v0.2 GET filters (#215)

* parse_filter parses v0.2 GET filters

* Don't allow index filters from query params

* Better variable conversion

* Parse filter improvements

- uses default dict
- renamed filter -> query_filter

* Cleanup Tasks (#216)

* Add test_api back into travis build

* Do custom JSON encoding the correct way

* Run cellxgene server in test setup

* Cleanup new tests too

* Option to bind to all interfaces (#225)

app.run("0.0.0.0") instead of app.run("127.0.0.1") binds to all interfaces.

Note: There are comments on the internet that says that the flask server is not up to the task of production serving.  I don't think that such scalability concerns apply here, but I was able to get cellxgene working with twistd relatively easily, and we could switch to that if there are scalability concerns.

Test plan: browsed to <ip>:5005/api/v0.2/config on a different host.

* Fix merge errors

- import warnings was improperly deleted
- scanpy engine tests were totally wrong

* Fix merge error with driver

* PUT /annotations (#235)

* Add query param for annotation name

* fix descriptions, eliminate else clause

* first cut at initial data load on rest 0.2 api

* Annotation var (#248)

* Fix bug strings are always objects in pandas

* Add axis to annotation method

* Add /annotation/var to REST api

* Csweaver/expressiondata (#242)

* Refactor expression method for REST v2

* Add message to QueryStringError

* Fix range filters

* Add GET route for /data

* /data PUT route

* rename expression to data_frame

* clarification of error

* Improve accept type handling

* support all schema types for 0.2 REST API

* remove REST 0.1 code; connect var annotations loading

* config reducer; use config to set data set title; remove obsolete templating code for data set title

* REST 0.2 expression conversion support

* partial port of expression to REST 0.2

*  diffexp (#273)

* Add diffexp method to scanpy

and test

* Minor tweaks to diffexp

Get a minimal working version to unblock FE development

* Fixing things git deleted

* cleanup print statements

* Add index test

* additional, partial REST 0.2 bring up of diffexp

* Ignore unstructured annotations for data (#275)

This is a temp hack, need to figure out how to include data.uns if there is only one gene

* diffexp REST 0.2 port finish

* ignore unstructured annotaitons on all routes except layout

* correctly use varDataCache; maintain state during world rebuild

* correct varDataCache use

* temporarily disable all memoization

* refinements to expression data caching

* clear cell sets upon regraph/reset

* update version of REST to 0.2

* Travis build fixes

- comment out cache import
- fix duplicate test name

* Remove dependency from travis

* clarify semantics of config variables

* move generic action helpers into util
2018-10-01 14:58:46 -07:00

103 lines
3.5 KiB
Python

from abc import ABCMeta, abstractmethod
class CXGDriver(metaclass=ABCMeta):
def __init__(self, data, layout_method=None, diffexp_method=None):
self.data = self._load_data(data)
self.layout_method = layout_method
self.diffexp_method = diffexp_method
self.cluster = None
@property
def features(self):
features = {
"cluster": {"available": False},
"layout": {
"obs": {"available": False},
"var": {"available": False},
},
"diffexp": {"available": False}
}
# TODO - Interactive limit should be generated from the actual available methods see GH issue #94
if self.layout_method:
# TODO handle "var" when gene layout becomes available
features["layout"]["obs"] = {"available": True, "interactiveLimit": 15000}
if self.diffexp_method:
features["diffexp"] = {"available": True, "interactiveLimit": 5000}
if self.cluster:
features["cluster"] = {"available": True, "interactiveLimit": 45000}
return features
@staticmethod
@abstractmethod
def _load_data(data):
pass
@abstractmethod
def cells(self):
pass
@abstractmethod
def genes(self):
pass
@abstractmethod
def filter_dataframe(self, filter):
"""
Filter cells from data and return a subset of the data. They can operate on both obs and var dimension with
indexing and filtering by annotation value. Filters are combined with the and operator.
See REST specs for info on filter format:
https://docs.google.com/document/d/1Fxjp1SKtCk7l8QP9-7KAjGXL0eldi_qEnNT0NmlGzXI/edit#heading=h.8qc9q57amldx
:param filter: dictionary with filter parames
:return: View into scanpy object with cells/genes filtered
"""
pass
@abstractmethod
def annotation(self, df, axis, fields=None):
"""
Gets annotation value for each observation
:param axis:
:param df: from filter_cells, dataframe
:param fields: list of keys for annotation to return, returns all annotation values if not set.
:return: dict: names - list of fields in order, data - list of lists or metadata [idx, val1, val2...]
"""
pass
@abstractmethod
def layout(self, df):
"""
Computes a n-d layout for cells through dimensionality reduction.
:param df: from filter_cells, dataframe
:return: [cellid, x, y]
"""
pass
@abstractmethod
def diffexp(self, df1, df2, genes):
"""
Computes the top differentially expressed variables between two observation sets. If dataframes
contain a subset of variables, then statistics for all variables will be returned, otherwise
only the top N vars will be returned.
:param df1: from filter_cells, dataframe containing first set of observations
:param df2: from filter_cells, dataframe containing second set of observations
:param topN: Limit results to top N (Top var mode only)
:return: top genes, stats and expression values for variables
"""
pass
@abstractmethod
def data_frame(self, df):
"""
Retrieves expression for each gene for cells in data frame
:param df: from filter_cells, dataframe
:return: {
"var": list of variable ids,
"obs": [cellid, var1 expression, var2 expression, ...],
}
"""
pass