mirror of
https://github.com/chanzuckerberg/cellxgene.git
synced 2026-10-02 12:38:12 +08:00
Restv2 feature branch merge to master (#284)
Move to new REST v0.2 communication between front and back-end. This is a first cut implementation which is functional, but will need follow-up enhancements for performance, error checking, etc. Protocol spec is in docs directory. * Add filtering via indexing * Using new filter specs Indexing working * Added filtering by annotation value * factor out common methods * Documentation * create enum for axis (obs/var) * Better description for filter's return * Add boolean to enumerated types * Augmented enum for scanpy axis * Create schema for annotations Based on datatype within scanpy/anndata + tests * remove obsolete schema parse script * Update rest api to remove old routes and add schema route * Separate development requirements * Warning for unsupported datatypes * include -r requirements.txt in dev * Merged downcast warnings * Fixed bug where names were NaNs Needed to include the index too when creating the series * Add config endpoint * Generate app features from CLI selections * Move features to driver * Add tests for schema * Clearer version wording * python3 version of super * version from engine to package level * move features to driver * Revise layout function to match the new spec * GET for layout/obs * PUT Layout (#211) * PUT Layout * Csweaver/annotations (#212) * Update scanpy engine to support the rest v0.2 annotation requests * GET endpoint for obs annotations + tests * Documentation * Test annotations in scanpy engine * Description for annotation-keys param * annotation->annotations * clarified return for annotations * Use URL query list for annotations fields * parse_filter parses v0.2 GET filters (#215) * parse_filter parses v0.2 GET filters * Don't allow index filters from query params * Better variable conversion * Parse filter improvements - uses default dict - renamed filter -> query_filter * Cleanup Tasks (#216) * Add test_api back into travis build * Do custom JSON encoding the correct way * Run cellxgene server in test setup * Cleanup new tests too * Option to bind to all interfaces (#225) app.run("0.0.0.0") instead of app.run("127.0.0.1") binds to all interfaces. Note: There are comments on the internet that says that the flask server is not up to the task of production serving. I don't think that such scalability concerns apply here, but I was able to get cellxgene working with twistd relatively easily, and we could switch to that if there are scalability concerns. Test plan: browsed to <ip>:5005/api/v0.2/config on a different host. * Add filtering via indexing * Using new filter specs Indexing working * Added filtering by annotation value * factor out common methods * Documentation * create enum for axis (obs/var) * Better description for filter's return * Add boolean to enumerated types * Augmented enum for scanpy axis * Create schema for annotations Based on datatype within scanpy/anndata + tests * remove obsolete schema parse script * Update rest api to remove old routes and add schema route * Separate development requirements * Warning for unsupported datatypes * include -r requirements.txt in dev * Merged downcast warnings * Fixed bug where names were NaNs Needed to include the index too when creating the series * Add config endpoint * Generate app features from CLI selections * Move features to driver * Add tests for schema * Clearer version wording * python3 version of super * version from engine to package level * move features to driver * Revise layout function to match the new spec * GET for layout/obs * PUT Layout (#211) * PUT Layout * Csweaver/annotations (#212) * Update scanpy engine to support the rest v0.2 annotation requests * GET endpoint for obs annotations + tests * Documentation * Test annotations in scanpy engine * Description for annotation-keys param * annotation->annotations * clarified return for annotations * Use URL query list for annotations fields * parse_filter parses v0.2 GET filters (#215) * parse_filter parses v0.2 GET filters * Don't allow index filters from query params * Better variable conversion * Parse filter improvements - uses default dict - renamed filter -> query_filter * Cleanup Tasks (#216) * Add test_api back into travis build * Do custom JSON encoding the correct way * Run cellxgene server in test setup * Cleanup new tests too * Option to bind to all interfaces (#225) app.run("0.0.0.0") instead of app.run("127.0.0.1") binds to all interfaces. Note: There are comments on the internet that says that the flask server is not up to the task of production serving. I don't think that such scalability concerns apply here, but I was able to get cellxgene working with twistd relatively easily, and we could switch to that if there are scalability concerns. Test plan: browsed to <ip>:5005/api/v0.2/config on a different host. * Fix merge errors - import warnings was improperly deleted - scanpy engine tests were totally wrong * Fix merge error with driver * PUT /annotations (#235) * Add query param for annotation name * fix descriptions, eliminate else clause * first cut at initial data load on rest 0.2 api * Annotation var (#248) * Fix bug strings are always objects in pandas * Add axis to annotation method * Add /annotation/var to REST api * Csweaver/expressiondata (#242) * Refactor expression method for REST v2 * Add message to QueryStringError * Fix range filters * Add GET route for /data * /data PUT route * rename expression to data_frame * clarification of error * Improve accept type handling * support all schema types for 0.2 REST API * remove REST 0.1 code; connect var annotations loading * config reducer; use config to set data set title; remove obsolete templating code for data set title * REST 0.2 expression conversion support * partial port of expression to REST 0.2 * diffexp (#273) * Add diffexp method to scanpy and test * Minor tweaks to diffexp Get a minimal working version to unblock FE development * Fixing things git deleted * cleanup print statements * Add index test * additional, partial REST 0.2 bring up of diffexp * Ignore unstructured annotations for data (#275) This is a temp hack, need to figure out how to include data.uns if there is only one gene * diffexp REST 0.2 port finish * ignore unstructured annotaitons on all routes except layout * correctly use varDataCache; maintain state during world rebuild * correct varDataCache use * temporarily disable all memoization * refinements to expression data caching * clear cell sets upon regraph/reset * update version of REST to 0.2 * Travis build fixes - comment out cache import - fix duplicate test name * Remove dependency from travis * clarify semantics of config variables * move generic action helpers into util
This commit is contained in:
Bruce Martin
@@ -28,7 +28,7 @@ obs/cell.
|
||||
|
||||
NOTE: world.obsAnnotation should be identical to the old state.cells value,
|
||||
EXCEPT that
|
||||
* __cellIndex__ renamed to __obsIndex__
|
||||
* __cellIndex__ renamed to __index__
|
||||
* __x__ and __y__ are now in world.obsLayout
|
||||
* __color__ and __colorRBG__ should be moved to controls reducer
|
||||
|
||||
@@ -46,47 +46,50 @@ obs/cell.
|
||||
|
||||
*/
|
||||
|
||||
/*
|
||||
Summary information for each annotation, keyed by annotation name.
|
||||
Value will be an object, containing either 'range' or 'options' object,
|
||||
depending on the annotation schema type (categorical or continuous).
|
||||
/* varDataCache config - see kvCache for semantics */
|
||||
const VarDataCacheLowWatermark = 32; // cache element count
|
||||
const VarDataCacheTTLMs = 1000; // min cache time in MS
|
||||
|
||||
Summarize for BOTH obs and var annotations. Result format:
|
||||
|
||||
{
|
||||
obs: {
|
||||
annotation_name: { ... },
|
||||
...
|
||||
},
|
||||
var: {
|
||||
annotation_name: { ... },
|
||||
...
|
||||
}
|
||||
}
|
||||
|
||||
Example:
|
||||
{
|
||||
"Splice_sites_Annotated": {
|
||||
"range": {
|
||||
"min": 26,
|
||||
"max": 1075869
|
||||
}
|
||||
},
|
||||
"Selection": {
|
||||
"options": {
|
||||
"Astrocytes(HEPACAM)": 714,
|
||||
"Endothelial(BSC)": 123,
|
||||
"Oligodendrocytes(GC)": 294,
|
||||
"Neurons(Thy1)": 685,
|
||||
"Microglia(CD45)": 1108,
|
||||
"Unpanned": 665
|
||||
}
|
||||
}
|
||||
}
|
||||
*/
|
||||
function summarizeAnnotations(schema, obsAnnotations) {
|
||||
/*
|
||||
Build and return obs/var summary using any annotation in the schema
|
||||
|
||||
Summary information for each annotation, keyed by annotation name.
|
||||
Value will be an object, containing either 'range' or 'options' object,
|
||||
depending on the annotation schema type (categorical or continuous).
|
||||
|
||||
Summarize for BOTH obs and var annotations. Result format:
|
||||
|
||||
{
|
||||
obs: {
|
||||
annotation_name: { ... },
|
||||
...
|
||||
},
|
||||
var: {
|
||||
annotation_name: { ... },
|
||||
...
|
||||
}
|
||||
}
|
||||
|
||||
Example:
|
||||
{
|
||||
"Splice_sites_Annotated": {
|
||||
"range": {
|
||||
"min": 26,
|
||||
"max": 1075869
|
||||
}
|
||||
},
|
||||
"Selection": {
|
||||
"options": {
|
||||
"Astrocytes(HEPACAM)": 714,
|
||||
"Endothelial(BSC)": 123,
|
||||
"Oligodendrocytes(GC)": 294,
|
||||
"Neurons(Thy1)": 685,
|
||||
"Microglia(CD45)": 1108,
|
||||
"Unpanned": 665
|
||||
}
|
||||
}
|
||||
}
|
||||
*/
|
||||
const obsSummary = _(schema.annotations.obs)
|
||||
.keyBy("name")
|
||||
@@ -115,7 +118,8 @@ function summarizeAnnotations(schema, obsAnnotations) {
|
||||
})
|
||||
.value();
|
||||
|
||||
const varSummary = {}; // TODO XXX - not currently used, so skip it
|
||||
// TODO XXX - not currently used, so skip it
|
||||
const varSummary = {};
|
||||
|
||||
return {
|
||||
obs: obsSummary,
|
||||
@@ -124,9 +128,6 @@ function summarizeAnnotations(schema, obsAnnotations) {
|
||||
}
|
||||
|
||||
function templateWorld() {
|
||||
const VarDataCacheLowWatermark = 32;
|
||||
const VarDataCacheTTLMs = 1000;
|
||||
|
||||
return {
|
||||
// map from universe obsIndex to world offset.
|
||||
// Undefined / null indicates identity mapping.
|
||||
@@ -186,6 +187,13 @@ export function createWorldFromEntireUniverse(universe) {
|
||||
/* derived data & summaries */
|
||||
world.summary = summarizeAnnotations(world.schema, world.obsAnnotations);
|
||||
|
||||
/* build the varDataCache */
|
||||
world.varDataCache = kvCache.map(
|
||||
universe.varDataCache,
|
||||
val => subsetVarData(world, universe, val),
|
||||
{ lowWatermark: VarDataCacheLowWatermark, minTTL: VarDataCacheTTLMs }
|
||||
);
|
||||
|
||||
return world;
|
||||
}
|
||||
|
||||
@@ -227,13 +235,21 @@ export function createWorldFromCurrentSelection(universe, world, crossfilter) {
|
||||
// build index to our world offset
|
||||
newWorld.worldObsIndex.fill(-1); // default - aka unused
|
||||
for (let i = 0; i < newWorld.nObs; i += 1) {
|
||||
newWorld.worldObsIndex[newWorld.obsAnnotations[i].__obsIndex__] = i;
|
||||
newWorld.worldObsIndex[newWorld.obsAnnotations[i].__index__] = i;
|
||||
}
|
||||
|
||||
/* derived data & summaries */
|
||||
newWorld.summary = summarizeAnnotations(
|
||||
newWorld.schema,
|
||||
newWorld.obsAnnotations
|
||||
);
|
||||
|
||||
/* build the varDataCache */
|
||||
newWorld.varDataCache = kvCache.map(
|
||||
universe.varDataCache,
|
||||
val => subsetVarData(newWorld, universe, val),
|
||||
{ lowWatermark: VarDataCacheLowWatermark, minTTL: VarDataCacheTTLMs }
|
||||
);
|
||||
return newWorld;
|
||||
}
|
||||
|
||||
@@ -243,20 +259,19 @@ export function createWorldFromCurrentSelection(universe, world, crossfilter) {
|
||||
*/
|
||||
function deduceDimensionType(attributes, fieldName) {
|
||||
let dimensionType;
|
||||
if (attributes.type === "string") {
|
||||
const { type } = attributes;
|
||||
if (type === "string" || type === "categorical" || type === "boolean") {
|
||||
dimensionType = "enum";
|
||||
} else if (attributes.type === "int32") {
|
||||
} else if (type === "int32") {
|
||||
dimensionType = Int32Array;
|
||||
} else if (attributes.type === "float32") {
|
||||
} else if (type === "float32") {
|
||||
dimensionType = Float32Array;
|
||||
} else {
|
||||
/*
|
||||
Currently not supporting boolean and categorical types.
|
||||
*/
|
||||
console.error(
|
||||
`Warning - REST API returned unknown metadata schema (${
|
||||
attributes.type
|
||||
}) for field ${fieldName}.`
|
||||
`Warning - REST API returned unknown metadata schema (${type}) for field ${fieldName}.`
|
||||
);
|
||||
// skip it - we don't know what to do with this type
|
||||
}
|
||||
@@ -286,11 +301,11 @@ export function createObsDimensionMap(crossfilter, world) {
|
||||
*/
|
||||
const worldIndex = worldObsIndex ? idx => worldObsIndex[idx] : idx => idx;
|
||||
dimensionMap.x = crossfilter.dimension(
|
||||
r => obsLayout.X[worldIndex(r.__obsIndex__)],
|
||||
r => obsLayout.X[worldIndex(r.__index__)],
|
||||
Float32Array
|
||||
);
|
||||
dimensionMap.y = crossfilter.dimension(
|
||||
r => obsLayout.Y[worldIndex(r.__obsIndex__)],
|
||||
r => obsLayout.Y[worldIndex(r.__index__)],
|
||||
Float32Array
|
||||
);
|
||||
|
||||
@@ -309,7 +324,7 @@ export function subsetVarData(world, universe, varData) {
|
||||
|
||||
const newVarData = new Float32Array(world.nObs);
|
||||
for (let i = 0; i < world.nObs; i += 1) {
|
||||
newVarData[i] = varData[world.obsAnnotations[i].__obsIndex__];
|
||||
newVarData[i] = varData[world.obsAnnotations[i].__index__];
|
||||
}
|
||||
return newVarData;
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user