mirror of
https://github.com/chanzuckerberg/cellxgene.git
synced 2026-10-01 06:28:12 +08:00
Restv2 feature branch merge to master (#284)
Move to new REST v0.2 communication between front and back-end. This is a first cut implementation which is functional, but will need follow-up enhancements for performance, error checking, etc. Protocol spec is in docs directory. * Add filtering via indexing * Using new filter specs Indexing working * Added filtering by annotation value * factor out common methods * Documentation * create enum for axis (obs/var) * Better description for filter's return * Add boolean to enumerated types * Augmented enum for scanpy axis * Create schema for annotations Based on datatype within scanpy/anndata + tests * remove obsolete schema parse script * Update rest api to remove old routes and add schema route * Separate development requirements * Warning for unsupported datatypes * include -r requirements.txt in dev * Merged downcast warnings * Fixed bug where names were NaNs Needed to include the index too when creating the series * Add config endpoint * Generate app features from CLI selections * Move features to driver * Add tests for schema * Clearer version wording * python3 version of super * version from engine to package level * move features to driver * Revise layout function to match the new spec * GET for layout/obs * PUT Layout (#211) * PUT Layout * Csweaver/annotations (#212) * Update scanpy engine to support the rest v0.2 annotation requests * GET endpoint for obs annotations + tests * Documentation * Test annotations in scanpy engine * Description for annotation-keys param * annotation->annotations * clarified return for annotations * Use URL query list for annotations fields * parse_filter parses v0.2 GET filters (#215) * parse_filter parses v0.2 GET filters * Don't allow index filters from query params * Better variable conversion * Parse filter improvements - uses default dict - renamed filter -> query_filter * Cleanup Tasks (#216) * Add test_api back into travis build * Do custom JSON encoding the correct way * Run cellxgene server in test setup * Cleanup new tests too * Option to bind to all interfaces (#225) app.run("0.0.0.0") instead of app.run("127.0.0.1") binds to all interfaces. Note: There are comments on the internet that says that the flask server is not up to the task of production serving. I don't think that such scalability concerns apply here, but I was able to get cellxgene working with twistd relatively easily, and we could switch to that if there are scalability concerns. Test plan: browsed to <ip>:5005/api/v0.2/config on a different host. * Add filtering via indexing * Using new filter specs Indexing working * Added filtering by annotation value * factor out common methods * Documentation * create enum for axis (obs/var) * Better description for filter's return * Add boolean to enumerated types * Augmented enum for scanpy axis * Create schema for annotations Based on datatype within scanpy/anndata + tests * remove obsolete schema parse script * Update rest api to remove old routes and add schema route * Separate development requirements * Warning for unsupported datatypes * include -r requirements.txt in dev * Merged downcast warnings * Fixed bug where names were NaNs Needed to include the index too when creating the series * Add config endpoint * Generate app features from CLI selections * Move features to driver * Add tests for schema * Clearer version wording * python3 version of super * version from engine to package level * move features to driver * Revise layout function to match the new spec * GET for layout/obs * PUT Layout (#211) * PUT Layout * Csweaver/annotations (#212) * Update scanpy engine to support the rest v0.2 annotation requests * GET endpoint for obs annotations + tests * Documentation * Test annotations in scanpy engine * Description for annotation-keys param * annotation->annotations * clarified return for annotations * Use URL query list for annotations fields * parse_filter parses v0.2 GET filters (#215) * parse_filter parses v0.2 GET filters * Don't allow index filters from query params * Better variable conversion * Parse filter improvements - uses default dict - renamed filter -> query_filter * Cleanup Tasks (#216) * Add test_api back into travis build * Do custom JSON encoding the correct way * Run cellxgene server in test setup * Cleanup new tests too * Option to bind to all interfaces (#225) app.run("0.0.0.0") instead of app.run("127.0.0.1") binds to all interfaces. Note: There are comments on the internet that says that the flask server is not up to the task of production serving. I don't think that such scalability concerns apply here, but I was able to get cellxgene working with twistd relatively easily, and we could switch to that if there are scalability concerns. Test plan: browsed to <ip>:5005/api/v0.2/config on a different host. * Fix merge errors - import warnings was improperly deleted - scanpy engine tests were totally wrong * Fix merge error with driver * PUT /annotations (#235) * Add query param for annotation name * fix descriptions, eliminate else clause * first cut at initial data load on rest 0.2 api * Annotation var (#248) * Fix bug strings are always objects in pandas * Add axis to annotation method * Add /annotation/var to REST api * Csweaver/expressiondata (#242) * Refactor expression method for REST v2 * Add message to QueryStringError * Fix range filters * Add GET route for /data * /data PUT route * rename expression to data_frame * clarification of error * Improve accept type handling * support all schema types for 0.2 REST API * remove REST 0.1 code; connect var annotations loading * config reducer; use config to set data set title; remove obsolete templating code for data set title * REST 0.2 expression conversion support * partial port of expression to REST 0.2 * diffexp (#273) * Add diffexp method to scanpy and test * Minor tweaks to diffexp Get a minimal working version to unblock FE development * Fixing things git deleted * cleanup print statements * Add index test * additional, partial REST 0.2 bring up of diffexp * Ignore unstructured annotations for data (#275) This is a temp hack, need to figure out how to include data.uns if there is only one gene * diffexp REST 0.2 port finish * ignore unstructured annotaitons on all routes except layout * correctly use varDataCache; maintain state during world rebuild * correct varDataCache use * temporarily disable all memoization * refinements to expression data caching * clear cell sets upon regraph/reset * update version of REST to 0.2 * Travis build fixes - comment out cache import - fix duplicate test name * Remove dependency from travis * clarify semantics of config variables * move generic action helpers into util
This commit is contained in:
Bruce Martin
+114
-148
@@ -1,60 +1,30 @@
|
||||
// jshint esversion: 6
|
||||
import _ from "lodash";
|
||||
import memoize from "memoize-one";
|
||||
import * as globals from "../globals";
|
||||
import store from "../reducers";
|
||||
import { Universe } from "../util/stateManager";
|
||||
import { Universe, kvCache } from "../util/stateManager";
|
||||
import { catchErrorsWrap, doJsonRequest } from "../util/actionHelpers";
|
||||
|
||||
/*
|
||||
Catch unexpected errors and make sure we don't lose them!
|
||||
*/
|
||||
function catchErrorsWrap(fn) {
|
||||
return (dispatch, getState) => {
|
||||
fn(dispatch, getState).catch(error => {
|
||||
console.error(error);
|
||||
dispatch({ type: "UNEXPECTED ERROR", error });
|
||||
});
|
||||
};
|
||||
}
|
||||
|
||||
async function doRequestInitialize() {
|
||||
const res = await fetch(
|
||||
`${globals.API.prefix}${globals.API.version}initialize`,
|
||||
{
|
||||
method: "get",
|
||||
headers: new Headers({
|
||||
"Content-Type": "application/json"
|
||||
})
|
||||
}
|
||||
);
|
||||
return res.json();
|
||||
}
|
||||
|
||||
async function doRequestCells(query) {
|
||||
const res = await fetch(
|
||||
`${globals.API.prefix}${globals.API.version}cells${query}`,
|
||||
{
|
||||
method: "get",
|
||||
headers: new Headers({
|
||||
"Content-Type": "application/json"
|
||||
})
|
||||
}
|
||||
);
|
||||
return res.json();
|
||||
}
|
||||
|
||||
function doInitialDataLoad(query = "") {
|
||||
return catchErrorsWrap(async dispatch => {
|
||||
const doInitialDataLoad = () =>
|
||||
catchErrorsWrap(async dispatch => {
|
||||
dispatch({ type: "initial data load start" });
|
||||
|
||||
try {
|
||||
const res = await Promise.all([
|
||||
doRequestInitialize(),
|
||||
doRequestCells(query)
|
||||
]);
|
||||
const universe = Universe.createUniverseFromRESTv01Response(
|
||||
res[0],
|
||||
res[1]
|
||||
);
|
||||
const requests = _([
|
||||
"config",
|
||||
"schema",
|
||||
"annotations/obs",
|
||||
"annotations/var",
|
||||
"layout/obs"
|
||||
])
|
||||
.map(r => `${globals.API.prefix}${globals.API.version}${r}`)
|
||||
.map(url => doJsonRequest(url))
|
||||
.value();
|
||||
const results = await Promise.all(requests);
|
||||
const universe = Universe.createUniverseFromRestV02Response(...results);
|
||||
dispatch({
|
||||
type: "configuration load complete",
|
||||
config: results[0].config
|
||||
});
|
||||
dispatch({
|
||||
type: "initial data load complete (universe exists)",
|
||||
universe
|
||||
@@ -63,7 +33,6 @@ function doInitialDataLoad(query = "") {
|
||||
dispatch({ type: "initial data load error", error });
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
// XXX TODO - this is the old code for doing a regraph. Preserving it solely
|
||||
// until we port to 0.2 API. The new UX for regraph can't be implemented on
|
||||
@@ -131,66 +100,56 @@ const resetGraph = () => (dispatch, getState) =>
|
||||
universe: getState().controls.universe
|
||||
});
|
||||
|
||||
// This code defends against the case where /expression returns a cellname
|
||||
// never seen before (ie, not returned by /cells). This should not happen
|
||||
// (see https://github.com/chanzuckerberg/cellxgene-rest-api/issues/34) but
|
||||
// occasionally does.
|
||||
//
|
||||
// XXX TODO - this code is only relevant in v0.1 REST API, and can be retired
|
||||
// when we port to 0.2.
|
||||
//
|
||||
const makeMetadataMap = memoize(metadata => _.keyBy(metadata, "CellName"));
|
||||
function cleanupExpressionResponse(data) {
|
||||
const s = store.getState();
|
||||
const { universe } = s.controls;
|
||||
const metadata = makeMetadataMap(universe.obsAnnotations);
|
||||
let errorFound = false;
|
||||
data.data.cells = _.filter(data.data.cells, cell => {
|
||||
if (!errorFound && !metadata[cell.cellname]) {
|
||||
errorFound = true;
|
||||
console.error(
|
||||
"Warning: /expression REST API returned unexpected cell names -- discarding surprises."
|
||||
);
|
||||
}
|
||||
return metadata[cell.cellname];
|
||||
});
|
||||
|
||||
return data;
|
||||
}
|
||||
|
||||
/*
|
||||
Fetch [gene, ...] from V0.1 API. Not an action function - just a helper
|
||||
which implements the new expression data caching.
|
||||
Fetch expression vectors for each gene in genes. This is NOT an action
|
||||
function, but rather a helper to be called from an action helper that
|
||||
needs expression data.
|
||||
|
||||
Transparently utilizes cached data if it is already present.
|
||||
*/
|
||||
async function _doRequestExpressionData(dispatch, getState, genes) {
|
||||
const state = getState();
|
||||
/* check cache and only fetch data we do not already have */
|
||||
const { universe } = state.controls;
|
||||
const genesToFetch = _.filter(genes, g => !universe.varDataCache[g]);
|
||||
/* preload data already in cache */
|
||||
let expressionData = _.transform(genes, (expData, g) => {
|
||||
const data = kvCache.get(universe.varDataCache, g);
|
||||
if (data) {
|
||||
expData[g] = data;
|
||||
}
|
||||
}); // --> { gene: data }
|
||||
/* make a list of genes for which we do not have data */
|
||||
const genesToFetch = _.filter(genes, g => expressionData[g] === undefined);
|
||||
|
||||
dispatch({ type: "expression load start" });
|
||||
let expressionData = {}; // { gene: data }
|
||||
|
||||
/* Fetch data for any genes not in cache */
|
||||
if (genesToFetch.length) {
|
||||
try {
|
||||
// XXX: TODO - this could be using /data/var rather than /data/obs,
|
||||
// as that would simplify the transformation in
|
||||
// convertExpressionRESTv02ToObject
|
||||
const res = await fetch(
|
||||
`${globals.API.prefix}${globals.API.version}expression`,
|
||||
`${globals.API.prefix}${globals.API.version}data/obs`,
|
||||
{
|
||||
method: "POST",
|
||||
method: "PUT",
|
||||
body: JSON.stringify({
|
||||
genelist: genes
|
||||
filter: {
|
||||
var: {
|
||||
annotation_value: [{ name: "name", values: genesToFetch }]
|
||||
}
|
||||
}
|
||||
}),
|
||||
headers: new Headers({
|
||||
accept: "application/json",
|
||||
"Accept-Encoding": "gzip, deflate, br",
|
||||
"Content-Type": "application/json"
|
||||
})
|
||||
}
|
||||
);
|
||||
let data = await res.json();
|
||||
data = cleanupExpressionResponse(data);
|
||||
data = Universe.convertExpressionRESTv01ToObject(universe, data);
|
||||
const data = await res.json();
|
||||
expressionData = {
|
||||
...expressionData,
|
||||
...data
|
||||
...Universe.convertExpressionRESTv02ToObject(universe, data)
|
||||
};
|
||||
} catch (error) {
|
||||
dispatch({ type: "expression load error", error });
|
||||
@@ -198,26 +157,24 @@ async function _doRequestExpressionData(dispatch, getState, genes) {
|
||||
}
|
||||
}
|
||||
|
||||
// add the cached values
|
||||
_.forEach(genes, g => {
|
||||
if (expressionData[g] === undefined) {
|
||||
expressionData[g] = universe.varDataCache[g];
|
||||
}
|
||||
});
|
||||
|
||||
return dispatch({ type: "expression load success", expressionData });
|
||||
dispatch({ type: "expression load success", expressionData });
|
||||
return expressionData;
|
||||
}
|
||||
|
||||
function requestSingleGeneExpressionCountsForColoringPOST(gene) {
|
||||
return async (dispatch, getState) => {
|
||||
dispatch({ type: "get single gene expression for coloring started" });
|
||||
try {
|
||||
await _doRequestExpressionData(dispatch, getState, [gene]);
|
||||
const expressionData = await _doRequestExpressionData(
|
||||
dispatch,
|
||||
getState,
|
||||
[gene]
|
||||
);
|
||||
dispatch({
|
||||
type: "color by expression",
|
||||
gene,
|
||||
data: {
|
||||
[gene]: getState().controls.world.varDataCache[gene]
|
||||
[gene]: expressionData[gene]
|
||||
}
|
||||
});
|
||||
} catch (error) {
|
||||
@@ -232,65 +189,74 @@ function requestSingleGeneExpressionCountsForColoringPOST(gene) {
|
||||
const requestGeneExpressionCountsPOST = genes => async (dispatch, getState) => {
|
||||
dispatch({ type: "get expression started" });
|
||||
try {
|
||||
await _doRequestExpressionData(dispatch, getState, genes);
|
||||
const expressionData = await _doRequestExpressionData(
|
||||
dispatch,
|
||||
getState,
|
||||
genes
|
||||
);
|
||||
return dispatch({
|
||||
type: "get expression success",
|
||||
genes,
|
||||
data: _.transform(
|
||||
genes,
|
||||
(res, gene) => {
|
||||
res[gene] = getState().controls.world.varDataCache[gene];
|
||||
},
|
||||
{}
|
||||
)
|
||||
data: expressionData
|
||||
});
|
||||
} catch (error) {
|
||||
return dispatch({ type: "get expression error", error });
|
||||
}
|
||||
};
|
||||
|
||||
const requestDifferentialExpression = (
|
||||
celllist1,
|
||||
celllist2,
|
||||
num_genes = 7
|
||||
) => dispatch => {
|
||||
const requestDifferentialExpression = (set1, set2, num_genes = 10) => async (
|
||||
dispatch,
|
||||
getState
|
||||
) => {
|
||||
dispatch({ type: "request differential expression started" });
|
||||
fetch(`${globals.API.prefix}${globals.API.version}diffexpression`, {
|
||||
method: "POST",
|
||||
body: JSON.stringify({
|
||||
celllist1,
|
||||
celllist2,
|
||||
num_genes
|
||||
}),
|
||||
headers: new Headers({
|
||||
accept: "application/json",
|
||||
"Content-Type": "application/json"
|
||||
})
|
||||
})
|
||||
.then(res => res.json())
|
||||
.then(
|
||||
data => {
|
||||
/*
|
||||
kick off a secondary action to get all expression counts for all cells
|
||||
now that we know what the top expressed are
|
||||
*/
|
||||
dispatch(
|
||||
requestGeneExpressionCountsPOST(
|
||||
_.union(data.data.celllist1.topgenes, data.data.celllist2.topgenes)
|
||||
)
|
||||
);
|
||||
/* then send the success case action through */
|
||||
return dispatch({
|
||||
type: "request differential expression success",
|
||||
data
|
||||
});
|
||||
},
|
||||
error =>
|
||||
dispatch({
|
||||
type: "request differential expression error",
|
||||
error
|
||||
try {
|
||||
/*
|
||||
Steps:
|
||||
1. get the most differentially expressed genes
|
||||
2. get expression data for each
|
||||
*/
|
||||
const state = getState();
|
||||
const { universe } = state.controls;
|
||||
const set1ByIndex = _.map(set1, s => universe.obsNameToIndexMap[s]);
|
||||
const set2ByIndex = _.map(set2, s => universe.obsNameToIndexMap[s]);
|
||||
const diffExpFetch = await fetch(
|
||||
`${globals.API.prefix}${globals.API.version}diffexp/obs`,
|
||||
{
|
||||
method: "POST",
|
||||
headers: new Headers({
|
||||
Accept: "application/json",
|
||||
"Accept-Encoding": "gzip, deflate, br",
|
||||
"Content-Type": "application/json"
|
||||
}),
|
||||
body: JSON.stringify({
|
||||
mode: "topN",
|
||||
count: num_genes,
|
||||
set1: { filter: { obs: { index: set1ByIndex } } },
|
||||
set2: { filter: { obs: { index: set2ByIndex } } }
|
||||
})
|
||||
}
|
||||
);
|
||||
const data = await diffExpFetch.json();
|
||||
// result is [ [varIdx, ...], ... ]
|
||||
const topNGenes = _.map(data, r => universe.varAnnotations[r[0]].name);
|
||||
|
||||
/*
|
||||
Kick off secondary action to fetch all of the expression data for the
|
||||
topN expressed genes.
|
||||
*/
|
||||
dispatch(requestGeneExpressionCountsPOST(topNGenes));
|
||||
|
||||
/* then send the success case action through */
|
||||
return dispatch({
|
||||
type: "request differential expression success",
|
||||
data
|
||||
});
|
||||
} catch (error) {
|
||||
return dispatch({
|
||||
type: "request differential expression error",
|
||||
error
|
||||
});
|
||||
}
|
||||
};
|
||||
|
||||
export default {
|
||||
|
||||
Reference in New Issue
Block a user