fork/cellxgene
1
0
Fork 0
mirror of https://github.com/chanzuckerberg/cellxgene.git synced 2026-09-28 16:28:12 +08:00
Code Issues Packages Projects Releases Wiki Activity

Restv2 feature branch merge to master (#284)

Browse Source 
Move to new REST v0.2 communication between front and back-end.   This is a first cut implementation which is functional, but will need follow-up enhancements for performance, error checking, etc.    Protocol spec is in docs directory.

* Add filtering via indexing

* Using new filter specs

Indexing working

* Added filtering by annotation value

* factor out common methods

* Documentation

* create enum for axis (obs/var)

* Better description for filter's return

* Add boolean to enumerated types

* Augmented enum for scanpy axis

* Create schema for annotations

Based on datatype within scanpy/anndata
+ tests

* remove obsolete schema parse script

* Update rest api to remove old routes and add schema route

* Separate development requirements

* Warning for unsupported datatypes

* include -r requirements.txt in dev

* Merged downcast warnings

* Fixed bug where names were NaNs

Needed to include the index too when creating the series

* Add config endpoint

* Generate app features from CLI selections

* Move features to driver

* Add tests for schema

* Clearer version wording

* python3 version of super

* version from engine to package level

* move features to driver

* Revise layout function to match the new spec

* GET for layout/obs

* PUT Layout (#211)

* PUT Layout

* Csweaver/annotations (#212)


* Update scanpy engine to support the rest v0.2 annotation requests

* GET endpoint for obs annotations + tests

* Documentation

* Test annotations in scanpy engine

* Description for annotation-keys param

* annotation->annotations

* clarified return for annotations

* Use URL query list for annotations fields

* parse_filter parses v0.2 GET filters (#215)

* parse_filter parses v0.2 GET filters

* Don't allow index filters from query params

* Better variable conversion

* Parse filter improvements

- uses default dict
- renamed filter -> query_filter

* Cleanup Tasks (#216)

* Add test_api back into travis build

* Do custom JSON encoding the correct way

* Run cellxgene server in test setup

* Cleanup new tests too

* Option to bind to all interfaces (#225)

app.run("0.0.0.0") instead of app.run("127.0.0.1") binds to all interfaces.

Note: There are comments on the internet that says that the flask server is not up to the task of production serving.  I don't think that such scalability concerns apply here, but I was able to get cellxgene working with twistd relatively easily, and we could switch to that if there are scalability concerns.

Test plan: browsed to <ip>:5005/api/v0.2/config on a different host.

* Add filtering via indexing

* Using new filter specs

Indexing working

* Added filtering by annotation value

* factor out common methods

* Documentation

* create enum for axis (obs/var)

* Better description for filter's return

* Add boolean to enumerated types

* Augmented enum for scanpy axis

* Create schema for annotations

Based on datatype within scanpy/anndata
+ tests

* remove obsolete schema parse script

* Update rest api to remove old routes and add schema route

* Separate development requirements

* Warning for unsupported datatypes

* include -r requirements.txt in dev

* Merged downcast warnings

* Fixed bug where names were NaNs

Needed to include the index too when creating the series

* Add config endpoint

* Generate app features from CLI selections

* Move features to driver

* Add tests for schema

* Clearer version wording

* python3 version of super

* version from engine to package level

* move features to driver

* Revise layout function to match the new spec

* GET for layout/obs

* PUT Layout (#211)

* PUT Layout

* Csweaver/annotations (#212)


* Update scanpy engine to support the rest v0.2 annotation requests

* GET endpoint for obs annotations + tests

* Documentation

* Test annotations in scanpy engine

* Description for annotation-keys param

* annotation->annotations

* clarified return for annotations

* Use URL query list for annotations fields

* parse_filter parses v0.2 GET filters (#215)

* parse_filter parses v0.2 GET filters

* Don't allow index filters from query params

* Better variable conversion

* Parse filter improvements

- uses default dict
- renamed filter -> query_filter

* Cleanup Tasks (#216)

* Add test_api back into travis build

* Do custom JSON encoding the correct way

* Run cellxgene server in test setup

* Cleanup new tests too

* Option to bind to all interfaces (#225)

app.run("0.0.0.0") instead of app.run("127.0.0.1") binds to all interfaces.

Note: There are comments on the internet that says that the flask server is not up to the task of production serving.  I don't think that such scalability concerns apply here, but I was able to get cellxgene working with twistd relatively easily, and we could switch to that if there are scalability concerns.

Test plan: browsed to <ip>:5005/api/v0.2/config on a different host.

* Fix merge errors

- import warnings was improperly deleted
- scanpy engine tests were totally wrong

* Fix merge error with driver

* PUT /annotations (#235)

* Add query param for annotation name

* fix descriptions, eliminate else clause

* first cut at initial data load on rest 0.2 api

* Annotation var (#248)

* Fix bug strings are always objects in pandas

* Add axis to annotation method

* Add /annotation/var to REST api

* Csweaver/expressiondata (#242)

* Refactor expression method for REST v2

* Add message to QueryStringError

* Fix range filters

* Add GET route for /data

* /data PUT route

* rename expression to data_frame

* clarification of error

* Improve accept type handling

* support all schema types for 0.2 REST API

* remove REST 0.1 code; connect var annotations loading

* config reducer; use config to set data set title; remove obsolete templating code for data set title

* REST 0.2 expression conversion support

* partial port of expression to REST 0.2

*  diffexp (#273)

* Add diffexp method to scanpy

and test

* Minor tweaks to diffexp

Get a minimal working version to unblock FE development

* Fixing things git deleted

* cleanup print statements

* Add index test

* additional, partial REST 0.2 bring up of diffexp

* Ignore unstructured annotations for data (#275)

This is a temp hack, need to figure out how to include data.uns if there is only one gene

* diffexp REST 0.2 port finish

* ignore unstructured annotaitons on all routes except layout

* correctly use varDataCache; maintain state during world rebuild

* correct varDataCache use

* temporarily disable all memoization

* refinements to expression data caching

* clear cell sets upon regraph/reset

* update version of REST to 0.2

* Travis build fixes

- comment out cache import
- fix duplicate test name

* Remove dependency from travis

* clarify semantics of config variables

* move generic action helpers into util
This commit is contained in:
Bruce Martin
2018-10-01 14:58:46 -07:00
committed by GitHub
parent f0d9d873be
commit eeec842ad0
31 changed files with 2139 additions and 1373 deletions
Download Patch File Download Diff File Expand all files Collapse all files
client/src/actions/index.js
+114 -148
View File
@@ -1,60 +1,30 @@
// jshint esversion: 6
import _ from "lodash";
import memoize from "memoize-one";
import * as globals from "../globals";
import store from "../reducers";
import { Universe } from "../util/stateManager";
import { Universe, kvCache } from "../util/stateManager";
import { catchErrorsWrap, doJsonRequest } from "../util/actionHelpers";
/*
Catch unexpected errors and make sure we don't lose them!
*/
function catchErrorsWrap(fn) {
return (dispatch, getState) => {
fn(dispatch, getState).catch(error => {
console.error(error);
dispatch({ type: "UNEXPECTED ERROR", error });
});
};
}
async function doRequestInitialize() {
const res = await fetch(
`${globals.API.prefix}${globals.API.version}initialize`,
{
method: "get",
headers: new Headers({
"Content-Type": "application/json"
})
}
);
return res.json();
}
async function doRequestCells(query) {
const res = await fetch(
`${globals.API.prefix}${globals.API.version}cells${query}`,
{
method: "get",
headers: new Headers({
"Content-Type": "application/json"
})
}
);
return res.json();
}
function doInitialDataLoad(query = "") {
return catchErrorsWrap(async dispatch => {
const doInitialDataLoad = () =>
catchErrorsWrap(async dispatch => {
dispatch({ type: "initial data load start" });
try {
const res = await Promise.all([
doRequestInitialize(),
doRequestCells(query)
]);
const universe = Universe.createUniverseFromRESTv01Response(
res[0],
res[1]
);
const requests = _([
"config",
"schema",
"annotations/obs",
"annotations/var",
"layout/obs"
])
.map(r => `${globals.API.prefix}${globals.API.version}${r}`)
.map(url => doJsonRequest(url))
.value();
const results = await Promise.all(requests);
const universe = Universe.createUniverseFromRestV02Response(...results);
dispatch({
type: "configuration load complete",
config: results[0].config
});
dispatch({
type: "initial data load complete (universe exists)",
universe
@@ -63,7 +33,6 @@ function doInitialDataLoad(query = "") {
dispatch({ type: "initial data load error", error });
}
});
}
// XXX TODO - this is the old code for doing a regraph. Preserving it solely
// until we port to 0.2 API. The new UX for regraph can't be implemented on
@@ -131,66 +100,56 @@ const resetGraph = () => (dispatch, getState) =>
universe: getState().controls.universe
});
// This code defends against the case where /expression returns a cellname
// never seen before (ie, not returned by /cells). This should not happen
// (see https://github.com/chanzuckerberg/cellxgene-rest-api/issues/34) but
// occasionally does.
//
// XXX TODO - this code is only relevant in v0.1 REST API, and can be retired
// when we port to 0.2.
//
const makeMetadataMap = memoize(metadata => _.keyBy(metadata, "CellName"));
function cleanupExpressionResponse(data) {
const s = store.getState();
const { universe } = s.controls;
const metadata = makeMetadataMap(universe.obsAnnotations);
let errorFound = false;
data.data.cells = _.filter(data.data.cells, cell => {
if (!errorFound && !metadata[cell.cellname]) {
errorFound = true;
console.error(
"Warning: /expression REST API returned unexpected cell names -- discarding surprises."
);
}
return metadata[cell.cellname];
});
return data;
}
/*
Fetch [gene, ...] from V0.1 API. Not an action function - just a helper
which implements the new expression data caching.
Fetch expression vectors for each gene in genes. This is NOT an action
function, but rather a helper to be called from an action helper that
needs expression data.
Transparently utilizes cached data if it is already present.
*/
async function _doRequestExpressionData(dispatch, getState, genes) {
const state = getState();
/* check cache and only fetch data we do not already have */
const { universe } = state.controls;
const genesToFetch = _.filter(genes, g => !universe.varDataCache[g]);
/* preload data already in cache */
let expressionData = _.transform(genes, (expData, g) => {
const data = kvCache.get(universe.varDataCache, g);
if (data) {
expData[g] = data;
}
}); // --> { gene: data }
/* make a list of genes for which we do not have data */
const genesToFetch = _.filter(genes, g => expressionData[g] === undefined);
dispatch({ type: "expression load start" });
let expressionData = {}; // { gene: data }
/* Fetch data for any genes not in cache */
if (genesToFetch.length) {
try {
// XXX: TODO - this could be using /data/var rather than /data/obs,
// as that would simplify the transformation in
// convertExpressionRESTv02ToObject
const res = await fetch(
`${globals.API.prefix}${globals.API.version}expression`,
`${globals.API.prefix}${globals.API.version}data/obs`,
{
method: "POST",
method: "PUT",
body: JSON.stringify({
genelist: genes
filter: {
var: {
annotation_value: [{ name: "name", values: genesToFetch }]
}
}
}),
headers: new Headers({
accept: "application/json",
"Accept-Encoding": "gzip, deflate, br",
"Content-Type": "application/json"
})
}
);
let data = await res.json();
data = cleanupExpressionResponse(data);
data = Universe.convertExpressionRESTv01ToObject(universe, data);
const data = await res.json();
expressionData = {
...expressionData,
...data
...Universe.convertExpressionRESTv02ToObject(universe, data)
};
} catch (error) {
dispatch({ type: "expression load error", error });
@@ -198,26 +157,24 @@ async function _doRequestExpressionData(dispatch, getState, genes) {
}
}
// add the cached values
_.forEach(genes, g => {
if (expressionData[g] === undefined) {
expressionData[g] = universe.varDataCache[g];
}
});
return dispatch({ type: "expression load success", expressionData });
dispatch({ type: "expression load success", expressionData });
return expressionData;
}
function requestSingleGeneExpressionCountsForColoringPOST(gene) {
return async (dispatch, getState) => {
dispatch({ type: "get single gene expression for coloring started" });
try {
await _doRequestExpressionData(dispatch, getState, [gene]);
const expressionData = await _doRequestExpressionData(
dispatch,
getState,
[gene]
);
dispatch({
type: "color by expression",
gene,
data: {
[gene]: getState().controls.world.varDataCache[gene]
[gene]: expressionData[gene]
}
});
} catch (error) {
@@ -232,65 +189,74 @@ function requestSingleGeneExpressionCountsForColoringPOST(gene) {
const requestGeneExpressionCountsPOST = genes => async (dispatch, getState) => {
dispatch({ type: "get expression started" });
try {
await _doRequestExpressionData(dispatch, getState, genes);
const expressionData = await _doRequestExpressionData(
dispatch,
getState,
genes
);
return dispatch({
type: "get expression success",
genes,
data: _.transform(
genes,
(res, gene) => {
res[gene] = getState().controls.world.varDataCache[gene];
},
{}
)
data: expressionData
});
} catch (error) {
return dispatch({ type: "get expression error", error });
}
};
const requestDifferentialExpression = (
celllist1,
celllist2,
num_genes = 7
) => dispatch => {
const requestDifferentialExpression = (set1, set2, num_genes = 10) => async (
dispatch,
getState
) => {
dispatch({ type: "request differential expression started" });
fetch(`${globals.API.prefix}${globals.API.version}diffexpression`, {
method: "POST",
body: JSON.stringify({
celllist1,
celllist2,
num_genes
}),
headers: new Headers({
accept: "application/json",
"Content-Type": "application/json"
})
})
.then(res => res.json())
.then(
data => {
/*
kick off a secondary action to get all expression counts for all cells
now that we know what the top expressed are
*/
dispatch(
requestGeneExpressionCountsPOST(
_.union(data.data.celllist1.topgenes, data.data.celllist2.topgenes)
)
);
/* then send the success case action through */
return dispatch({
type: "request differential expression success",
data
});
},
error =>
dispatch({
type: "request differential expression error",
error
try {
/*
Steps:
1. get the most differentially expressed genes
2. get expression data for each
*/
const state = getState();
const { universe } = state.controls;
const set1ByIndex = _.map(set1, s => universe.obsNameToIndexMap[s]);
const set2ByIndex = _.map(set2, s => universe.obsNameToIndexMap[s]);
const diffExpFetch = await fetch(
`${globals.API.prefix}${globals.API.version}diffexp/obs`,
{
method: "POST",
headers: new Headers({
Accept: "application/json",
"Accept-Encoding": "gzip, deflate, br",
"Content-Type": "application/json"
}),
body: JSON.stringify({
mode: "topN",
count: num_genes,
set1: { filter: { obs: { index: set1ByIndex } } },
set2: { filter: { obs: { index: set2ByIndex } } }
})
}
);
const data = await diffExpFetch.json();
// result is [ [varIdx, ...], ... ]
const topNGenes = _.map(data, r => universe.varAnnotations[r[0]].name);
/*
Kick off secondary action to fetch all of the expression data for the
topN expressed genes.
*/
dispatch(requestGeneExpressionCountsPOST(topNGenes));
/* then send the success case action through */
return dispatch({
type: "request differential expression success",
data
});
} catch (error) {
return dispatch({
type: "request differential expression error",
error
});
}
};
export default {
client/src/components/expression/diffExpHeatmap.js
+79 -100
View File
@@ -3,15 +3,13 @@ import React from "react";
import _ from "lodash";
import memoize from "memoize-one";
import { connect } from "react-redux";
import * as globals from "../../globals";
import styles from "./expression.css";
import SectionHeader from "../framework/sectionHeader";
import actions from "../../actions";
import ReactAutocomplete from "react-autocomplete"; /* http://emilebres.github.io/react-virtualized-checkbox/ */
import getContrast from "font-color-contrast"; // https://www.npmjs.com/package/font-color-contrast
import FaPaintBrush from "react-icons/lib/fa/paint-brush";
import * as d3 from "d3";
import { interpolateGreys } from "d3-scale-chromatic";
import * as globals from "../../globals";
import actions from "../../actions";
class HeatmapSquare extends React.Component {
constructor(props) {
@@ -22,9 +20,10 @@ class HeatmapSquare extends React.Component {
}
render() {
const { backgroundColor, text } = this.props;
const contrastColor = getContrast(
this.props.backgroundColor
.substring(4, this.props.backgroundColor.length - 1)
backgroundColor
.substring(4, backgroundColor.length - 1)
.replace(/ /g, "")
.split(",")
);
@@ -38,10 +37,10 @@ class HeatmapSquare extends React.Component {
flexShrink: 0,
fontSize: 12,
margin: 0,
backgroundColor: this.props.backgroundColor
backgroundColor
}}
>
{this.props.text}
{text}
</p>
);
}
@@ -54,13 +53,11 @@ class HeatmapSquare extends React.Component {
***********************************
***********************************
**********************************/
@connect(state => {
return {
scatterplotXXaccessor: state.controls.scatterplotXXaccessor,
scatterplotYYaccessor: state.controls.scatterplotYYaccessor,
colorAccessor: state.controls.colorAccessor
};
})
@connect(state => ({
scatterplotXXaccessor: state.controls.scatterplotXXaccessor,
scatterplotYYaccessor: state.controls.scatterplotYYaccessor,
colorAccessor: state.controls.colorAccessor
}))
class HeatmapRow extends React.Component {
constructor(props) {
super(props);
@@ -69,35 +66,44 @@ class HeatmapRow extends React.Component {
};
}
handleGeneColorScaleClick(gene) {
handleGeneColorScaleClick() {
return () => {
this.props.dispatch(
actions.requestSingleGeneExpressionCountsForColoringPOST(
this.props.gene
)
);
const { dispatch, gene } = this.props;
dispatch(actions.requestSingleGeneExpressionCountsForColoringPOST(gene));
};
}
handleSetGeneAsScatterplotX(gene) {
handleSetGeneAsScatterplotX() {
return () => {
this.props.dispatch({
const { dispatch, gene } = this.props;
dispatch({
type: "set scatterplot x",
data: this.props.gene
data: gene
});
};
}
handleSetGeneAsScatterplotY(gene) {
handleSetGeneAsScatterplotY() {
return () => {
this.props.dispatch({
const { dispatch, gene } = this.props;
dispatch({
type: "set scatterplot y",
data: this.props.gene
data: gene
});
};
}
render() {
const {
gene,
aveDiff,
set1exp,
set2exp,
greyColorScale,
scatterplotXXaccessor,
scatterplotYYaccessor,
colorAccessor
} = this.props;
return (
<div
style={{
@@ -113,35 +119,33 @@ class HeatmapRow extends React.Component {
fontSize: 14
}}
>
{this.props.gene}
{gene}
</span>
</div>
<HeatmapSquare
backgroundColor={this.props.greyColorScale(this.props.set1exp)}
text={this.props.set1exp}
backgroundColor={greyColorScale(set1exp)}
text={set1exp.toPrecision(2)}
/>
<HeatmapSquare
backgroundColor={this.props.greyColorScale(this.props.set2exp)}
text={this.props.set2exp}
backgroundColor={greyColorScale(set2exp)}
text={set2exp.toPrecision(2)}
/>
<span
title={this.props.aveDiff}
title={aveDiff}
style={{
fontSize: 12,
marginLeft: 10,
marginRight: 10
}}
>
{this.props.aveDiff.toFixed(2)}
{aveDiff.toFixed(2)}
</span>
<span
onClick={this.handleSetGeneAsScatterplotX(this.props.gene).bind(this)}
onClick={this.handleSetGeneAsScatterplotX(gene).bind(this)}
style={{
fontSize: 16,
color:
this.props.scatterplotXXaccessor === this.props.gene
? "white"
: globals.brightBlue,
scatterplotXXaccessor === gene ? "white" : globals.brightBlue,
cursor: "pointer",
position: "relative",
top: 1,
@@ -150,21 +154,17 @@ class HeatmapRow extends React.Component {
borderRadius: 3,
padding: "2px 3px",
backgroundColor:
this.props.scatterplotXXaccessor === this.props.gene
? globals.brightBlue
: "inherit"
scatterplotXXaccessor === gene ? globals.brightBlue : "inherit"
}}
>
X
</span>
<span
onClick={this.handleSetGeneAsScatterplotY(this.props.gene).bind(this)}
onClick={this.handleSetGeneAsScatterplotY(gene).bind(this)}
style={{
fontSize: 16,
color:
this.props.scatterplotYYaccessor === this.props.gene
? "white"
: globals.brightBlue,
scatterplotYYaccessor === gene ? "white" : globals.brightBlue,
cursor: "pointer",
position: "relative",
top: 1,
@@ -173,15 +173,13 @@ class HeatmapRow extends React.Component {
borderRadius: 3,
padding: "2px 3px",
backgroundColor:
this.props.scatterplotYYaccessor === this.props.gene
? globals.brightBlue
: "inherit"
scatterplotYYaccessor === gene ? globals.brightBlue : "inherit"
}}
>
Y
</span>
<span
onClick={this.handleGeneColorScaleClick(this.props.gene).bind(this)}
onClick={this.handleGeneColorScaleClick(gene).bind(this)}
style={{
fontSize: 16,
cursor: "pointer",
@@ -189,14 +187,9 @@ class HeatmapRow extends React.Component {
marginRight: 6,
borderRadius: 3,
padding: "0px 2px 2px 2px",
color:
this.props.colorAccessor === this.props.gene
? "white"
: "inherit",
color: colorAccessor === gene ? "white" : "inherit",
backgroundColor:
this.props.colorAccessor === this.props.gene
? globals.brightBlue
: "inherit"
colorAccessor === gene ? globals.brightBlue : "inherit"
}}
>
<FaPaintBrush style={{ display: "inline-block" }} />
@@ -214,12 +207,10 @@ class HeatmapRow extends React.Component {
***********************************
**********************************/
@connect(state => {
return {
differential: state.differential,
world: state.controls.world
};
})
@connect(state => ({
differential: state.differential,
world: state.controls.world
}))
class Heatmap extends React.Component {
constructor(props) {
super(props);
@@ -228,24 +219,31 @@ class Heatmap extends React.Component {
};
}
getAllGeneNames = memoize(world =>
_.map(this.props.world.varAnnotations, "name")
);
// XXX TODO unused at the moment
// getAllGeneNames = memoize(world =>
// _.map(this.props.world.varAnnotations, "name")
// );
render() {
if (!this.props.differential.diffExp)
const { world, differential } = this.props;
if (!differential.diffExp) {
return <p>Select cells & compute differential to see heatmap</p>;
}
const topGenesForCellSet1 = this.props.differential.diffExp.data.celllist1;
const topGenesForCellSet2 = this.props.differential.diffExp.data.celllist2;
// const allGeneNames = this.getAllGeneNames(this.props.world);
// summarize the information for display.
const topGenes = _.map(differential.diffExp, val => ({
varIndex: val[0],
geneName: world.varAnnotations[val[0]].name,
avgDiff: val[1],
set1AvgExp: val[4],
set2AvgExp: val[5]
}));
// average expression extent
const extent = d3.extent(
_.union(
topGenesForCellSet1.mean_expression_cellset1,
topGenesForCellSet1.mean_expression_cellset2,
topGenesForCellSet2.mean_expression_cellset1,
topGenesForCellSet2.mean_expression_cellset2
_.concat(
_.map(differential.diffExp, val => val[4]),
_.map(differential.diffExp, val => val[5])
)
);
@@ -269,35 +267,16 @@ class Heatmap extends React.Component {
<p style={{ marginRight: 20 }}>2</p>
<p>ave diff</p>
</div>
{topGenesForCellSet1.topgenes.map((gene, i) => {
{topGenes.map(g => {
const { geneName, avgDiff, set1AvgExp, set2AvgExp } = g;
return (
<HeatmapRow
key={gene}
gene={gene}
key={geneName}
gene={geneName}
greyColorScale={greyColorScale}
aveDiff={topGenesForCellSet1.ave_diff[i]}
set1exp={Math.floor(
topGenesForCellSet1.mean_expression_cellset1[i]
)}
set2exp={Math.floor(
topGenesForCellSet1.mean_expression_cellset2[i]
)}
/>
);
})}
{topGenesForCellSet2.topgenes.map((gene, i) => {
return (
<HeatmapRow
key={gene}
gene={gene}
greyColorScale={greyColorScale}
aveDiff={topGenesForCellSet2.ave_diff[i]}
set1exp={Math.floor(
topGenesForCellSet2.mean_expression_cellset1[i]
)}
set2exp={Math.floor(
topGenesForCellSet2.mean_expression_cellset2[i]
)}
aveDiff={avgDiff}
set1exp={set1AvgExp}
set2exp={set2AvgExp}
/>
);
})}
client/src/components/expression/expressionButtons.js
+18 -28
View File
@@ -6,14 +6,12 @@ import * as globals from "../../globals";
import actions from "../../actions";
import CellSetButton from "./cellSetButtons";
@connect(state => {
return {
differential: state.differential,
world: state.controls.world,
crossfilter: _.get(state.controls, "crossfilter", null),
selectionUpdate: _.get(state.controls, "crossfilter.updateTime", null)
};
})
@connect(state => ({
differential: state.differential,
world: state.controls.world,
crossfilter: state.controls.crossfilter,
selectionUpdate: _.get(state.controls, "crossfilter.updateTime", null)
}))
class Expression extends React.Component {
constructor(props) {
super(props);
@@ -22,24 +20,27 @@ class Expression extends React.Component {
handleClick(gene) {
return () => {
this.props.dispatch({
const { dispatch } = this.props;
dispatch({
type: "color by expression",
gene: gene
gene
});
};
}
computeDiffExp() {
this.props.dispatch(
const { dispatch, differential } = this.props;
dispatch(
actions.requestDifferentialExpression(
this.props.differential.celllist1,
this.props.differential.celllist2
differential.celllist1,
differential.celllist2
)
);
}
render() {
if (!this.props.differential) {
const { differential } = this.props;
if (!differential) {
return null;
}
return (
@@ -47,6 +48,7 @@ class Expression extends React.Component {
<CellSetButton {...this.props} eitherCellSetOneOrTwo={1} />
<CellSetButton {...this.props} eitherCellSetOneOrTwo={2} />
<button
type="button"
style={{
fontSize: 14,
fontWeight: 400,
@@ -55,14 +57,12 @@ class Expression extends React.Component {
height: 30,
borderRadius: 2,
backgroundColor:
this.props.differential.celllist1 &&
this.props.differential.celllist2
differential.celllist1 && differential.celllist2
? globals.brightBlue
: globals.lightGrey,
border: "none",
cursor:
this.props.differential.celllist1 &&
this.props.differential.celllist2
differential.celllist1 && differential.celllist2
? "pointer"
: "auto"
}}
@@ -76,13 +76,3 @@ class Expression extends React.Component {
}
export default Expression;
// <div style={{ marginBottom: 15, width: 300 }}>
// There are currently
// {" " +
// (this.props.crossfilter
// ? this.props.crossfilter.cells.countFiltered()
// : 0) +
// " "}
// cells selected, click a cell set button to store them.
// </div>
client/src/components/leftsidebar.js
+6 -4
View File
@@ -1,4 +1,5 @@
// jshint esversion: 6
import _ from "lodash";
import React from "react";
import { connect } from "react-redux";
import Categorical from "./categorical/categorical";
@@ -9,7 +10,8 @@ import * as globals from "../globals";
import DynamicScatterplot from "./scatterplot/scatterplot";
@connect(state => ({
responsive: state.responsive
responsive: state.responsive,
datasetTitle: _.get(state.config, "displayNames.dataset")
}))
class LeftSideBar extends React.Component {
constructor(props) {
@@ -21,7 +23,7 @@ class LeftSideBar extends React.Component {
render() {
const { currentTab } = this.state;
const { responsive } = this.props;
const { responsive, datasetTitle } = this.props;
/*
this magic number should be made less fragile,
@@ -39,8 +41,8 @@ class LeftSideBar extends React.Component {
width: "100%"
}}
>
cellxgene
{globals.datasetTitle}{" "}
cellxgene &nbsp;
{datasetTitle}
</p>
<div style={{ padding: 10 }}>
<button
client/src/components/scatterplot/scatterplot.js
+3 -2
View File
@@ -17,6 +17,7 @@ import _drawPoints from "./drawPointsRegl";
import { scaleLinear } from "../../util/scaleLinear";
import { margin, width, height } from "./util";
import { kvCache } from "../../util/stateManager";
@connect(state => {
const {
@@ -27,11 +28,11 @@ import { margin, width, height } from "./util";
} = state.controls;
const expressionX =
world && scatterplotXXaccessor
? state.controls.world.varDataCache[scatterplotXXaccessor]
? kvCache.get(world.varDataCache, scatterplotXXaccessor)
: null;
const expressionY =
world && scatterplotYYaccessor
? state.controls.world.varDataCache[scatterplotYYaccessor]
? kvCache.get(world.varDataCache, scatterplotYYaccessor)
: null;
return {
client/src/globals.js
+4 -9
View File
@@ -49,19 +49,14 @@ export const bolder = 700;
export let API = {
// prefix: "http://api.clustering.czi.technology/api/",
//prefix: "http://tabulamuris.cxg.czi.technology/api/",
prefix: "http://api-staging.clustering.czi.technology/api/",
version: "v0.1/"
// prefix: "http://tabulamuris.cxg.czi.technology/api/",
// prefix: "http://api-staging.clustering.czi.technology/api/",
prefix: "http://localhost:5005/api/",
version: "v0.2/"
};
if (window.CELLXGENE && window.CELLXGENE.API) API = window.CELLXGENE.API;
export let datasetTitle = "";
if (window.CELLXGENE && window.CELLXGENE.datasetTitle)
datasetTitle = window.CELLXGENE.datasetTitle;
export const accentFont = "Georgia,Times,Times New Roman,serif";
export const maxParagraphWidth = 600;
export const maxControlsWidth = 800;
client/src/reducers/config.js
+33
View File
@@ -0,0 +1,33 @@
// jshint esversion: 6
const Config = (
state = {
displayNames: null,
features: null
},
action
) => {
switch (action.type) {
case "initial data load start":
return {
...state,
loading: true,
error: null
};
case "configuration load complete":
return {
...state,
loading: false,
error: null,
...action.config
};
case "initial data load error":
return {
...state,
error: action.error
};
default:
return state;
}
};
export default Config;
client/src/reducers/differential.js
+9
View File
@@ -1,4 +1,6 @@
// jshint esversion: 6
import _ from "lodash";
const Differential = (
state = {
diffExp: null,
@@ -39,6 +41,13 @@ const Differential = (
...state,
celllist2: action.data
};
case "reset World to eq Universe":
case "set World to current selection":
return {
...state,
celllist1: null,
celllist2: null
};
default:
return state;
}
client/src/reducers/index.js
+2
View File
@@ -4,11 +4,13 @@ import thunk from "redux-thunk";
import updateURLMiddleware from "../middleware/updateURLMiddleware";
import updateCellColors from "../middleware/updateCellColors";
import config from "./config";
import differential from "./differential";
import responsive from "./responsive";
import controls from "./controls";
const Reducer = combineReducers({
config,
responsive,
controls,
differential
client/src/util/actionHelpers.js
+28
View File
@@ -0,0 +1,28 @@
/*
Catch unexpected errors and make sure we don't lose them!
*/
export function catchErrorsWrap(fn) {
return (dispatch, getState) => {
fn(dispatch, getState).catch(error => {
console.error(error);
dispatch({ type: "UNEXPECTED ERROR", error });
});
};
}
/*
Bootstrap application with the initial data loading.
* /config - application configuration
* /schema - schema of dataframe
* /annotations/obs - all metadata annotation
*/
export const doJsonRequest = async url => {
const res = await fetch(url, {
method: "get",
headers: new Headers({
"Content-Type": "application/json",
"Accept-Encoding": "gzip, deflate, br"
})
});
return res.json();
};
client/src/util/stateManager/keyvalcache.js
+20 -4
View File
@@ -4,13 +4,12 @@ import _ from "lodash";
/*
Very simple key/value cache for use by World & Universe.
* constructor(lowWatermark, cachekey):
* constructor(lowWatermark, minTTL):
- lowWatermark defines the number of cache elements below which
flushing will not occur.
- minTTL defines minimum time in MS that cache entries will live.
- minTTL defines minimum time in milliseconds that cache entries will live.
A value of -1 disables automatic flushing (flush() can still
be called by external user).
- cachekey is a key that will be assigned to any value to track age
* set() - add a key/val pair.
* get() - get a value or undefined if not present.
* flush(minAgeMs) - flush cache entries in excess of lowWatermark if those
@@ -69,4 +68,21 @@ function flush(kvcache, minAgeMs = 0) {
return kvcache;
}
export { create, get, set, flush };
/*
use to create a cache that is a transformation of another cache.
*/
function map(srcKvCache, cb, createOptions) {
const keysInSrcKvCache = _(srcKvCache)
.keys()
.filter(k => k !== cachePrivateKey)
.value();
const newKvCache = create(createOptions.lowWatermark, createOptions.minTTL);
_.forEach(keysInSrcKvCache, key => {
const val = cb(get(srcKvCache, key));
newKvCache[key] = val;
val[cachePrivateKey] = Date.now();
});
return newKvCache;
}
export { create, get, set, flush, map };
client/src/util/stateManager/universe.js
+159 -185
View File
@@ -3,6 +3,44 @@
import _ from "lodash";
import * as kvCache from "./keyvalcache";
/*
Private helper function - create and return a template Universe
*/
function templateUniverse() {
/* default universe template */
/* varDataCache config - see kvCache for semantics */
const VarDataCacheLowWatermark = 32; // cache element count
const VarDataCacheTTLMs = 1000; // min cache time in MS
return {
api: null,
finalized: false, // XXX: may not be needed
nObs: 0,
nVar: 0,
schema: {},
/*
Annotations
*/
obsAnnotations: [] /* all obs annotations, by obs index */,
varAnnotations: [] /* all var annotations, by var index */,
obsNameToIndexMap: {} /* reverse map 'name' to index */,
varNameToIndexMap: {} /* reverse map 'name' to index */,
obsLayout: { X: [], Y: [] } /* xy layout */,
/*
Cache of var data (expression), by var annotation name. Data can be
accesses as a POJO, but if you want caching semantics, use the kvCache
API (eg., kvCache.get(), kvCache.set(), ...), which will maintain the
LRU semantics.
*/
varDataCache: kvCache.create(VarDataCacheLowWatermark, VarDataCacheTTLMs)
};
}
/*
This module implements functions that support storage of "Universe",
aka all of the var/obs data and annotations.
@@ -12,130 +50,9 @@ build an internal POJO for use by the rendering components.
*/
/*
Cherry pick from /api/v0.1 response format to make somethign similar
to the v0.2 schema, which we use for internal interfaces.
generate any client-side transformations or summarization that
is independent of REST API response formats.
*/
function RESTv01ResponseToSchema(response) {
/*
Annotation schemas in V02 (our target) look like:
annotations: {
obs: [
{ name: "name", type: "string" },
{ name: "num_reads", type: "int32" },
{
name: "clusters",
type: "categorical",
categories=[ 99, 1, "unknown cluster" ]
},
{ name: "QScore", type: "float32" }
],
var: [
{ "name": "name", "type": "string" },
{ "name": "gene", "type": "string" }
]
}
In V01, our source, it looks like:
"schema": {
"CellName": {
"displayname": "Name",
"include": true,
"type": "string",
"variabletype": "categorical"
},
"Cluster_2d": {
"displayname": "Cluster2d",
"include": true,
"type": "string",
"variabletype": "categorical"
},
"ERCC_reads": {
"displayname": "ERCC Reads",
"include": true,
"type": "int",
"variabletype": "continuous"
},
...
}
Mapping between the two assumes:
- V01 only has schema for observations
- CellName is mapped to 'name'
- type conversion: float->float32, int->int32, string->string
*/
return {
annotations: {
obs: _.map(response.data.schema, (val, key) => {
const name = key === "CellName" ? "name" : key;
let { type } = val;
if (type === "int") {
type = "int32";
}
if (type === "float") {
type = "float32";
}
return {
name,
type
};
}),
var: [{ name: "name", type: "string" }]
}
};
}
function RESTv01ResponseToVarAnnotations(response) {
/*
v0.1 initialize response contains 'genes' - names of all genes
in order.
*/
return _.map(response.data.genes, (g, i) => ({ __varIndex__: i, name: g }));
}
function RESTv01ResponseToObsAnnotations(response) {
/*
v0.1 format for metadata:
metadata: [ { key: val, key: val, ... }, ... ]
Target format is essentially the same, except the CellName key becomes name.
*/
return _.map(response.data.metadata, (c, i) => ({
__obsIndex__: i,
name: c.CellName,
...c
}));
}
function RESTv01ResponseToLayout(obsAnnotations, response) {
/*
v0.1 format for the graph is:
[ [ 'cellname', x, y ], [ 'cellname', x, y, ], ... ]
NOTE XXX: this code does not assume any particular array ordering in the V0.1
response. But for Universe initial load, the layout will be in the same
order as annotations, so this extra work isn't really necessary.
*/
const obsAnnotationsByName = _.keyBy(obsAnnotations, "name");
const { graph } = response.data;
const layout = {
X: new Float32Array(graph.length),
Y: new Float32Array(graph.length)
};
for (let i = 0; i < graph.length; i += 1) {
const [name, x, y] = graph[i];
const anno = obsAnnotationsByName[name];
const idx = anno.__obsIndex__;
layout.X[idx] = x;
layout.Y[idx] = y;
}
return layout;
}
function finalize(universe) {
/* A bit of sanity checking! */
const { nObs, nVar } = universe;
@@ -147,7 +64,14 @@ function finalize(universe) {
) {
throw new Error("Universe dimensionality mismatch - failed to load");
}
// TODO: add more sanity checks, such as:
// - all annotations in the schema
// - layout has supported number of dimensions
// - ...
/*
Create all derived (convenience) data structures.
*/
universe.obsNameToIndexMap = _.transform(
universe.obsAnnotations,
(acc, value, idx) => {
@@ -166,85 +90,135 @@ function finalize(universe) {
return universe;
}
function templateUniverse() {
/* default universe template */
const VarDataCacheLowWatermark = 32;
const VarDataCacheTTLMs = 1000;
function RESTv02AnnotationsResponseToInternal(response) {
/*
Source per the spec:
{
names: [
'tissue_type', 'sex', 'num_reads', 'clusters'
],
data: [
[ 0, 'lung', 'F', 39844, 99 ],
[ 1, 'heart', 'M', 83, 1 ],
[ 49, 'spleen', null, 2, "unknown cluster" ],
// [ obsOrVarIndex, value, value, value, value ],
// ...
]
}
return {
api: "0.1",
finalized: true, // XXX: may not be needed
nObs: 0,
nVar: 0,
schema: {},
/*
Annotations
*/
obsAnnotations: [] /* all obs annotations, by obs index */,
varAnnotations: [] /* all var annotations, by var index */,
obsNameToIndexMap: {} /* reverse map 'name' to index */,
varNameToIndexMap: {} /* reverse map 'name' to index */,
obsLayout: { X: [], Y: [] } /* xy layout */,
varDataCache: kvCache.create(
VarDataCacheLowWatermark,
VarDataCacheTTLMs
) /* cache of var data (expression) */
};
Internal (target) format:
[
{ __index__: 0, tissue_type: "lung", sex: "F", ... },
...
]
*/
const { names, data } = response;
const keys = ["__index__", ...names];
return _(data)
.map(obs => _.zipObject(keys, obs))
.sortBy("__index__")
.value();
}
export function createUniverseFromRESTv01Response(initResponse, cellsResponse) {
function RESTv02LayoutResponseToInternal(response) {
/*
build & return universe from a REST 0.1 /init and /cells response
*/
Source per the spec:
{
layout: {
ndims: 2,
coordinates: [
[ 0, 0.284483, 0.983744 ],
[ 1, 0.038844, 0.739444 ],
// [ obsOrVarIndex, X_coord, Y_coord ],
// ...
]
}
}
Target (internal) format:
{
X: Float32Array(numObs),
Y: Float32Array(numObs)
}
In the same order as obsAnnotations
*/
const { ndims, coordinates } = response.layout;
if (ndims !== 2) {
throw new Error("Unsupported layout dimensionality");
}
const layout = {
X: new Float32Array(coordinates.length),
Y: new Float32Array(coordinates.length)
};
for (let i = 0; i < coordinates.length; i += 1) {
const [idx, x, y] = coordinates[i];
layout.X[idx] = x;
layout.Y[idx] = y;
}
return layout;
}
export function createUniverseFromRestV02Response(
configResponse,
schemaResponse,
annotationsObsResponse,
annotationsVarResponse,
layoutObsResponse
) {
/*
build & return universe from a REST 0.2 /config, /schema and /annotations/obs response
*/
const { schema } = schemaResponse;
const universe = templateUniverse();
/* extract information from init OTA response */
universe.schema = RESTv01ResponseToSchema(initResponse);
universe.varAnnotations = RESTv01ResponseToVarAnnotations(initResponse);
universe.nVar = universe.varAnnotations.length;
/* constants */
universe.api = "0.2";
/* extract information fron cells REST json response */
/*
NOTE: this code *assumes* that cell order in data.metadata and data.graph
are the same. TODO: error checking.
*/
universe.obsAnnotations = RESTv01ResponseToObsAnnotations(cellsResponse);
universe.nObs = universe.obsAnnotations.length;
universe.obsLayout = RESTv01ResponseToLayout(
universe.obsAnnotations,
cellsResponse
/* schema related */
universe.schema = schema;
universe.nObs = schema.dataframe.nObs;
universe.nVar = schema.dataframe.nVar;
/* annotations */
universe.obsAnnotations = RESTv02AnnotationsResponseToInternal(
annotationsObsResponse
);
universe.varAnnotations = RESTv02AnnotationsResponseToInternal(
annotationsVarResponse
);
/* layout */
universe.obsLayout = RESTv02LayoutResponseToInternal(layoutObsResponse);
return finalize(universe);
}
export function convertExpressionRESTv01ToObject(universe, response) {
export function convertExpressionRESTv02ToObject(universe, response) {
/*
v0.1 ota looks like:
{
genes: [ "name1", "name2", ... ],
cells: [
{ cellname: 'cell1', e: [ 3, 4, n, x, y, ... ] },
...
]
}
/data/obs response looks like:
{
var: [ varIndices fetched ],
obs: [
[ obsIndex, evalue, ... ],
...
]
}
convert expression to a simple Float32Array, and return
[ [geneName, array], [geneName, array], ... ]
*/
convert expression toa simple Float32Array, and return
{ geneName: array, geneName: array, ... }
NOTE: geneName, not varIndex
*/
const vars = response.var;
const { obs } = response;
const result = {};
const { genes, cells } = response.data;
for (let idx = 0; idx < genes.length; idx += 1) {
const gene = genes[idx];
// XXX TODO: could this use _.unzip and have less code?
for (let varIdx = 0; varIdx < vars.length; varIdx += 1) {
const gene = universe.varAnnotations[vars[varIdx]].name;
const data = new Float32Array(universe.nObs);
for (let c = 0; c < cells.length; c += 1) {
const obsIndex = universe.obsNameToIndexMap[cells[c].cellname];
data[obsIndex] = cells[c].e[idx];
for (let obsIdx = 0; obsIdx < obs.length; obsIdx += 1) {
data[obsIdx] = obs[obsIdx][varIdx + 1];
}
result[gene] = data;
}
client/src/util/stateManager/world.js
+67 -52
View File
@@ -28,7 +28,7 @@ obs/cell.
NOTE: world.obsAnnotation should be identical to the old state.cells value,
EXCEPT that
* __cellIndex__ renamed to __obsIndex__
* __cellIndex__ renamed to __index__
* __x__ and __y__ are now in world.obsLayout
* __color__ and __colorRBG__ should be moved to controls reducer
@@ -46,47 +46,50 @@ obs/cell.
*/
/*
Summary information for each annotation, keyed by annotation name.
Value will be an object, containing either 'range' or 'options' object,
depending on the annotation schema type (categorical or continuous).
/* varDataCache config - see kvCache for semantics */
const VarDataCacheLowWatermark = 32; // cache element count
const VarDataCacheTTLMs = 1000; // min cache time in MS
Summarize for BOTH obs and var annotations. Result format:
{
obs: {
annotation_name: { ... },
...
},
var: {
annotation_name: { ... },
...
}
}
Example:
{
"Splice_sites_Annotated": {
"range": {
"min": 26,
"max": 1075869
}
},
"Selection": {
"options": {
"Astrocytes(HEPACAM)": 714,
"Endothelial(BSC)": 123,
"Oligodendrocytes(GC)": 294,
"Neurons(Thy1)": 685,
"Microglia(CD45)": 1108,
"Unpanned": 665
}
}
}
*/
function summarizeAnnotations(schema, obsAnnotations) {
/*
Build and return obs/var summary using any annotation in the schema
Summary information for each annotation, keyed by annotation name.
Value will be an object, containing either 'range' or 'options' object,
depending on the annotation schema type (categorical or continuous).
Summarize for BOTH obs and var annotations. Result format:
{
obs: {
annotation_name: { ... },
...
},
var: {
annotation_name: { ... },
...
}
}
Example:
{
"Splice_sites_Annotated": {
"range": {
"min": 26,
"max": 1075869
}
},
"Selection": {
"options": {
"Astrocytes(HEPACAM)": 714,
"Endothelial(BSC)": 123,
"Oligodendrocytes(GC)": 294,
"Neurons(Thy1)": 685,
"Microglia(CD45)": 1108,
"Unpanned": 665
}
}
}
*/
const obsSummary = _(schema.annotations.obs)
.keyBy("name")
@@ -115,7 +118,8 @@ function summarizeAnnotations(schema, obsAnnotations) {
})
.value();
const varSummary = {}; // TODO XXX - not currently used, so skip it
// TODO XXX - not currently used, so skip it
const varSummary = {};
return {
obs: obsSummary,
@@ -124,9 +128,6 @@ function summarizeAnnotations(schema, obsAnnotations) {
}
function templateWorld() {
const VarDataCacheLowWatermark = 32;
const VarDataCacheTTLMs = 1000;
return {
// map from universe obsIndex to world offset.
// Undefined / null indicates identity mapping.
@@ -186,6 +187,13 @@ export function createWorldFromEntireUniverse(universe) {
/* derived data & summaries */
world.summary = summarizeAnnotations(world.schema, world.obsAnnotations);
/* build the varDataCache */
world.varDataCache = kvCache.map(
universe.varDataCache,
val => subsetVarData(world, universe, val),
{ lowWatermark: VarDataCacheLowWatermark, minTTL: VarDataCacheTTLMs }
);
return world;
}
@@ -227,13 +235,21 @@ export function createWorldFromCurrentSelection(universe, world, crossfilter) {
// build index to our world offset
newWorld.worldObsIndex.fill(-1); // default - aka unused
for (let i = 0; i < newWorld.nObs; i += 1) {
newWorld.worldObsIndex[newWorld.obsAnnotations[i].__obsIndex__] = i;
newWorld.worldObsIndex[newWorld.obsAnnotations[i].__index__] = i;
}
/* derived data & summaries */
newWorld.summary = summarizeAnnotations(
newWorld.schema,
newWorld.obsAnnotations
);
/* build the varDataCache */
newWorld.varDataCache = kvCache.map(
universe.varDataCache,
val => subsetVarData(newWorld, universe, val),
{ lowWatermark: VarDataCacheLowWatermark, minTTL: VarDataCacheTTLMs }
);
return newWorld;
}
@@ -243,20 +259,19 @@ export function createWorldFromCurrentSelection(universe, world, crossfilter) {
*/
function deduceDimensionType(attributes, fieldName) {
let dimensionType;
if (attributes.type === "string") {
const { type } = attributes;
if (type === "string" || type === "categorical" || type === "boolean") {
dimensionType = "enum";
} else if (attributes.type === "int32") {
} else if (type === "int32") {
dimensionType = Int32Array;
} else if (attributes.type === "float32") {
} else if (type === "float32") {
dimensionType = Float32Array;
} else {
/*
Currently not supporting boolean and categorical types.
*/
console.error(
`Warning - REST API returned unknown metadata schema (${
attributes.type
}) for field ${fieldName}.`
`Warning - REST API returned unknown metadata schema (${type}) for field ${fieldName}.`
);
// skip it - we don't know what to do with this type
}
@@ -286,11 +301,11 @@ export function createObsDimensionMap(crossfilter, world) {
*/
const worldIndex = worldObsIndex ? idx => worldObsIndex[idx] : idx => idx;
dimensionMap.x = crossfilter.dimension(
r => obsLayout.X[worldIndex(r.__obsIndex__)],
r => obsLayout.X[worldIndex(r.__index__)],
Float32Array
);
dimensionMap.y = crossfilter.dimension(
r => obsLayout.Y[worldIndex(r.__obsIndex__)],
r => obsLayout.Y[worldIndex(r.__index__)],
Float32Array
);
@@ -309,7 +324,7 @@ export function subsetVarData(world, universe, varData) {
const newVarData = new Float32Array(world.nObs);
for (let i = 0; i < world.nObs; i += 1) {
newVarData[i] = varData[world.obsAnnotations[i].__obsIndex__];
newVarData[i] = varData[world.obsAnnotations[i].__index__];
}
return newVarData;
}