Commit Graph
49 Commits
Author SHA1 Message Date
Bruce Martin e770db1e2c load annotations incrementally (#1107)
* load annotations individually

* fix type check to be more general

* update node CI version from 10 to 12

* node 11

* debug print node version

* travis node version to latest

* try nvm

* remove extraneous node_js statement

* remove node version debugging printf

* incrementally load all annotations and layout

* process annotations and layout as they are loaded

* fix tests

* sort categories incrementally

* incrementally build category view summary; add category loading spinner

* add spinner to continuous metadata

* configure undoable reducer

* incremental crossfilter creation

* improve busy layout

* more layout cleanup

* correctly reconcile categories in schema

* refine layout of lsb spinners

* more spinner layout work

* more spinner layout

* always load layout before obs annotations
2020-02-10 11:22:27 -08:00
Matt Weiden ddbfc458d7 Remove d3 dependency in crossfilter (#1108)
* Remove d3 dependency in crossfilter

Fixes https://github.com/chanzuckerberg/cellxgene/issues/648

* Revert "Remove d3 dependency in crossfilter"

This reverts commit 03b478f7dc.

* Remove d3 dependency in crossfilter

Fixes https://github.com/chanzuckerberg/cellxgene/issues/648

* Fix typo in docstring
2020-01-14 13:46:53 -08:00
Matt Weiden 4c1ecf9e52 Specify histogram thresholds in fixed-length array (#1086)
* Specify histogram thresholds in fixed-length array

Fixes https://github.com/chanzuckerberg/cellxgene/issues/1082

For background, see the following:
* https://github.com/d3/d3-array/issues/46
* https://stackoverflow.com/questions/15880058/d3-js-ticks-function-giving-more-elements-than-needed

* Oops, off by one!
2020-01-06 11:13:02 -08:00
Matt Weiden b172ff7f15 Add margins to histogram SVGs (#1085)
* Add a left margin to histogram SVGs

Fixes https://github.com/chanzuckerberg/cellxgene/issues/1042

1. Added "margins" to the SVG as in https://bl.ocks.org/mbostock/3019563
2. Tailored the brushable area to be just where the data is and locked
   puppeteerUtils to use that in e2e testing.

* Dial in margins to avoid y-axis clipping
2020-01-06 10:46:06 -08:00
Colin Megill 4c3d783fc8 Continuous data in right sidebar (#985)
* Create right sidebar files, move histos over

* restructure continuous legend + all padding

* right sidebar file

* proper graph positioning and resize

* new lasso values since right sidebar

* fix autosave position
2019-11-05 20:29:21 -05:00
Bruce Martin a16c2e2747 crossfilter immutability error (#1009)
* add rename immutability test

* make rename immutable
2019-11-04 14:14:17 -08:00
Bruce Martin 3f2811d9da rendering performance improvements (#968)
* freeze objects

* component rendering perf work

* use PureComponent where safe

* remove obsolete WorldUtil code

* make brushable histogram a pure component
2019-10-04 07:59:22 -07:00
Bruce Martin 3660a6cc27 Experimental - manual annotations (#837)
* icons, partway

* redux for values

* onChange

* cancel

* annotations lifecycle for category names

* copy categorical

* edit category

* add Dataframe.withColsFrom

* render user annotations; default add/delete annotation category

* add label name to actions

* category name edit

* error checking improvements

* change schema field isUserAnnotation to writable

* always have an unassigned label; implement delete label

* implement add new label and edit label name

* label current cell selection

* fix select exact bug in crossfilter

* clean up categorical reducer

* fix tests

* remove debugging printf

* implement subset/reset for user annotations

* undo redo support for user annotations

* remove duplicate button from categories

* add modal

* remove obsolete duplicate annotation reducers

* remove old debugging printf

* connect modal to annotation create and dup

* initial full-stack wiring

* finish up end-to-end wiring

* fix existing unit tests

* fix pytests to match new schema API

* remove debugging printfs

* add label file rotation

* remove obsolete comment

* add fbs encode/decode tests

* add tests for writable annotations

* simplify code

* fix hashing bug with FBS encoding

* lint

* fix smoke tests

* improve error checking in Dataframe.withColsFrom

* add unit test for Dataframe.withColsFrom

* add unit test for Dataframe.columns and Dataframe.renameCol

* fix bug in FBS encode, add better error checks, refactor

* add FBS encode/decode test

* add clarifying comment

* clean up action type names; fix state inconsistency in crossfilter update

* change autosave timer to 2.5sec

* sort categorical metadata render order so it remains consistent

* add temporary autogenerated label for add-new-label operation

* fix hover-over label menu interference with cell highlighting

* remove debugging code

* add missing reducer cases & fix typo

* make dataframe memoize more general purpose

* add dev mode for annos

* fix error on select duplicate

* handle zero occupancy categories

* correctly maintain unclipped AND clipped world

* correctly handle zero length FBS matrix and label files

* ensure all writable categorical schema contains an unassigned category

* handle case where building occupancy stack for category with no members

* dialog for creating label, disable button if duplicate or empty

* visually separate writeable

* edit category

* fix edit category name

* remove debugging code

* fix edit annotation label

* visually define unassigned, change options

* Pull in requirements.txt from `master`

* label currently selected cells

* duplicate label

* lint

* fix pytest merge issues

* rename --label-file to --experimental-label-file

* remove debugging console log

* spelling error fix; fix bug found in PR review.

* lint
2019-09-18 07:33:41 -04:00
Bruce Martin 0f36b49840 Graph/scatterplot improvements (#893)
* many graph/scatterplot fixes: 722, 882, 758, 886

* fix e2e test breakage caused by graph transform work

* convert webgl point shaders to use flags rather than explicit point size

* remove unused packages from dependencies

* fix typo in regl prop name

* factor common code into util helper file

* legibility tweaks
2019-08-22 20:56:05 -07:00
Bruce Martin 906d65c06f fix binning error in continuous data histogram (#869) 2019-08-05 10:40:56 -07:00
Severiano Badajoz 941c297363 categorical vs continuous mini histograms (#827)
* comment

* add histogram functionality to Dataframe; port category occupancy to use it

* fix binning and create histogram for continous by catagorical

* Remove unnecessary logs

* Begin work on KDE

* Replace broken KDE with working histogram

* Define domain and range based on data from histogram

* Fix occupancy

* Add continuous obs and switch to canvas

* Stop value from always rerendering

* clear before render

* Clear canvas on render

* refactor categorical occupancy to canvas

* Remove log

* simplify finding max

* refactor kde->histogram and occupancy->bins

* refactor svg -> canvas

* rename to occupancy stack

* create popup

* add metadata and categorical values to popup

* fix overflow

* remove zeros info

* style graph

* fix shouldComponentUpdate to look for world changes

* change categorySelected -> categoryValueSelected

* refactor out render

* remove comment

* conditionally have bottom border

* remove diff comp

* remove comments

* remove unnecessary mapping

* Add comments describing drawing functions

* comments

* flip comparison order

* remove logging

* move default to parameter

* move defaults to parameter

* disable popover if not showing histogram

* fix wording and styling

* add line break
2019-07-09 11:19:01 -07:00
Colin Megill 6aeefb0fe6 menubar (#804)
* menubar 1

* zoom switching

* centering, pixel perfect canvas

* remove dead args and code

* clipping

* remove log

* if

* connect props

* lint

* undo

* logo left, componetize

* graph back to full height

* shadow to top

* do not prematurely call event handlers during render

* change test to deal with async histogram creation

* left section padding

* lint

* adjust graph to account for top bar,

* lasso tests

* refine histogram tests

* remove testing (onlys)
2019-06-13 15:15:43 -07:00
Charlotte Weaver 1d6bb032a3 remove async from describe blocks (#805) 2019-06-03 14:59:16 -07:00
Bruce Martin a6142bdf93 improve graph scale and centering (#796)
* add gutter to embedding canvas

* improve layout scale and translate

* fix lint

* pin tables to version 3.5.1

* fix lasso coordinate smoke tests
2019-05-30 13:31:41 -07:00
Bruce Martin 3dc45d6330 do not hard-wire column names in annotations (#785)
* enforce column name uniqueness for obs and var

* parameterize the column name containing obs and var user-readable names

* use the new annotation index value from schema

* update f/e unit tests

* PR review suggestions

* lint
2019-05-24 21:00:54 -07:00
Bruce Martin e941c1a496 scaling omitted from event handlers (#789)
* scaling omitted from event handlers

* fix smoke tests
2019-05-24 07:00:04 -07:00
Colin Megill a657eb3152 Logo (#782)
* logo, black

* fixes

* remove template, move header
2019-05-23 11:47:24 -04:00
Bruce Martin 2357d0c1b8 layout change UI (#776)
* add layout to schema

* add layout choice action and reducer

* multi layout UI

* update unit tests

* add missing file

* update test schema

* fix duplicate test id

* fix tabs

* PR lint

* fix pytest
2019-05-22 13:21:33 -07:00
Bruce Martin fcc05f6a00 coordinate system fixes for embedded graph (#768)
* change pan speed to 1 per issue #722

* correct handle scaling of graph when aspect ratio less than one

* add package lock

* add invert to our scale functions

* correctly transform to/from gl coordinates

* remove unused import

* fix naming of import

* update smoke tests
2019-05-20 14:22:41 -07:00
Charlotte Weaver 2d4e827bea wait for element before getting text/html (#777) 2019-05-20 11:35:53 -07:00
Bruce Martin efa1709158 add multi-layout support to back-end (#766)
* add multi-layout support to back-end

* remove obsolete code

* temporary code to apply heuristic choice of default layout

* fix tests

* update python tests

* more py lint

* PR review changes

* more PR lint

* PR lint
2019-05-16 14:49:22 -07:00
Charlotte Weaver d522cc8f91 Add clipping test to smoke tests (#757)
* Add clipping test to smoke tests

* devtools on in debug
2019-05-09 15:31:55 -07:00
Bruce Martin 846b8d15bd lodash cleanup (#747)
* add own range() function

* lodash cleanup

* remove redundant fill range implementations

* remove use of _.get

* sync test babel config with build

* update tests to match new range implementation
2019-05-06 20:28:32 -04:00
Sidney Bell a08e19bbd0 Clip continuous values based on percentile cutoffs (#672)
* Add numeric inputs for percentiles

* Define initial values for percentile cutoffs in world reducer

* add percentil to crossfilter dimensions

* worldEqUniverse now handles cloned worlds

* add Dataframe.mapColumns

* Wire up handlers for percentile inputs

* World reducer and stateManager know about continuousPercentileMin/Max

* Create world as universe clone (not pointer) to avoid clobbering vals

* Define basic actions for setting continuousPercentileMin/Max

* Under the hood, deal with percentiles between 0 and 1

* Move percentile inputs to visualization settings menu

* Fix padding for undo/redo buttons

* Trigger world rebuild from percentile actions

* BROKEN - pseudocode for clamping dataframe by percentiles upon world rebuild

* fix error handling on clip quantiles; start world clipping implementation

* more unclipped reorg

* rename crossfilter.percentile to quantile

* simplify schema access

* update continuous legend when scale changes

* update color cache when clip changes

* clip obs annotations and var data when clip quantile changes

* use own fromEntries

* fix tests

* stable non-finite float sort/search

* clarify comments

* fix syntax typo

* use new stand-alone clip

* clip expresssion data

* add select tests for non-finite scalars

* basic styles

* clip UI now requires explicit commit

* reset enable/disable accounts for clip percentiles

* better error messages

* fix bug in undo interaction with programatic min brush selection

* small refactoring

* support clipping of int data

* do not perform unnecessary summarizations

* improve caching of dataframe compiled columns

* add percentile precompute to Dataframe.summarize

* use Dataframe.summarize for clip percentiles

* remove obsolete quantile code from corssfilter

* histogram scale and label Y axis, add unclipped X range labels

* layout tweaks

* scatterplot now updates when clip changes

* improve comments

* remove debugging comment

* rework clip number entry validation for usability

* ui tweaks to histogram colors and layout

* enable undo/redo for clip user action

* refine UI on clip value entry

* api cleanup

* update confusing comment

* clarify purpose of isValidDigitKeyEvent

* fix misleading comment

* apply appropriate button-group classes; do not mix span and div

* variable name and comment changes suggested in PR review

* rename sort to sortArray; remove unused and dead code path

* naming changes suggested in PR review

* code review improvements for clarity

* more small changes from PR review

* lint fixes for PR review

* fix spelling error

* clarify that function performs in-place modification of world

* add comment to clarify intent of range operation

* fix bad indents in comments

* clean up __columnsAccessor comments and code

* improve comments around clipPredicate

* field name consistency

* improve comment on quantiles params
2019-04-30 16:20:10 -07:00
Charlotte Weaver d581a0d460 color by gene smoketest (#707)
* add test for color by gene expression

gene expression and metadata color by are handled differently

* error on console.error

not just on thrown errors
2019-04-11 17:06:36 -07:00
Bruce Martin 7275d9d4dc Graph selection state management and history bug fixes (#679)
* save graph selection in redux state

* fix old graph brush select regressions

* refactor graph brush selection to work with undo/redo

* update tests to match new crossfilter spatial select API

* graph selection state now in redux

* remove dead code

* sync graph selection with redux state; improvements to undoable machinery

* fix regression in undoable

* differentiate graph selection cancel from deselect action

* simplify calculation

* remove debugging code

* fix responsive repaint bug in graph selection tool

* undoable debugging and code cleanliness

* undoable action filter state now merges, rather than replaces

* improve comments

* add debounce to undoable action filter; improve comments and debug sanity check code

* comments

* fix undoable bug with clear scatterplot actions

* disable undoable debug flag

* cleanup API and comments around statemachine

* add test id attribute to lasso

* add better error handling for gene fetch requests
2019-04-08 15:53:12 -07:00
Charlotte Weaver c9a8e3ea42 Add manual UX tests (#699) 2019-04-08 13:23:17 -07:00
Charlotte Weaver 5d4b11d287 Better smoke tests (#669) 2019-03-28 16:47:42 -07:00
Bruce Martin 7ef5203564 Undo/redo (#659)
* immutable crossfilter

* first cut at reducer refactor with cascade model

* add initial redo/undo implementation

* small optimization

* integrate expression with history

* add tests for new reducers and fix a couple of small initialization bugs

* treat tiny lasso selections as a clear

* better function name for clarity

* fix undo for differential expression

* remove logging

* fix regression due to bad merge

* cleanup and comments for clarity

* improve undoable configuration for flexibility

* fix stale comments

* remove debugging code from production build

* rename categoricalSelectionState

* rename file

* improve comments
2019-03-22 14:53:48 -07:00
Bruce Martin 996b06cecc refactoring - immutable crossfilter (#647)
* immutable crossfilter

* PR review changes
2019-03-20 14:32:35 -07:00
Charlotte Weaver f96fd36ecb Update test ui names (#638)
* Fixed changed testid

* Added debug mode for running tests
2019-03-12 10:17:09 -07:00
Bruce Martin ffd6273419 Dataframe, part deux - add varData and summarize() (#608)
* initial dataframe commit

* initial dataframe port of core app

* rename variables for clarity

* remove unused import

* comment out unused code

* fix array handling bug in crossfilter dimension creation

* allow creation of empty dataframes

* handle non-existent columns

* handle non-existent columns

* revise tests for new dataframe

* comments for clarity

* comments for clarity

* generate bulk add placeholder with real gene names

* fix bug in gene name adding

* more dataframe unit tests

* fix bug - subset from current world, not universe

* put cut and pasted code into a single function

* improve caching of crossfilter

* remove cascading update bug from graph

* more performance work

* improve state handling for scatterplot

* performance optimization of critical path

* add column summarization

* dataframe utils

* add callOnceLazy

* fix tests

* minor updates found during review

* fix misspelling

* remove RESTv02 from function names

* comment cleanup

* cut/icut col parameter defaults to null

* break up large test

* improve tests and comments on dataframe at/has functions

* add Dataframe withCol/dropCol

* expression varData now stored in a dataframe

* dead code cleanup

* use dataframe.summarize()

* test cases for Dataframe.col.summarize

* update test cases for new dataframe summarize

* improve naming

* use new hasCol API

* add comments

* add more Dataframe.withCol tests

* add ability to specify row index in cut operation

* retire subsetVarData function

* correctly handle expression subsetting

* lint and improve comments

* rename cut to subset

* changes based on PR review
2019-02-28 08:34:22 -08:00
Charlotte Weaver 2bae696986 Smoke tests (#604)
smoke tests
2019-02-27 15:46:58 -08:00
Bruce Martin 6b33315cbe Dataframe (#576)
* initial dataframe commit

* initial dataframe port of core app

* rename variables for clarity

* remove unused import

* comment out unused code

* fix array handling bug in crossfilter dimension creation

* allow creation of empty dataframes

* handle non-existent columns

* handle non-existent columns

* revise tests for new dataframe

* comments for clarity

* comments for clarity

* generate bulk add placeholder with real gene names

* fix bug in gene name adding

* more dataframe unit tests

* fix bug - subset from current world, not universe

* put cut and pasted code into a single function

* improve caching of crossfilter

* remove cascading update bug from graph

* more performance work

* improve state handling for scatterplot

* performance optimization of critical path

* add column summarization

* dataframe utils

* add callOnceLazy

* fix tests

* minor updates found during review

* fix misspelling

* remove RESTv02 from function names

* comment cleanup

* cut/icut col parameter defaults to null

* break up large test

* improve tests and comments on dataframe at/has functions
2019-02-22 11:31:34 -08:00
Colin Megill dbb3a309a9 Lasso (#586)
* lasso working

* break out invert into own function

* action

* add spatial dimension to crossfilter, in support of polygon lasso

* improve comments on new dimension API

* lasso vs zoom
2019-02-08 11:48:51 -08:00
Bruce Martin b90447c387 binary wire format with flatbuffers (#509)
* first flatbuffer schema

* do not lint auto-generated files

* add flatbuffers package

* add flatbuffer module

* wire up /data/X/T route

* use flatbuffers for matrix data fetc

* clarity and comments

* add flatbuffer layout route

* clean up obsolete code

* fix tests

* move flake8 config to setup.cfg

* add comments

* lint

* rework layout routes for fbs

* add more type support to fbs

* lint

* add flatbuffer support for annotations

* function name improvements

* fix botched merge with master

* remove unused import

* route cleanup for flatbuffers

* rename function for clarity

* add missing globals to Jest tests

* fix client JS tests

* fix routes for Python tests

* comments for clarity

* non-finite floating point hardening

* more non-finite number handling

* lint

* fix tests for summarizeAnnotations

* harden diffexp calculation against FP errors

* cleanup unused code

* lint

* add encoding tests for flatbuffers

* application type specified as strings

* fix spelling error

* improve variable names

* add note about documentation gap

* rename FBS DataFrame to Matrix
2019-01-09 14:26:05 -08:00
Bruce Martin 296ed752fa Improved summary counts of annotation values (#478)
* convert annotation summary to a Map

* add 2d annotation count summary

* add memoization on 2D annotation counting

* add tests for annotation summarization

* fix import/exports

* rename WorldOps to WorldUtil

* rename WorldOps to WorldUtil

* add comment
2018-12-03 09:15:03 -08:00
Bruce Martin 9f7e1f1cb0 use quicksort instead of default system sort (#413) 2018-11-07 13:35:00 -08:00
Bruce Martin 2d5bc9d0c9 Performance work, plus fix #405 (#406)
* remove memoization

* update to flash 1.0.2; turn on threading

* stand-alone helper routines for array slicing

* fix issue #405

* reset diffexp state when world changes

* performance work in dimension creation; fix world slicing bug

* update tests to match new state mgmt api

* update flask

* do not make dimensions for useless annotations

* update test to match optimizations
2018-11-05 13:57:58 -08:00
Colin Megill d221d76be6 test for world, graph border, legend label (#378)
* test for world, graph border, legend label

* typo
2018-10-25 12:16:55 -04:00
Bruce Martin 0c271bc815 categorical metadata performance work (#356)
* do not create unecessary option state for categories we will not display

* performance improvements for categorical metadata handling
2018-10-22 09:56:41 -07:00
Bruce Martin 2616b5c185 finish tests for world (#340)
* Create Release_Validation_Recipe.md

* nameCreators test and better error handling

* add test for createObsDimensionMap

* add last set of tests for world module
2018-10-18 09:54:57 -07:00
Bruce Martin 76ec29734a Performance work (#334)
* range encode filter range lists

* speed up data load

* add comment on scanpy read params

* update to latest scanpy/anndata

* performance improvments in data loading

* fix typo

* work around scanpy bug

* remove debugging print statements
2018-10-16 15:49:40 -07:00
Colin Megill 65f5244ff7 add name getter to world test 2018-10-15 11:20:52 -07:00
Bruce Martin b77d7b14b5 bkmartinjr/world tests (#306)
First cut at test for stateManager/world. Test data for REST responses factored out into separate file for shared use by Universe and World tests. Also cleaned up some debugging code.
2018-10-10 18:05:19 -07:00
Bruce Martin 19f2728a18 Bkmartinjr/state mgr tests (#298)
Jest tests for part of util/stateManager

* reorganize test directories

* lint

* add Jest config to eslint

* add Jest config to eslint

* lint

* fix babel configuration for tests

* start at kvCache tests

* reorganize test directories

* lint

* add Jest config to eslint

* add Jest config to eslint

* lint

* fix babel configuration for tests

* start at kvCache tests

* fix cache flush bug manifest by new test cases

* new kvCache tests

* skeletal universe test

* fix race in test case

* more universe test cases
2018-10-04 17:41:26 -07:00
Bruce Martin f0d9d873be Fix jest test suites (#268)
* CRLF to LF

* add babel-jest

* use ES6 style imports

* fix bugs manifest by test suite

* add babel test config
2018-09-27 17:52:17 -07:00
bkmartinjr f839653c00 add Jest tests for new crossfilter group functions 2018-08-04 13:04:09 -07:00
Charlotte Weaver 85506db971 Updating to latest cellxgene client code
Fixes the build error in the old version. 

Reverted the API version since we changed directions from updating the API to refactoring the server structure instead.
2018-07-05 13:37:26 -07:00