* Add user-defined category-label colors
Fixes https://github.com/chanzuckerberg/cellxgene/issues/1152
As described in https://github.com/chanzuckerberg/cellxgene/issues/1307
* Respond to feedback from @bkmartinjr in nodejs
* Respond to feedback from @bkmartinjr in python
* Add tests to the server module
* Autoformat python, run linter
* Make colors_get error handling specific
* Respond to feedback from @bkmartinjr
* Respond to feedback from @bkmartinjr
* Fix whitespace
* Fix python lint errrors
* Update documentation
* Add --disable-user-colors option to launch and cxgtool.py
* Fix python formatting
* Rename '--disable-user-colors' to '--disable-custom-colors'
* first cut at re-embedding route and back-end support
* update and expand config route tests
* add scanpy_umap
* add reembedding to config route parameters
* front-end support for reembedding fetch and UI
* remove unused imports
* add loading state
* save reembedding in reducer state
* improve withColsFrom
* transmit reembed schema to client; pick unique embedding names
* display embeddings
* format
* lint
* spaces, tab size 2
* lint
* test hack for smoke-test race
* back out hack sleep
* add check for backed mode
* add unit test for reembedding
* lint
* hide re-embedding CLI param from help
* Remove unused variables and imports
* Simplify conditional
* Fix typo
* Do not overprune var data cache
There is a bug in how the universe and world gene sets are constructed
and passed to `ControlsHelpers.pruneVarDataCache` that causes the var
data cache to be over-pruned. This commit fixes the issue.
Consider the following example from the node console:
```
❯ node
Welcome to Node.js v13.5.0.
Type ".help" for more information.
> new Set([1], [2], [3])
Set(1) { 1 }
```
What we really want is the set `Set(3) { 1, 2, 3 }`, which can be
constructed as:
```
> new Set([].concat([1], [2], [3]))
Set(3) { 1, 2, 3 }
```
* Undo selection appends diffExp genes to user gene list
Fixes https://github.com/chanzuckerberg/cellxgene/issues/1171
Need: When a user performs a differential expression from within
a sub-selection (world) of the data and then resets the selection to all
cells (universe), the differential expression results are no longer
valid.
Approach:
* When the selection is reset, move the top (maxUserDefinedGenes
- len(userDefinedGenes) from the differential expression results to the
list of user defined genes
* Raise maxUserDefinedGenes to 25 to give users more room and
accommodate the extra genes transferred in from differential expression
Other commits:
* Choose different button icons
* Add diff exp genes to user defined genes on subset too
* Respond to feedback from @liaprins-czi and @bkmartinjr
* add simple error message helper
* port all label name pickers to use the new LabelInput component
* use pure components where possible
* cleanup
* more cleanup
* lint
* change new label prompt
* load annotations individually
* fix type check to be more general
* update node CI version from 10 to 12
* node 11
* debug print node version
* travis node version to latest
* try nvm
* remove extraneous node_js statement
* remove node version debugging printf
* incrementally load all annotations and layout
* process annotations and layout as they are loaded
* fix tests
* sort categories incrementally
* incrementally build category view summary; add category loading spinner
* add spinner to continuous metadata
* configure undoable reducer
* incremental crossfilter creation
* improve busy layout
* more layout cleanup
* correctly reconcile categories in schema
* refine layout of lsb spinners
* more spinner layout work
* more spinner layout
* always load layout before obs annotations
* refactor reducer to no longer support hover state and hold many labels
* refactor to generate centroidCoordinates for all values of a category
* create hash for function and memoize export
* create button to display all labels for a category
* clear state
* create label for each thing
* calculate on each value
* change to in place modification of map
* switch to for loop with iterator instead of forEach
* use map from centroidLabel instead of creating copy
* adapt for map
* utilize tarrays
* begin documentation
* disable centroids if in zoom mode
* clean up
* persist uncalc coordinates
* document
* clean up and document
* cleanup and document
* fix
* fix undefined labels and document changes
* fix first element skip
* fix conditional recalc
* rename centroidLabel -> centroidLabels
* break out dilation on hover to new reducer
* numerous styling changes for readability
* change centroid icon
* remove colorAccessor from parameters
* recalc centroids on world change
* make label toggle undoable
* remove unused import
* highlight labels on hover
* remove special characters from svg id
* lighten backdrop
* only generate new centroids if they pre-exist
* fix issue with spaces in catagorical value name
* add label buttons to menubar
* change reducer to use colorAccessor and have single toggle
* fix check to see if svg should be rendered
* move svg overlays onto a single svg layer
* dilate on label hover
* remove logs
* allow centroid to update along side regl renders
* allow actions to pass through svg if in zoom mode
* remove artifact from circle
* remove comment
* remove disabling of centroid button
* fix conditional map to screen
* make styling label conditions stricter
* prettier
* refactor onto master
* refactor computePointFlags() to use pointDilation store
* notify when viewport changes
* move svg attributes out of lasso setup and prevent rerenders/writes
* begin playing with transform matrix
* first solution for camera interaction
* create transform using nested groups
* semi-working method using nested groups with transforms
* inversely scale text
* properly do final transform
* cleanup dead / test code
* reinstate original functionality
* breakout centroid labels labels into separate component
* default toggle on for testing
* separate lasso and centroid layers
* remove unnecessary attributes, working hover
* dilation on label hover
* fix dilation on scatterplot
* add dilation on label hover
* break overlay into separate component
* make overlay agnostic to children
* move label mouse actions to centroidlabels component, add overlay state
* remove lasso on switch to camera
* disallow user selection
* fix reducer
* fix subset with continuous color error
* reset labels on color by continuous
* revert centroids on by default
* refactor for nested restructuring
* remove update checking
* remove unused method
* readd deleted hover delay
* remove old centroid setup
* remove centroid from undoable
* cleanup dead code
* remove dead code
* rollback unnecessary changes
* begin adding annotation functionality
* add annotation functionality
* add reset and undo functionality
* change centroids on layout change
* don't create label for unassigned
* add comment pointing out POI for performance
* touch up matrix transform comment
* add comment explaining coordinate space and children's assumed space
* remove dead code
* switch to pure component
* connect centroidLabels to redux
* clean up camera check and null result
* tool tip change
* rename centroid toggle and the like
* fix the misalignment of buttons, also make blueprint use consistent
* fix comment spelling mistakes
* introduce variable for cleaner logic expressions and state assignment
* alter tooltip text to back color by interaction
* remove manual iterator manipulation for forEach()
* remove debounce
* nit fix
* tooltip wording fix
* lint
* rename config param label-file
* annotations rework - CLI params, file naming and backups
* lint
* improve cli option error checks
* enable session cookies
* enable session cookies
* add session id
* name annotations file in multi-dataset and multi-user safe manner
* pass data user hash to front-end
* add annotation collection name support to front-end
* add constant for annotation data collection name
* parameterize annotation collection name; make it sticky in the session
* clarify comments
* hard wire a temporary data collection name for testing
* prettier
* test comment
* package command
* set annotations filename dialog
* name and hash are visible
* wire up data collection capture
* enforce column name uniqueness for obs and var
* parameterize the column name containing obs and var user-readable names
* use the new annotation index value from schema
* update f/e unit tests
* PR review suggestions
* lint
* save graph selection in redux state
* fix old graph brush select regressions
* refactor graph brush selection to work with undo/redo
* update tests to match new crossfilter spatial select API
* graph selection state now in redux
* remove dead code
* sync graph selection with redux state; improvements to undoable machinery
* fix regression in undoable
* differentiate graph selection cancel from deselect action
* simplify calculation
* remove debugging code
* fix responsive repaint bug in graph selection tool
* undoable debugging and code cleanliness
* undoable action filter state now merges, rather than replaces
* improve comments
* add debounce to undoable action filter; improve comments and debug sanity check code
* comments
* fix undoable bug with clear scatterplot actions
* disable undoable debug flag
* cleanup API and comments around statemachine
* add test id attribute to lasso
* add better error handling for gene fetch requests
* immutable crossfilter
* first cut at reducer refactor with cascade model
* add initial redo/undo implementation
* small optimization
* integrate expression with history
* add tests for new reducers and fix a couple of small initialization bugs
* treat tiny lasso selections as a clear
* better function name for clarity
* fix undo for differential expression
* remove logging
* fix regression due to bad merge
* cleanup and comments for clarity
* improve undoable configuration for flexibility
* fix stale comments
* remove debugging code from production build
* rename categoricalSelectionState
* rename file
* improve comments
* initial dataframe commit
* initial dataframe port of core app
* rename variables for clarity
* remove unused import
* comment out unused code
* fix array handling bug in crossfilter dimension creation
* allow creation of empty dataframes
* handle non-existent columns
* handle non-existent columns
* revise tests for new dataframe
* comments for clarity
* comments for clarity
* generate bulk add placeholder with real gene names
* fix bug in gene name adding
* more dataframe unit tests
* fix bug - subset from current world, not universe
* put cut and pasted code into a single function
* improve caching of crossfilter
* remove cascading update bug from graph
* more performance work
* improve state handling for scatterplot
* performance optimization of critical path
* add column summarization
* dataframe utils
* add callOnceLazy
* fix tests
* minor updates found during review
* fix misspelling
* remove RESTv02 from function names
* comment cleanup
* cut/icut col parameter defaults to null
* break up large test
* improve tests and comments on dataframe at/has functions
* add Dataframe withCol/dropCol
* expression varData now stored in a dataframe
* dead code cleanup
* use dataframe.summarize()
* test cases for Dataframe.col.summarize
* update test cases for new dataframe summarize
* improve naming
* use new hasCol API
* add comments
* add more Dataframe.withCol tests
* add ability to specify row index in cut operation
* retire subsetVarData function
* correctly handle expression subsetting
* lint and improve comments
* rename cut to subset
* changes based on PR review
* initial dataframe commit
* initial dataframe port of core app
* rename variables for clarity
* remove unused import
* comment out unused code
* fix array handling bug in crossfilter dimension creation
* allow creation of empty dataframes
* handle non-existent columns
* handle non-existent columns
* revise tests for new dataframe
* comments for clarity
* comments for clarity
* generate bulk add placeholder with real gene names
* fix bug in gene name adding
* more dataframe unit tests
* fix bug - subset from current world, not universe
* put cut and pasted code into a single function
* improve caching of crossfilter
* remove cascading update bug from graph
* more performance work
* improve state handling for scatterplot
* performance optimization of critical path
* add column summarization
* dataframe utils
* add callOnceLazy
* fix tests
* minor updates found during review
* fix misspelling
* remove RESTv02 from function names
* comment cleanup
* cut/icut col parameter defaults to null
* break up large test
* improve tests and comments on dataframe at/has functions
* first flatbuffer schema
* do not lint auto-generated files
* add flatbuffers package
* add flatbuffer module
* wire up /data/X/T route
* use flatbuffers for matrix data fetc
* clarity and comments
* add flatbuffer layout route
* clean up obsolete code
* fix tests
* move flake8 config to setup.cfg
* add comments
* lint
* rework layout routes for fbs
* add more type support to fbs
* lint
* add flatbuffer support for annotations
* function name improvements
* fix botched merge with master
* remove unused import
* route cleanup for flatbuffers
* rename function for clarity
* add missing globals to Jest tests
* fix client JS tests
* fix routes for Python tests
* comments for clarity
* non-finite floating point hardening
* more non-finite number handling
* lint
* fix tests for summarizeAnnotations
* harden diffexp calculation against FP errors
* cleanup unused code
* lint
* add encoding tests for flatbuffers
* application type specified as strings
* fix spelling error
* improve variable names
* add note about documentation gap
* rename FBS DataFrame to Matrix
* only load annotation var names
* remove incorrect usage of var annotation data
* temporary workaround for issue #480
* lint
* issue warnings only once per item
* new diffexp REST API spec
* new diffexp REST API; faster diffexp and dataframe slicing
* first draft of fast diffexp
* convert variance calculation to two-pass method
* lint
* update front-end use of API
* fix typo in spec
* disable content compression
* catch index filter format errors
* clean up of dead code
* resolve PR review comments
* remove memoization
* update to flash 1.0.2; turn on threading
* stand-alone helper routines for array slicing
* fix issue #405
* reset diffexp state when world changes
* performance work in dimension creation; fix world slicing bug
* update tests to match new state mgmt api
* update flask
* do not make dimensions for useless annotations
* update test to match optimizations
* range encode filter range lists
* speed up data load
* add comment on scanpy read params
* update to latest scanpy/anndata
* performance improvments in data loading
* fix typo
* work around scanpy bug
* remove debugging print statements
Move to new REST v0.2 communication between front and back-end. This is a first cut implementation which is functional, but will need follow-up enhancements for performance, error checking, etc. Protocol spec is in docs directory.
* Add filtering via indexing
* Using new filter specs
Indexing working
* Added filtering by annotation value
* factor out common methods
* Documentation
* create enum for axis (obs/var)
* Better description for filter's return
* Add boolean to enumerated types
* Augmented enum for scanpy axis
* Create schema for annotations
Based on datatype within scanpy/anndata
+ tests
* remove obsolete schema parse script
* Update rest api to remove old routes and add schema route
* Separate development requirements
* Warning for unsupported datatypes
* include -r requirements.txt in dev
* Merged downcast warnings
* Fixed bug where names were NaNs
Needed to include the index too when creating the series
* Add config endpoint
* Generate app features from CLI selections
* Move features to driver
* Add tests for schema
* Clearer version wording
* python3 version of super
* version from engine to package level
* move features to driver
* Revise layout function to match the new spec
* GET for layout/obs
* PUT Layout (#211)
* PUT Layout
* Csweaver/annotations (#212)
* Update scanpy engine to support the rest v0.2 annotation requests
* GET endpoint for obs annotations + tests
* Documentation
* Test annotations in scanpy engine
* Description for annotation-keys param
* annotation->annotations
* clarified return for annotations
* Use URL query list for annotations fields
* parse_filter parses v0.2 GET filters (#215)
* parse_filter parses v0.2 GET filters
* Don't allow index filters from query params
* Better variable conversion
* Parse filter improvements
- uses default dict
- renamed filter -> query_filter
* Cleanup Tasks (#216)
* Add test_api back into travis build
* Do custom JSON encoding the correct way
* Run cellxgene server in test setup
* Cleanup new tests too
* Option to bind to all interfaces (#225)
app.run("0.0.0.0") instead of app.run("127.0.0.1") binds to all interfaces.
Note: There are comments on the internet that says that the flask server is not up to the task of production serving. I don't think that such scalability concerns apply here, but I was able to get cellxgene working with twistd relatively easily, and we could switch to that if there are scalability concerns.
Test plan: browsed to <ip>:5005/api/v0.2/config on a different host.
* Add filtering via indexing
* Using new filter specs
Indexing working
* Added filtering by annotation value
* factor out common methods
* Documentation
* create enum for axis (obs/var)
* Better description for filter's return
* Add boolean to enumerated types
* Augmented enum for scanpy axis
* Create schema for annotations
Based on datatype within scanpy/anndata
+ tests
* remove obsolete schema parse script
* Update rest api to remove old routes and add schema route
* Separate development requirements
* Warning for unsupported datatypes
* include -r requirements.txt in dev
* Merged downcast warnings
* Fixed bug where names were NaNs
Needed to include the index too when creating the series
* Add config endpoint
* Generate app features from CLI selections
* Move features to driver
* Add tests for schema
* Clearer version wording
* python3 version of super
* version from engine to package level
* move features to driver
* Revise layout function to match the new spec
* GET for layout/obs
* PUT Layout (#211)
* PUT Layout
* Csweaver/annotations (#212)
* Update scanpy engine to support the rest v0.2 annotation requests
* GET endpoint for obs annotations + tests
* Documentation
* Test annotations in scanpy engine
* Description for annotation-keys param
* annotation->annotations
* clarified return for annotations
* Use URL query list for annotations fields
* parse_filter parses v0.2 GET filters (#215)
* parse_filter parses v0.2 GET filters
* Don't allow index filters from query params
* Better variable conversion
* Parse filter improvements
- uses default dict
- renamed filter -> query_filter
* Cleanup Tasks (#216)
* Add test_api back into travis build
* Do custom JSON encoding the correct way
* Run cellxgene server in test setup
* Cleanup new tests too
* Option to bind to all interfaces (#225)
app.run("0.0.0.0") instead of app.run("127.0.0.1") binds to all interfaces.
Note: There are comments on the internet that says that the flask server is not up to the task of production serving. I don't think that such scalability concerns apply here, but I was able to get cellxgene working with twistd relatively easily, and we could switch to that if there are scalability concerns.
Test plan: browsed to <ip>:5005/api/v0.2/config on a different host.
* Fix merge errors
- import warnings was improperly deleted
- scanpy engine tests were totally wrong
* Fix merge error with driver
* PUT /annotations (#235)
* Add query param for annotation name
* fix descriptions, eliminate else clause
* first cut at initial data load on rest 0.2 api
* Annotation var (#248)
* Fix bug strings are always objects in pandas
* Add axis to annotation method
* Add /annotation/var to REST api
* Csweaver/expressiondata (#242)
* Refactor expression method for REST v2
* Add message to QueryStringError
* Fix range filters
* Add GET route for /data
* /data PUT route
* rename expression to data_frame
* clarification of error
* Improve accept type handling
* support all schema types for 0.2 REST API
* remove REST 0.1 code; connect var annotations loading
* config reducer; use config to set data set title; remove obsolete templating code for data set title
* REST 0.2 expression conversion support
* partial port of expression to REST 0.2
* diffexp (#273)
* Add diffexp method to scanpy
and test
* Minor tweaks to diffexp
Get a minimal working version to unblock FE development
* Fixing things git deleted
* cleanup print statements
* Add index test
* additional, partial REST 0.2 bring up of diffexp
* Ignore unstructured annotations for data (#275)
This is a temp hack, need to figure out how to include data.uns if there is only one gene
* diffexp REST 0.2 port finish
* ignore unstructured annotaitons on all routes except layout
* correctly use varDataCache; maintain state during world rebuild
* correct varDataCache use
* temporarily disable all memoization
* refinements to expression data caching
* clear cell sets upon regraph/reset
* update version of REST to 0.2
* Travis build fixes
- comment out cache import
- fix duplicate test name
* Remove dependency from travis
* clarify semantics of config variables
* move generic action helpers into util
* mocks for redux refactor - for discussion
* more API design on redux refactor
* add new reuqired dependencies for build
* change babel target to use modern browser
* remove dead code
* remove dead code - joy plots
* checkpoint on redux refactoring
* checkpoint on redux refactoring
* fix mistaken rebase conflict resolution
* dead code removal; add name to dataframe backmap
* rename dataframe to universe
* update eslint config to more closely match prettier
* lint
* more eslint updates to match prettier
* additional config to make eslint match prettier
* add expression data to Universe/World
* remove obsolete reducers
* lint fixes
* more eslint cleanup
* lint
* lint
* fix but in countAllOnes when dimensions gt 1
* lint; do not display name metadata field
* lint; colors refactor
* lint; colors refactor
* update comments
* first cut at regraph and reset
* enable object-curly-braces consistent mode
* lint, handle regraph with no selection
* fix expression scatterplot bugs
* fix regression legend display
* add expression data cache
* remove console logging
* reset cell color on regraph/reset
* remove obsolete server URLs
* rename UniverseV01 to Universe_REST_API_v01
* add additional comments on the varDataCache
* merge universe reducer into controls reducer; simplify initialization-related actions
* use spread operator
* fix erroneous comment
* convert universe and world state to plain objects, and functionalize supporting code (remove ES6 classes)
* use spread operator
* lint
* improve variable names
* rename obsCrossfilter to crossfilter and obsDimensionMap to dimensionMap
* rename controls2 to controls
Matt Weiden