* save graph selection in redux state
* fix old graph brush select regressions
* refactor graph brush selection to work with undo/redo
* update tests to match new crossfilter spatial select API
* graph selection state now in redux
* remove dead code
* sync graph selection with redux state; improvements to undoable machinery
* fix regression in undoable
* differentiate graph selection cancel from deselect action
* simplify calculation
* remove debugging code
* fix responsive repaint bug in graph selection tool
* undoable debugging and code cleanliness
* undoable action filter state now merges, rather than replaces
* improve comments
* add debounce to undoable action filter; improve comments and debug sanity check code
* comments
* fix undoable bug with clear scatterplot actions
* disable undoable debug flag
* cleanup API and comments around statemachine
* add test id attribute to lasso
* add better error handling for gene fetch requests
* immutable crossfilter
* first cut at reducer refactor with cascade model
* add initial redo/undo implementation
* small optimization
* integrate expression with history
* add tests for new reducers and fix a couple of small initialization bugs
* treat tiny lasso selections as a clear
* better function name for clarity
* fix undo for differential expression
* remove logging
* fix regression due to bad merge
* cleanup and comments for clarity
* improve undoable configuration for flexibility
* fix stale comments
* remove debugging code from production build
* rename categoricalSelectionState
* rename file
* improve comments
* initial dataframe commit
* initial dataframe port of core app
* rename variables for clarity
* remove unused import
* comment out unused code
* fix array handling bug in crossfilter dimension creation
* allow creation of empty dataframes
* handle non-existent columns
* handle non-existent columns
* revise tests for new dataframe
* comments for clarity
* comments for clarity
* generate bulk add placeholder with real gene names
* fix bug in gene name adding
* more dataframe unit tests
* fix bug - subset from current world, not universe
* put cut and pasted code into a single function
* improve caching of crossfilter
* remove cascading update bug from graph
* more performance work
* improve state handling for scatterplot
* performance optimization of critical path
* add column summarization
* dataframe utils
* add callOnceLazy
* fix tests
* minor updates found during review
* fix misspelling
* remove RESTv02 from function names
* comment cleanup
* cut/icut col parameter defaults to null
* break up large test
* improve tests and comments on dataframe at/has functions
* add Dataframe withCol/dropCol
* expression varData now stored in a dataframe
* dead code cleanup
* use dataframe.summarize()
* test cases for Dataframe.col.summarize
* update test cases for new dataframe summarize
* improve naming
* use new hasCol API
* add comments
* add more Dataframe.withCol tests
* add ability to specify row index in cut operation
* retire subsetVarData function
* correctly handle expression subsetting
* lint and improve comments
* rename cut to subset
* create helper file for controls reducer
* initial dataframe commit
* initial dataframe port of core app
* rename variables for clarity
* remove unused import
* comment out unused code
* fix array handling bug in crossfilter dimension creation
* allow creation of empty dataframes
* handle non-existent columns
* handle non-existent columns
* revise tests for new dataframe
* comments for clarity
* comments for clarity
* generate bulk add placeholder with real gene names
* fix bug in gene name adding
* more dataframe unit tests
* fix bug - subset from current world, not universe
* put cut and pasted code into a single function
* improve caching of crossfilter
* remove cascading update bug from graph
* more performance work
* improve state handling for scatterplot
* performance optimization of critical path
* add column summarization
* dataframe utils
* add callOnceLazy
* fix tests
* minor updates found during review
* fix misspelling
* remove RESTv02 from function names
* comment cleanup
* cut/icut col parameter defaults to null
* break up large test
* improve tests and comments on dataframe at/has functions
* add Dataframe withCol/dropCol
* expression varData now stored in a dataframe
* dead code cleanup
* use dataframe.summarize()
* test cases for Dataframe.col.summarize
* update test cases for new dataframe summarize
* improve naming
* use new hasCol API
* add comments
* add more Dataframe.withCol tests
* add ability to specify row index in cut operation
* retire subsetVarData function
* correctly handle expression subsetting
* lint and improve comments
* rename cut to subset
* suppress display of continous annotation withont a finite extent
* fix botched merge
* more fix of botched merged
* initial dataframe commit
* initial dataframe port of core app
* rename variables for clarity
* remove unused import
* comment out unused code
* fix array handling bug in crossfilter dimension creation
* allow creation of empty dataframes
* handle non-existent columns
* handle non-existent columns
* revise tests for new dataframe
* comments for clarity
* comments for clarity
* generate bulk add placeholder with real gene names
* fix bug in gene name adding
* more dataframe unit tests
* fix bug - subset from current world, not universe
* put cut and pasted code into a single function
* improve caching of crossfilter
* remove cascading update bug from graph
* more performance work
* improve state handling for scatterplot
* performance optimization of critical path
* add column summarization
* dataframe utils
* add callOnceLazy
* fix tests
* minor updates found during review
* fix misspelling
* remove RESTv02 from function names
* comment cleanup
* cut/icut col parameter defaults to null
* break up large test
* improve tests and comments on dataframe at/has functions
* add Dataframe withCol/dropCol
* expression varData now stored in a dataframe
* dead code cleanup
* use dataframe.summarize()
* test cases for Dataframe.col.summarize
* update test cases for new dataframe summarize
* improve naming
* use new hasCol API
* add comments
* add more Dataframe.withCol tests
* add ability to specify row index in cut operation
* retire subsetVarData function
* correctly handle expression subsetting
* lint and improve comments
* rename cut to subset
* changes based on PR review
* initial dataframe commit
* initial dataframe port of core app
* rename variables for clarity
* remove unused import
* comment out unused code
* fix array handling bug in crossfilter dimension creation
* allow creation of empty dataframes
* handle non-existent columns
* handle non-existent columns
* revise tests for new dataframe
* comments for clarity
* comments for clarity
* generate bulk add placeholder with real gene names
* fix bug in gene name adding
* more dataframe unit tests
* fix bug - subset from current world, not universe
* put cut and pasted code into a single function
* improve caching of crossfilter
* remove cascading update bug from graph
* more performance work
* improve state handling for scatterplot
* performance optimization of critical path
* add column summarization
* dataframe utils
* add callOnceLazy
* fix tests
* minor updates found during review
* fix misspelling
* remove RESTv02 from function names
* comment cleanup
* cut/icut col parameter defaults to null
* break up large test
* improve tests and comments on dataframe at/has functions
* dead code and route removal
* more dead code cleanup
* fix scanpy_engine tests
* lint
* add missing catch in filter parsing
* update scanpy NaN tests
* more fbs tests and dead test removal
* remove forced default for content type negotiation
* bit of cleanup
* more fbs test cleanup
* lint
* remove swagger
* swagger cleanup
* lint
* correctly handle lack of templates
* more dead code removal
* remove unused files
* fix dev build
* lint
* first flatbuffer schema
* do not lint auto-generated files
* add flatbuffers package
* add flatbuffer module
* wire up /data/X/T route
* use flatbuffers for matrix data fetc
* clarity and comments
* add flatbuffer layout route
* clean up obsolete code
* fix tests
* move flake8 config to setup.cfg
* add comments
* lint
* rework layout routes for fbs
* add more type support to fbs
* lint
* add flatbuffer support for annotations
* function name improvements
* fix botched merge with master
* remove unused import
* route cleanup for flatbuffers
* rename function for clarity
* add missing globals to Jest tests
* fix client JS tests
* fix routes for Python tests
* comments for clarity
* non-finite floating point hardening
* more non-finite number handling
* lint
* fix tests for summarizeAnnotations
* harden diffexp calculation against FP errors
* cleanup unused code
* lint
* add encoding tests for flatbuffers
* application type specified as strings
* fix spelling error
* improve variable names
* add note about documentation gap
* rename FBS DataFrame to Matrix
* only load annotation var names
* remove incorrect usage of var annotation data
* temporary workaround for issue #480
* lint
* issue warnings only once per item
* add cutoff for low expression genes in topN selection
* remove debugging printfs
* change cli param name for diffexp cutoff
* change CLI param name
* second try at diffexp - using lfc sort with pval cutoff
* use lfc cutoff
* update comments to match code; cap p-value adjustment to max of 1
* lint
* explain diffexp in readme
* add link
* add diffexp-lfc-cutoff to test config
* update test to match revised diffexp spec
* fix latent bug in GET arg handling that was breaking tests
* lint
* comment cleanup
* fix variance overestimation so it is symmetric
* lint
* new diffexp REST API spec
* new diffexp REST API; faster diffexp and dataframe slicing
* first draft of fast diffexp
* convert variance calculation to two-pass method
* lint
* update front-end use of API
* fix typo in spec
* disable content compression
* catch index filter format errors
* clean up of dead code
* resolve PR review comments
* remove memoization
* update to flash 1.0.2; turn on threading
* stand-alone helper routines for array slicing
* fix issue #405
* reset diffexp state when world changes
* performance work in dimension creation; fix world slicing bug
* update tests to match new state mgmt api
* update flask
* do not make dimensions for useless annotations
* update test to match optimizations
* add --obs-names and --var-names CLI params
* fix lint
* performance improvements in scanpy engine
* fix lint
* fix typo
* correctly handle sparse formats in diffexp
* fix diffexp and 1d slicing
* diffexp uses t-stat, not pval; clean up arg handling
* make _slice a static method
* revise scanpy tests to match new API
* Create Release_Validation_Recipe.md
* nameCreators test and better error handling
* add test for createObsDimensionMap
* add last set of tests for world module
* range encode filter range lists
* speed up data load
* add comment on scanpy read params
* update to latest scanpy/anndata
* performance improvments in data loading
* fix typo
* work around scanpy bug
* remove debugging print statements
Fixes#180
URL middleware unused, but retained in expectation that we will need it shortly.
Fixed a bit of lint, mostly by reformatting or commenting out currently unused code.
First cut at test for stateManager/world. Test data for REST responses factored out into separate file for shared use by Universe and World tests. Also cleaned up some debugging code.
REST API now requires that the server sort all dataframe responses by dimension index (var or obs index), allowing the client (front-end) to assume responses will be sorted. Given that the dataframe usually has a natural ordering, and the server gets to arbitrarily assign indices, this is the most computationally efficient design.
This PR includes changes to the REST API spec, and removing the explicit sort in the front-end code.
Jest tests for part of util/stateManager
* reorganize test directories
* lint
* add Jest config to eslint
* add Jest config to eslint
* lint
* fix babel configuration for tests
* start at kvCache tests
* reorganize test directories
* lint
* add Jest config to eslint
* add Jest config to eslint
* lint
* fix babel configuration for tests
* start at kvCache tests
* fix cache flush bug manifest by new test cases
* new kvCache tests
* skeletal universe test
* fix race in test case
* more universe test cases
Move to new REST v0.2 communication between front and back-end. This is a first cut implementation which is functional, but will need follow-up enhancements for performance, error checking, etc. Protocol spec is in docs directory.
* Add filtering via indexing
* Using new filter specs
Indexing working
* Added filtering by annotation value
* factor out common methods
* Documentation
* create enum for axis (obs/var)
* Better description for filter's return
* Add boolean to enumerated types
* Augmented enum for scanpy axis
* Create schema for annotations
Based on datatype within scanpy/anndata
+ tests
* remove obsolete schema parse script
* Update rest api to remove old routes and add schema route
* Separate development requirements
* Warning for unsupported datatypes
* include -r requirements.txt in dev
* Merged downcast warnings
* Fixed bug where names were NaNs
Needed to include the index too when creating the series
* Add config endpoint
* Generate app features from CLI selections
* Move features to driver
* Add tests for schema
* Clearer version wording
* python3 version of super
* version from engine to package level
* move features to driver
* Revise layout function to match the new spec
* GET for layout/obs
* PUT Layout (#211)
* PUT Layout
* Csweaver/annotations (#212)
* Update scanpy engine to support the rest v0.2 annotation requests
* GET endpoint for obs annotations + tests
* Documentation
* Test annotations in scanpy engine
* Description for annotation-keys param
* annotation->annotations
* clarified return for annotations
* Use URL query list for annotations fields
* parse_filter parses v0.2 GET filters (#215)
* parse_filter parses v0.2 GET filters
* Don't allow index filters from query params
* Better variable conversion
* Parse filter improvements
- uses default dict
- renamed filter -> query_filter
* Cleanup Tasks (#216)
* Add test_api back into travis build
* Do custom JSON encoding the correct way
* Run cellxgene server in test setup
* Cleanup new tests too
* Option to bind to all interfaces (#225)
app.run("0.0.0.0") instead of app.run("127.0.0.1") binds to all interfaces.
Note: There are comments on the internet that says that the flask server is not up to the task of production serving. I don't think that such scalability concerns apply here, but I was able to get cellxgene working with twistd relatively easily, and we could switch to that if there are scalability concerns.
Test plan: browsed to <ip>:5005/api/v0.2/config on a different host.
* Add filtering via indexing
* Using new filter specs
Indexing working
* Added filtering by annotation value
* factor out common methods
* Documentation
* create enum for axis (obs/var)
* Better description for filter's return
* Add boolean to enumerated types
* Augmented enum for scanpy axis
* Create schema for annotations
Based on datatype within scanpy/anndata
+ tests
* remove obsolete schema parse script
* Update rest api to remove old routes and add schema route
* Separate development requirements
* Warning for unsupported datatypes
* include -r requirements.txt in dev
* Merged downcast warnings
* Fixed bug where names were NaNs
Needed to include the index too when creating the series
* Add config endpoint
* Generate app features from CLI selections
* Move features to driver
* Add tests for schema
* Clearer version wording
* python3 version of super
* version from engine to package level
* move features to driver
* Revise layout function to match the new spec
* GET for layout/obs
* PUT Layout (#211)
* PUT Layout
* Csweaver/annotations (#212)
* Update scanpy engine to support the rest v0.2 annotation requests
* GET endpoint for obs annotations + tests
* Documentation
* Test annotations in scanpy engine
* Description for annotation-keys param
* annotation->annotations
* clarified return for annotations
* Use URL query list for annotations fields
* parse_filter parses v0.2 GET filters (#215)
* parse_filter parses v0.2 GET filters
* Don't allow index filters from query params
* Better variable conversion
* Parse filter improvements
- uses default dict
- renamed filter -> query_filter
* Cleanup Tasks (#216)
* Add test_api back into travis build
* Do custom JSON encoding the correct way
* Run cellxgene server in test setup
* Cleanup new tests too
* Option to bind to all interfaces (#225)
app.run("0.0.0.0") instead of app.run("127.0.0.1") binds to all interfaces.
Note: There are comments on the internet that says that the flask server is not up to the task of production serving. I don't think that such scalability concerns apply here, but I was able to get cellxgene working with twistd relatively easily, and we could switch to that if there are scalability concerns.
Test plan: browsed to <ip>:5005/api/v0.2/config on a different host.
* Fix merge errors
- import warnings was improperly deleted
- scanpy engine tests were totally wrong
* Fix merge error with driver
* PUT /annotations (#235)
* Add query param for annotation name
* fix descriptions, eliminate else clause
* first cut at initial data load on rest 0.2 api
* Annotation var (#248)
* Fix bug strings are always objects in pandas
* Add axis to annotation method
* Add /annotation/var to REST api
* Csweaver/expressiondata (#242)
* Refactor expression method for REST v2
* Add message to QueryStringError
* Fix range filters
* Add GET route for /data
* /data PUT route
* rename expression to data_frame
* clarification of error
* Improve accept type handling
* support all schema types for 0.2 REST API
* remove REST 0.1 code; connect var annotations loading
* config reducer; use config to set data set title; remove obsolete templating code for data set title
* REST 0.2 expression conversion support
* partial port of expression to REST 0.2
* diffexp (#273)
* Add diffexp method to scanpy
and test
* Minor tweaks to diffexp
Get a minimal working version to unblock FE development
* Fixing things git deleted
* cleanup print statements
* Add index test
* additional, partial REST 0.2 bring up of diffexp
* Ignore unstructured annotations for data (#275)
This is a temp hack, need to figure out how to include data.uns if there is only one gene
* diffexp REST 0.2 port finish
* ignore unstructured annotaitons on all routes except layout
* correctly use varDataCache; maintain state during world rebuild
* correct varDataCache use
* temporarily disable all memoization
* refinements to expression data caching
* clear cell sets upon regraph/reset
* update version of REST to 0.2
* Travis build fixes
- comment out cache import
- fix duplicate test name
* Remove dependency from travis
* clarify semantics of config variables
* move generic action helpers into util
* mocks for redux refactor - for discussion
* more API design on redux refactor
* add new reuqired dependencies for build
* change babel target to use modern browser
* remove dead code
* remove dead code - joy plots
* checkpoint on redux refactoring
* checkpoint on redux refactoring
* fix mistaken rebase conflict resolution
* dead code removal; add name to dataframe backmap
* rename dataframe to universe
* update eslint config to more closely match prettier
* lint
* more eslint updates to match prettier
* additional config to make eslint match prettier
* add expression data to Universe/World
* remove obsolete reducers
* lint fixes
* more eslint cleanup
* lint
* lint
* fix but in countAllOnes when dimensions gt 1
* lint; do not display name metadata field
* lint; colors refactor
* lint; colors refactor
* update comments
* first cut at regraph and reset
* enable object-curly-braces consistent mode
* lint, handle regraph with no selection
* fix expression scatterplot bugs
* fix regression legend display
* add expression data cache
* remove console logging
* reset cell color on regraph/reset
* remove obsolete server URLs
* rename UniverseV01 to Universe_REST_API_v01
* add additional comments on the varDataCache
* merge universe reducer into controls reducer; simplify initialization-related actions
* use spread operator
* fix erroneous comment
* convert universe and world state to plain objects, and functionalize supporting code (remove ES6 classes)
* use spread operator
* lint
* improve variable names
* rename obsCrossfilter to crossfilter and obsDimensionMap to dimensionMap
* rename controls2 to controls
Bruce Martin