* Improvements to the matrix cache
- Add a timelimit for the matrix in the cache.
Once the timelimit is reached, the matrix can be removed.
- If a DatasetAccessError occurs, then remove the dataset
from the matrix cache.
Fixes#1322
* Improved fix for matrix cache handling.
During the MatrixDataCacheItem acquire function there was a
time when the write lock was released and the read lock was taken.
During that time, the dataset could have been deleted, later
result in the MatrixDataCacheManageri data adaptor returning None.
The solution is to demote the writer lock to a reader lock instead
of unlocking and relocking.
Also, when a the cache needs to delete an entry, the delete
is done outside the MatrixDataCacheManager lock. This operation
only requires the write lock for the MatrixDataCacheItem.
Fixes#1255
* Add user-generated annotations tests to the server
Partially completes https://github.com/chanzuckerberg/cellxgene/issues/969
* Auto-format python code
* @skip_if: passing lambdas > than property strings
* Respond to feedback from @bkmartinjr
This PR contains a refactoring to make adding new features easier.
The new features include supporting the tiledb format, and the multi dataset application.
The refactoring includes
Simplifying the directory structure and files.
a class structure to handle annotations (currently one type: AnnotationsLocalFile).
a class to handle application configuration
a class structure to handle matrix data (currently AnndataAdaptor and CxgAdaptor). CxgAdaptor uses tiledb.
Algorithms that were previously dependent on the scanpy anndata object are now generalized to work with an abstract interface.
The multi dataset option is not fully supported yet, and so the option to use it is hidden.
Use "cli launch --dataroot ..."
To access this feature.
All combinations of app single dataset/ app multi dataset and AnndataAdaptor/CxgAdaptor work with all the features, such as annotations, ontologies, diffexp.
