This will give us the ability to specify different config options for
different dataroots.
the key of the dataroot dictionary is no longer the same as the dataroot_url.
Previously key==dataroot_url, and now those are separated.
Added an "is_multi_dataset" function to simplify logic where it branched on single vs multi.
Simplified the rest.py interface by no longer passing in the user annotations object, since
that can be retrieved from the dataset.
* Improvements to the matrix cache
- Add a timelimit for the matrix in the cache.
Once the timelimit is reached, the matrix can be removed.
- If a DatasetAccessError occurs, then remove the dataset
from the matrix cache.
Fixes#1322
* Improved fix for matrix cache handling.
During the MatrixDataCacheItem acquire function there was a
time when the write lock was released and the read lock was taken.
During that time, the dataset could have been deleted, later
result in the MatrixDataCacheManageri data adaptor returning None.
The solution is to demote the writer lock to a reader lock instead
of unlocking and relocking.
Also, when a the cache needs to delete an entry, the delete
is done outside the MatrixDataCacheManager lock. This operation
only requires the write lock for the MatrixDataCacheItem.
Fixes#1255
* black reformat
* tighten up error reporting
* lint
* fine tuning
* additional improvements in exception handling
* lint
* include exception and traceback in log
* fix typo
* Introduce a config file to cellxgene
The config file format is in yaml. The default config is located
in server/common/default_config.py. A user may create a yaml file
that contains a subset of these fields. It can be used during cellxgene
launch, or for hosted cellxgene.
The code has also been refactored. Much of the logic to check arguments
has moved from launch to app config.
It is now possible to set the tiledb context parameters using the config
file. Other feature will soon be handled in a similar way.
* Improve hosted cellxgene
- option to turn off the test index page, or supply a page for redirect.
For EB, The default is to return 404. For cli launch, the default is the test page.
- option to select which matrix types are allowed for multi dataset servers.
For EB, The default is CXG only. For cli launch, the default is any matrix type.
- Return early with an error response if diffexp is requested when not configured
- Verified that reembedings and user annotations also return with an error response
if used when not enabled.
TODO: The new options cannot currently be set by the user.
I plan to add a configuration file where these and all other settings can be set.
Fixes#1210Fixes#1228Fixes#1229
* allow DataLocator to accept another locator as init param
* migrate to DataLocator
* migrate to DataLocator
* lint
* migrate to DataLocator
* add check for erroroneous use of remote path and annotations
* lint
* revert default data location - now back go CWD
* remove unused import
* Add user-generated annotations tests to the server
Partially completes https://github.com/chanzuckerberg/cellxgene/issues/969
* Auto-format python code
* @skip_if: passing lambdas > than property strings
* Respond to feedback from @bkmartinjr
This PR contains a refactoring to make adding new features easier.
The new features include supporting the tiledb format, and the multi dataset application.
The refactoring includes
Simplifying the directory structure and files.
a class structure to handle annotations (currently one type: AnnotationsLocalFile).
a class to handle application configuration
a class structure to handle matrix data (currently AnndataAdaptor and CxgAdaptor). CxgAdaptor uses tiledb.
Algorithms that were previously dependent on the scanpy anndata object are now generalized to work with an abstract interface.
The multi dataset option is not fully supported yet, and so the option to use it is hidden.
Use "cli launch --dataroot ..."
To access this feature.
All combinations of app single dataset/ app multi dataset and AnndataAdaptor/CxgAdaptor work with all the features, such as annotations, ontologies, diffexp.
