* Fix error message when datapath and dataroot are not provided
Previously:
$ cellxgene launch
cellxgene] Starting the CLI...
AttributeError: 'NoneType' object has no attribute 'startswith'
With this fix:
$ cellxgene launch
[cellxgene] Starting the CLI...
Error: missing datapath
* lint
* oauth support, add the token in a configuration specified cookie
Previously, the id token was stored in the session token.
Now, it can be placed in a different cookie with different properties.
* add oauth authentication
Add support for OAuth2.
Change the interface to AuthTypeBase
- better handling of config parameters
- add a complete_setup function for additional setup steps
Added a function wrapper to enforce authentication for the
routes that require authenticaiton.
* change fsspec requirement
fsspec 0.8.0 breaks our tests
it imports a module that is does not require.
* Add basic authentication in the server
A pattern for creating authentication methods is introduced, with three
authentication types defined:
none - no authentication
session - like the current session based auth used for user annotations
test - used to test the login/logout process end to end
The config endpoint now returns informations about the authentication, like if
the user is authenticated and their username. The redirect uri's for login and
logout are also returned if the authentication type requires login
This is the first a several PRs for authentication.
*. Update server tests to avoid hardcoded ports
test_api and test_nan_rest now use a common function for starting a test server,
than will initially choose a random port.
* Small fix for handling display versions
Making a distinction between __version__ and the version we display in the info panel (displayr_version).
The hosted cellxgene can overwrite the display_version using a plugin.
Improve version handling in the customized assets
This will give us the ability to specify different config options for
different dataroots.
the key of the dataroot dictionary is no longer the same as the dataroot_url.
Previously key==dataroot_url, and now those are separated.
Added an "is_multi_dataset" function to simplify logic where it branched on single vs multi.
Simplified the rest.py interface by no longer passing in the user annotations object, since
that can be retrieved from the dataset.
* add sentry webpack plugin
* allow override of webpack config
* work around cheerio inability to parse jinga templates
* webpack can not minify jinja templates
* allow script injection to specify other attributes
* allow script injection to specify other attributes
* Adjustments to make plugin systems work
* Add sourcemaps for javascript in prod webpack
* Update .gitignore
* Fix spelling errors
Co-authored-by: Matt Weiden <538456+mweiden@users.noreply.github.com>
* app_config, fix bug with list/tuple command line arguments.
There was a error caused by pyyaml using lists, and click using tuples.
Now tuples are automatically converted to lists when the config is
updated.
* Add server plugin system
Plugins are optional modules loaded at runtime. Specification:
* Plugins are loaded from the server.plugins module (directory
server/plugins)
* The import_plugins method is run as part of the initialization of the
server module in __init__.py
* Add plugins to the EB build process
* Remove bit of dead code
* Respond to feedback from @bmccandless
* Add user-defined category-label colors
Fixes https://github.com/chanzuckerberg/cellxgene/issues/1152
As described in https://github.com/chanzuckerberg/cellxgene/issues/1307
* Respond to feedback from @bkmartinjr in nodejs
* Respond to feedback from @bkmartinjr in python
* Add tests to the server module
* Autoformat python, run linter
* Make colors_get error handling specific
* Respond to feedback from @bkmartinjr
* Respond to feedback from @bkmartinjr
* Fix whitespace
* Fix python lint errrors
* Update documentation
* Add --disable-user-colors option to launch and cxgtool.py
* Fix python formatting
* Rename '--disable-user-colors' to '--disable-custom-colors'
* Improve diffexp for tiledb
- The rows from the A and B sets are gathered and processed at the same time. In this
way the matrix is only accessed once instead of twice for each tile.
- There is now a single thread queue that gets shared between all callers of the diffexp.
This will slow down work if diffexp gets too busy.
- There is a target_workunit amount of work given to each thread. Previously the
workunit was (rows selected * width of tile), which could be small. Now multiple
column tiles can be combined into one workunit. If the target is too small then
thread and other overheads may reduce performance. If target_workunit is too large
then the size of the gathered sub matrix may take up too much memory.
- add configuration parameters (max_workers, cpu_multiplier, and target_workunit)
* Specialize diffexp for tiledb
This patch adds a new diffexp algorithm which is tuned for tiledb.
This algorithm was written by Bruce and is adapted here to plug into the
current framework. The anndata_adaptor still calls the original
algotithm (which was move from diffexp.py to diffexp_generic.py).
The cxg_adaptor now calls the new diffexp_tiledb version. Some
code is shared between the two.
This is part 1 of the diffexp for tiledb. Further tuning and
global throttles are still needed.
A script to run and time diffexp with various options is also
added: test/run_diffexp.py.
* s3 region should have a single config param
The s3 region can also now be automatically determined to further
reduce errors.
This patch also fixes a bug with order of handling the config params.
The tiledb config needs to be fixed before attempting to load
(need to handle_adaptor before handle_single_dataset)
* warning on maxCount for diffexp
* cleanup logging
* clarification
* make the limits configurable
* make diff exp limit work
* danger!
* remove debugging code
* fix merge with master
* fix unit tests
Co-authored-by: Colin Megill <colinmegill@gmail.com>
fixes an issue with "cellxgene launch" which had a bad interaction between
command line parameters and config file parameters.
Now, the config files are applied first, followed by the parameters that
were provided in the command line.
There is also now a check that each of the config attributes is type checked.
* Improvements to the matrix cache
- Add a timelimit for the matrix in the cache.
Once the timelimit is reached, the matrix can be removed.
- If a DatasetAccessError occurs, then remove the dataset
from the matrix cache.
Fixes#1322
There is a small chicken and egg problem.
The config file could be in s3, therefore when using the DataLocator to
download the config file, we don't yet have an app_config object.
Adding a check to handle this case.
* Added a config hook for secret key into the app.
the server first looks in an environment variable,
then looks in a config file.
For the cellxgene launch app, a default key is used if none is provided.
For the eb app, a secret key must be provided.
* Improved fix for matrix cache handling.
During the MatrixDataCacheItem acquire function there was a
time when the write lock was released and the read lock was taken.
During that time, the dataset could have been deleted, later
result in the MatrixDataCacheManageri data adaptor returning None.
The solution is to demote the writer lock to a reader lock instead
of unlocking and relocking.
Also, when a the cache needs to delete an entry, the delete
is done outside the MatrixDataCacheManager lock. This operation
only requires the write lock for the MatrixDataCacheItem.
Fixes#1255
