* hosted, update order to look for config file.
The app now uses a local config.yaml file bundled with the artifact
(if present), if it exists, then looks in the CXG_CONFIG_FILE
environment variable. This is the reverse of previous behavior.
The purpose of this change is to move away from using the
config file on s3, since that could lead to problem where an older
version of the app uses a newer version of the config.
Also in this PR:
1. Changed documentation around dataroot, to describe the posibility of using lustre.
2. Added a few improvements around the secret manager region name. If we use lustre for dataroot and a local config file, then we will no longer be able to
auto determine the region for the secret manager. I plan to start using the
environment variable option for hosted cellxgene.
* small edit to README
Co-authored-by: Severiano Badajoz <sbadajoz@chanzuckerberg.com>
* add unsafe-inline directive to style-src
* debugging - turn on csp reporting
* revert reporting only csp
* do not inline JS and CSS in build
* enable HTTPs only when in production mode
* remove debug printf
* fix clean target
* revert force_https removal
* Return empty colors for .cxg v0.0 files
Fixes https://github.com/chanzuckerberg/cellxgene/issues/1440
The CxgAdaptor.get_colors method currently assumes that the .cxg file has
cxg_group_metadata. As a result, the /api/v0.2/colors endpoint always fails for
.cxg v0.0 files.
* Add test fixture
* Add user-defined category-label colors
Fixes https://github.com/chanzuckerberg/cellxgene/issues/1152
As described in https://github.com/chanzuckerberg/cellxgene/issues/1307
* Respond to feedback from @bkmartinjr in nodejs
* Respond to feedback from @bkmartinjr in python
* Add tests to the server module
* Autoformat python, run linter
* Make colors_get error handling specific
* Respond to feedback from @bkmartinjr
* Respond to feedback from @bkmartinjr
* Fix whitespace
* Fix python lint errrors
* Update documentation
* Add --disable-user-colors option to launch and cxgtool.py
* Fix python formatting
* Rename '--disable-user-colors' to '--disable-custom-colors'
* hello world
* stuff
* successful build
* updates"
* maybe a basic example
* simplify
* reamde into dockerfile
* some more stuff
* Release procfile
* package.json at top levle
* don't release in procfile for now
* more package.json stuff
* copy assets
* merge master
* not in the relase phase
* revert not necessary
* pin gunicorn version
* reset common.mk
* Update package.json
Co-authored-by: Madison Dunitz <dunitzm@gmail.com>
* Improve diffexp for tiledb
- The rows from the A and B sets are gathered and processed at the same time. In this
way the matrix is only accessed once instead of twice for each tile.
- There is now a single thread queue that gets shared between all callers of the diffexp.
This will slow down work if diffexp gets too busy.
- There is a target_workunit amount of work given to each thread. Previously the
workunit was (rows selected * width of tile), which could be small. Now multiple
column tiles can be combined into one workunit. If the target is too small then
thread and other overheads may reduce performance. If target_workunit is too large
then the size of the gathered sub matrix may take up too much memory.
- add configuration parameters (max_workers, cpu_multiplier, and target_workunit)
* Specialize diffexp for tiledb
This patch adds a new diffexp algorithm which is tuned for tiledb.
This algorithm was written by Bruce and is adapted here to plug into the
current framework. The anndata_adaptor still calls the original
algotithm (which was move from diffexp.py to diffexp_generic.py).
The cxg_adaptor now calls the new diffexp_tiledb version. Some
code is shared between the two.
This is part 1 of the diffexp for tiledb. Further tuning and
global throttles are still needed.
A script to run and time diffexp with various options is also
added: test/run_diffexp.py.
* s3 region should have a single config param
The s3 region can also now be automatically determined to further
reduce errors.
This patch also fixes a bug with order of handling the config params.
The tiledb config needs to be fixed before attempting to load
(need to handle_adaptor before handle_single_dataset)
* warning on maxCount for diffexp
* cleanup logging
* clarification
* make the limits configurable
* make diff exp limit work
* danger!
* remove debugging code
* fix merge with master
* fix unit tests
Co-authored-by: Colin Megill <colinmegill@gmail.com>
fixes an issue with "cellxgene launch" which had a bad interaction between
command line parameters and config file parameters.
Now, the config files are applied first, followed by the parameters that
were provided in the command line.
There is also now a check that each of the config attributes is type checked.
* Improvements to the matrix cache
- Add a timelimit for the matrix in the cache.
Once the timelimit is reached, the matrix can be removed.
- If a DatasetAccessError occurs, then remove the dataset
from the matrix cache.
Fixes#1322
There is a small chicken and egg problem.
The config file could be in s3, therefore when using the DataLocator to
download the config file, we don't yet have an app_config object.
Adding a check to handle this case.
Mostly this is just instructions for how to do this,
with a small addition to the makefile.
This enables support for serving the about_legal_tos and about_legal_privacy
from the cellxgene server.
* Added a config hook for secret key into the app.
the server first looks in an environment variable,
then looks in a config file.
For the cellxgene launch app, a default key is used if none is provided.
For the eb app, a secret key must be provided.
See sample of current behavior below:
```
venv❯ cellxgene prepare example-dataset/pbmc3k.h5ad
[cellxgene] Starting CLI...
Error: [cellxgene] cellxgene prepare has not been installed. Please run
`pip install cellxgene[prepare]` to install the necessary requirements.
~/workspace/cellxgene mweiden/446-custom-color-palette*
1 venv❯ pip install cellxgene[prepare]
zsh: no matches found: cellxgene[prepare]
```
Fix:
Wrap cellxgene[prepare] in single quotes.
