* Cleanup the backend-dev convenience method
* Add the frontend_dev convenience method
frontend_dev is a soup-to-nuts convenience method for setting up the FE
development environment with node running a the client code on port 3000
with the a separate cellxgene package serving the API over port 5005 in
the background.
The script can be run from Finder.
* Update the developer scripts documentation
* Remove the 'test' make target in the client Makefile
Rationale:
* Given how long the smoke tests take to run, it is unlikely that
developers will want to run all tests together.
* It is unlikely that developers will have set up the backend server
properly for the tests to pass.
* Available commands should be safe-ish and not lend themselves to
confusing errors.
* You can still group tests by concatenating them in a make command, as
in `make unit-test smoke-test`.
* This target isn't used in any of our CI pipelines -- KISS.
* Some version of python3...
* Minor typos in docs
* Respond to feedback from @bkmartinjr
* Make adjustments so that DATASET path is predictable
* Simplify environment defaults a bit
* add lint-client and lint-diff-client targets
* add lint-diff and lint targets
* prettier
* add lint-diff call to lint task
* temp
* tweak lint-diff
* add lint for PRs and lint for master
* remove temp
* remove incorrect branches syntax, use github_ref
* pull all branches
* format
* proper target and comment
* create separate steps with conditionals
* fix indentation
* refactor lint->lint-server, introduce lint to lint all
* trade diff-index for diff, do check against base instead of master
* remove fetching all branches
* Revert "remove fetching all branches"
This reverts commit 26ce7a0f05.
* tweak comparison
* use local eslint
* add eslint dep install
* temp
* grab only base
* simplify fetch
* add pull_request type trigger
* specify pushes only to master
* change conditionals to be based on event name
* Revert "temp"
This reverts commit 3d59134cc0.
* "branch" => "branches"
* create separate installation step
* change command based on os
* Improve diffexp for tiledb
- The rows from the A and B sets are gathered and processed at the same time. In this
way the matrix is only accessed once instead of twice for each tile.
- There is now a single thread queue that gets shared between all callers of the diffexp.
This will slow down work if diffexp gets too busy.
- There is a target_workunit amount of work given to each thread. Previously the
workunit was (rows selected * width of tile), which could be small. Now multiple
column tiles can be combined into one workunit. If the target is too small then
thread and other overheads may reduce performance. If target_workunit is too large
then the size of the gathered sub matrix may take up too much memory.
- add configuration parameters (max_workers, cpu_multiplier, and target_workunit)
* Specialize diffexp for tiledb
This patch adds a new diffexp algorithm which is tuned for tiledb.
This algorithm was written by Bruce and is adapted here to plug into the
current framework. The anndata_adaptor still calls the original
algotithm (which was move from diffexp.py to diffexp_generic.py).
The cxg_adaptor now calls the new diffexp_tiledb version. Some
code is shared between the two.
This is part 1 of the diffexp for tiledb. Further tuning and
global throttles are still needed.
A script to run and time diffexp with various options is also
added: test/run_diffexp.py.
* s3 region should have a single config param
The s3 region can also now be automatically determined to further
reduce errors.
This patch also fixes a bug with order of handling the config params.
The tiledb config needs to be fixed before attempting to load
(need to handle_adaptor before handle_single_dataset)
* warning on maxCount for diffexp
* cleanup logging
* clarification
* make the limits configurable
* make diff exp limit work
* danger!
* remove debugging code
* fix merge with master
* fix unit tests
Co-authored-by: Colin Megill <colinmegill@gmail.com>
fixes an issue with "cellxgene launch" which had a bad interaction between
command line parameters and config file parameters.
Now, the config files are applied first, followed by the parameters that
were provided in the command line.
There is also now a check that each of the config attributes is type checked.
Fixes https://github.com/chanzuckerberg/cellxgene/issues/1349
For more information see https://github.com/d3/d3-format
Note that does not _fully_ fix the issue described in #1349, but rather
makes the formatting issue far less likely. It is _still_ possible for
this to occur if the difference between two ticks in axes happes in the
a significant digit cropped by the scientific notation format
Currently, if there is no build directory, the `make install` target
will attempt to cd into the build directory, fail, and run pip install
-e . in the root directory anyway. This causes cellxgene to be installed
from the source tree instead of what the user would expect.
This commit changes the behavior such that the `make install` will fail
if there is no build directory.
* Improvements to the matrix cache
- Add a timelimit for the matrix in the cache.
Once the timelimit is reached, the matrix can be removed.
- If a DatasetAccessError occurs, then remove the dataset
from the matrix cache.
Fixes#1322
There is a small chicken and egg problem.
The config file could be in s3, therefore when using the DataLocator to
download the config file, we don't yet have an app_config object.
Adding a check to handle this case.
Mostly this is just instructions for how to do this,
with a small addition to the makefile.
This enables support for serving the about_legal_tos and about_legal_privacy
from the cellxgene server.
* Added a config hook for secret key into the app.
the server first looks in an environment variable,
then looks in a config file.
For the cellxgene launch app, a default key is used if none is provided.
For the eb app, a secret key must be provided.
See sample of current behavior below:
```
venv❯ cellxgene prepare example-dataset/pbmc3k.h5ad
[cellxgene] Starting CLI...
Error: [cellxgene] cellxgene prepare has not been installed. Please run
`pip install cellxgene[prepare]` to install the necessary requirements.
~/workspace/cellxgene mweiden/446-custom-color-palette*
1 venv❯ pip install cellxgene[prepare]
zsh: no matches found: cellxgene[prepare]
```
Fix:
Wrap cellxgene[prepare] in single quotes.
* Improved fix for matrix cache handling.
During the MatrixDataCacheItem acquire function there was a
time when the write lock was released and the read lock was taken.
During that time, the dataset could have been deleted, later
result in the MatrixDataCacheManageri data adaptor returning None.
The solution is to demote the writer lock to a reader lock instead
of unlocking and relocking.
Also, when a the cache needs to delete an entry, the delete
is done outside the MatrixDataCacheManager lock. This operation
only requires the write lock for the MatrixDataCacheItem.
Fixes#1255
