mirror of
https://github.com/chanzuckerberg/cellxgene.git
synced 2026-09-15 12:47:56 +08:00
* csweaver changes from PR #912 * fix formatting error
This commit is contained in:
Bruce Martin
120
dev_docs/developer_scripts.md
Normal file
120
dev_docs/developer_scripts.md
Normal file
@@ -0,0 +1,120 @@
|
||||
## Node scripts in package.json
|
||||
|
||||
#### backend-dev
|
||||
|
||||
**About** This scripts enables FE developers to run the rest API necessary to back the development server for the front end. It is intended to ensure that the FE developer gets the current version of the backend with a single command and no knowledge of python necessary. It creates and activates a virtual environment and installs cellxgene from the current branch.
|
||||
|
||||
**Requires** Python3.6 - `virtual-env`, `pip`
|
||||
|
||||
**Usage** `npm run backend-dev <dataset>` then you can launch the node development server to serve the current state of the FE `npm run start`
|
||||
|
||||
**Tips** Developers will probably want to run this in parallel with the node dev server. You can either do this by running each in a separate termanial window or by running `backend-dev` in the background (add an `&` at the end of the command to run in the background: `npm run backend-dev <dataset> &`).
|
||||
|
||||
**Breakdown**
|
||||
|
||||
| command | purpose |
|
||||
| ---------------------------------------- | --------------------------------------------------------- |
|
||||
| python3.6 -m venv cellxgene | creates cellxgene virtual environment |
|
||||
| source cellxgene/bin/activate | activates virtual environment |
|
||||
| yes \| pip uninstall cellxgene \|\| true | uninstalls cellxgene (if installed) |
|
||||
| pip install -e .. | installs current local version of cellxgene |
|
||||
| cellxgene launch | launches cellxgene (must supply dataset as last parameter |
|
||||
|
||||
### Test scripts
|
||||
|
||||
#### test
|
||||
|
||||
**About** Run test locally. In order for this command to succeed you will need to give it a specific unit test to run. It won't pass if you run all tests as may be expected. This is because the unit tests and end to end (e2e) tests require different testing environments.
|
||||
|
||||
**Usage** `npm run test <test file or pattern>`
|
||||
|
||||
#### unit-test
|
||||
|
||||
**About** Runs all unit tests. It excludes any tests in the e2e folder. This is used by travis to run unit tests.
|
||||
|
||||
**Usage** `npm run unit-test`
|
||||
|
||||
#### smoke-test
|
||||
|
||||
**About** Starts backend development server and runs end to end tests. This is what travis runs. It depends on the `e2e` and the `start-server-for-test` node scripts. One starts the server, the other runs the tests. If developing a front-end feature and just checking if tests pass, this is probabaly the one you want to run.
|
||||
|
||||
**Requirements** Must set env variable for port. In terminal `export JEST_ENV='dev'` `export JEST_CXG_PORT=5000`
|
||||
|
||||
**Usage** `npm run smoke-test`
|
||||
|
||||
#### e2e
|
||||
|
||||
**About** Runs backend tests without starting the server. You will need to start the rest api separately with the pbmc3k.h5ad file.
|
||||
|
||||
**Requirements** Must set env variable for port. In terminal `export JEST_ENV='dev'` `export JEST_CXG_PORT=5000`
|
||||
|
||||
**Usage** `npm run e2e`
|
||||
|
||||
## makefile
|
||||
|
||||
### Build commands
|
||||
|
||||
builds source code
|
||||
|
||||
```
|
||||
build - builds whole app client and server
|
||||
build-server - makes build dir and moves python and client files
|
||||
build-client - runs webpack build
|
||||
build-for-server-dev - builds client and copies output directly into source tree (only for server devlopment)
|
||||
```
|
||||
|
||||
### Clean commands
|
||||
|
||||
deletes generated files
|
||||
|
||||
```
|
||||
clean - cleans everything including node modules (means build with take a while
|
||||
clean-lite - cleans built directories
|
||||
clean-server - cleans source tree
|
||||
```
|
||||
|
||||
### Dist commands
|
||||
|
||||
creates distribution for python module to upload to pypi
|
||||
|
||||
```
|
||||
pydist - builds code and then builds sdist
|
||||
```
|
||||
|
||||
### Release commands
|
||||
|
||||
see release_process.md
|
||||
|
||||
### Env commands
|
||||
|
||||
Installs requirements files
|
||||
|
||||
```
|
||||
dev-env - installs requirements and requirments-dev (for building code)
|
||||
gui-env - installs requirements and requirments-dev and requirments-gui (for building native app)
|
||||
```
|
||||
|
||||
### install commands
|
||||
|
||||
Installs cellxgene from different locations
|
||||
|
||||
```
|
||||
install - installs from local build directory
|
||||
install-dev - installs from local source tree
|
||||
install-release-test - installs from test pypi
|
||||
install-release - installs from pypi
|
||||
install-dist - installs from local dist folder
|
||||
uninstall - uninstalls cellxgene
|
||||
```
|
||||
|
||||
### gui
|
||||
|
||||
Commands for building the native app
|
||||
|
||||
```
|
||||
build-assets - builds the image and icon assets for the gui to pull from
|
||||
gui-spec-osx - creates the initial spec file for osx, do not run unless you are starting from scratch, one time only
|
||||
gui-spec-windows - creates the initial spec file for windows, do not run unless you are starting from scratch, one time only
|
||||
gui-build-osx - builds the app from the osx spec file
|
||||
gui-build-windows - builds the app from the windows spec file
|
||||
```
|
||||
@@ -9,84 +9,124 @@ to PyPi, with a matching tagged release on github.
|
||||
|
||||
The release process should result in the following side-effects:
|
||||
|
||||
- Version number bump, using semantic versioning
|
||||
- JS assets built & packaged, committed to the repo
|
||||
- Tagged github release
|
||||
- Publication to PyPi
|
||||
- Version number bump, using semantic versioning
|
||||
- JS assets built & packaged, committed to the repo
|
||||
- Tagged github release
|
||||
- Publication to PyPi
|
||||
|
||||
## Recipe
|
||||
|
||||
Follow these steps to create a release.
|
||||
|
||||
1. Preparation:
|
||||
- python3.6 environment, and a cellxgene clone
|
||||
- Define the release version number, using [semantic versioning](https://semver.org/),
|
||||
and specifying all three digits (eg, 0.3.0)
|
||||
- Write the release title and release notes and add to
|
||||
[release notes document](https://docs.google.com/document/d/1KnHwkYfhyWO5H8BDcMu7y3ogjvq5Yi4OwpmZ8DB6w0Y/edit)
|
||||
- python3.6 environment, and a cellxgene clone
|
||||
- Define the release version number, using [semantic versioning](https://semver.org/),
|
||||
and specifying all three digits (eg, 0.3.0)
|
||||
- Write the release title and release notes and add to
|
||||
[release notes document](https://docs.google.com/document/d/1KnHwkYfhyWO5H8BDcMu7y3ogjvq5Yi4OwpmZ8DB6w0Y/edit)
|
||||
2. Create a release branch, eg, `release-version`
|
||||
3. In the release branch:
|
||||
- Run `make release-stage-1 PART=[major | minor | patch]` where you choose major/minor/patch depending on which part of the version
|
||||
is being bumped (eg, 0.2.9->0.3 is minor).
|
||||
- Run `make release-stage-1 PART=[major | minor | patch]` where you choose major/minor/patch depending on which part of the version
|
||||
is being bumped (eg, 0.2.9->0.3 is minor).
|
||||
4. Commit and push the new branch
|
||||
5. Create a PR for the release.
|
||||
- [optional] As needed, conduct PR review.
|
||||
- [optional] As needed, conduct PR review.
|
||||
6. Merge to master
|
||||
7. Create Github release using the version number and release notes
|
||||
7. Publish to pypi by performing the following steps (assumes you that you have registered for pypi,
|
||||
and that you have write access to the cellxgene pypi package):
|
||||
- Build the distribution and upload to test pypi `make release-stage-2`
|
||||
- Test the test installation in a fresh virtual environment using `make install-release-test`
|
||||
- Upload the package to real pypi using `make release-stage-final`
|
||||
- Test the installation in a fresh virtual environment using `pip install cellxgene`
|
||||
8. Create Github release using the version number and release notes
|
||||
([instructions](https://help.github.com/articles/creating-releases/)).
|
||||
- Draft new release
|
||||
- Type version name matching release version number from (1)
|
||||
- Select `master` as release branch (ensure you merged the release PR)
|
||||
- Type title `Release {version num}`
|
||||
- [optional] Check pre-release if this release is not ready for production
|
||||
- Publish Release
|
||||
8. Publish to pypi by performing the following steps (assumes you that you have registered for pypi,
|
||||
and that you have write access to the cellxgene pypi package):
|
||||
- Build the distribution and upload to test pypi `make release-stage-2`
|
||||
- Test the test installation in a fresh virtual environment using `make install-release-test`
|
||||
- Upload the package to real pypi using `make release-stage-final`
|
||||
- Test the installation in a fresh virtual environment using
|
||||
`pip install cellxgene`
|
||||
- Draft new release
|
||||
- Type version name matching release version number from (1)
|
||||
- Select `master` as release branch (ensure you merged the release PR)
|
||||
- Type title `Release {version num}`
|
||||
- [optional] Check pre-release if this release is not ready for production
|
||||
- Publish Release
|
||||
|
||||
The optional steps are for testing purposes, and are recommended
|
||||
for publishing any major releases, and any releases that significantly
|
||||
change the packaging (e.g. new bundled files, new dependencies, etc.)
|
||||
|
||||
### Point release (special case)
|
||||
|
||||
To make a bugfix release (a point release) when there are already other changes in master we need to do a modified version of our release process. The difference is that instead of using master we are going make our release branch off of the tag for the release we want to patch. We cherrypick the commits that we want to include in the patch. Then instead of merging to master, we create the release directly off of the branch.
|
||||
|
||||
1. (same as above) Preparation:
|
||||
- python3.6 environment, and a cellxgene clone
|
||||
- Define the release version number, using [semantic versioning](https://semver.org/),
|
||||
and specifying all three digits (eg, 0.3.0) (for this you will update the last digit to represent a bugfix change)
|
||||
- Write the release title and release notes and add to
|
||||
[release notes document](https://docs.google.com/document/d/1KnHwkYfhyWO5H8BDcMu7y3ogjvq5Yi4OwpmZ8DB6w0Y/edit)
|
||||
2. Create a release branch off of the tag for the release you want to update.
|
||||
- Checkout the tag for the release you want to fix. ex. if we are fixing 0.9.0: `git checkout 0.9.0`
|
||||
- Create a branch from that tag. `git branch release-0.9.1`
|
||||
3. Cherrypick the commits that you want included in this patch.
|
||||
- Test that the cherrypicked commits landed and fixed the issue
|
||||
- We WILL NOT merge this branch back into master, these commits should already exist in master.
|
||||
4. In the release branch:
|
||||
- Run `make release-stage-1 PART=patch`.
|
||||
5. Commit and push the new branch. DO NOT MAKE A PR OR MERGE TO MASTER.
|
||||
- wait for release to pass the tests
|
||||
6. Publish to pypi by performing the following steps (assumes you that you have registered for pypi,
|
||||
and that you have write access to the cellxgene pypi package): - Build the distribution and upload to test pypi `make release-stage-2` - Test the test installation in a fresh virtual environment using `make install-release-test` - Upload the package to real pypi using `make release-stage-final` - Test the installation in a fresh virtual environment using
|
||||
`pip install --no-cache-dir cellxgene`
|
||||
7. Create Github release using the version number and release notes
|
||||
([instructions](https://help.github.com/articles/creating-releases/)).
|
||||
- Draft new release
|
||||
- Type version name matching release version number from (1)
|
||||
- _Different than above_ Select the release-branch you pushed at step 5 as release branch
|
||||
- Type title `Release {version num}`
|
||||
- [optional] Check pre-release if this release is not ready for production
|
||||
- Publish Release
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Fails to upload to test.pypi
|
||||
|
||||
_PyPi doesn't allow you to reupload a release with the same version number_
|
||||
|
||||
_PyPi doesn't allow you to reupload a release with the same version number_
|
||||
If you accidentally burned a release number you want to use on prod, you have a few options:
|
||||
1) OPTION 1: Create distribution `make pydist`; test release locally `pip install dist/<release tarball>`;
|
||||
then upload to prod `make release-stage-final`.
|
||||
2) OPTION 2: (DANGER) release directly to prod: `make release-directly-to-prod`.
|
||||
3) OPTION 3: If the release was burned on prod as well run from Step 3 again with option
|
||||
PART=patch until you get to an unburned version.
|
||||
|
||||
|
||||
1. OPTION 1: Create distribution `make pydist`; test release locally `pip install dist/<release tarball>`;
|
||||
then upload to prod `make release-stage-final`.
|
||||
2. OPTION 2: (DANGER) release directly to prod: `make release-directly-to-prod`.
|
||||
3. OPTION 3: If the release was burned on prod as well run from Step 3 again with option
|
||||
PART=patch until you get to an unburned version.
|
||||
|
||||
### The release doesn't install or fails your tests when you install it
|
||||
|
||||
Delete it from pypi - Go to pypi.org -> sign in -> go to the cellxgene package -> click manage -> then in the options drop down click delete -> follow the instructions. You will not be able to use that release number again. If it is a minor bug and not a major regression, you can just release a patch.
|
||||
|
||||
Delete it from pypi - Go to pypi.org -> sign in -> go to the cellxgene package -> click manage -> then in the options drop down click delete -> follow the instructions. You will not be able to use that release number again. If it is a minor bug and not a major regression, you can just release a patch.
|
||||
|
||||
### If you need to run stage final on a different computer than stage 2
|
||||
If you run stage final without running stage 2 first, the dist will not have been build on the computer running stage final. The solution is to run `make release-directly-to-prod`. This both builds the distribution files and then releases directly to prod pypi.org.
|
||||
|
||||
If you run stage final without running stage 2 first, the dist will not have been build on the computer running stage final. The solution is to run `make release-directly-to-prod`. This both builds the distribution files and then releases directly to prod pypi.org.
|
||||
|
||||
## Stage Details
|
||||
|
||||
### Stage 1 - `make release-stage-1`
|
||||
|
||||
1. Pip installs requirements-dev
|
||||
2. Bumps version by [PART]
|
||||
3. Deletes build directory, client/build, dist and cellxgene.egg-info
|
||||
4. Creates the package-lock.json
|
||||
|
||||
### Stage 2 - `make release-stage-2`
|
||||
|
||||
1. Pip installs requirements-dev
|
||||
2. Builds client and server
|
||||
3. Creates distribution release (sdist)
|
||||
4. Uploads to test.pypi.org
|
||||
|
||||
### Stage final - `make release-stage-final`
|
||||
|
||||
** Does not build distribution **
|
||||
1. Uploads to pypi.org
|
||||
|
||||
|
||||
### (DANGER) Release directly to prod `make release-directly-to-prod`
|
||||
|
||||
** builds distribution and uploads directly to prod **
|
||||
Only use this if you are directed to by the troubleshooting guide
|
||||
1. Pip installs requirements-dev
|
||||
|
||||
Reference in New Issue
Block a user