fork/cellxgene
1
0
Fork 0
mirror of https://github.com/chanzuckerberg/cellxgene.git synced 2026-09-23 11:38:12 +08:00
Code Issues Packages Projects Releases Wiki Activity

1510-smoke-test (#1548)

Browse Source 
* 1510-smoke-test

* config default

* update tests

* update test config

* fix linter errors

* more comments

* address comments

* use npm install in push_tests.yml

* use environment.default.json

* adding docs

* Take care of @mweiden's nits

* Save screenshots in the __tests__/screenshots/ directory

* typo

* docs

* Add chart tests (#1580)

* merge tests

* check if bin creation returned null before rendering charts (#1576)

* check if bin creation returned null before rendering charts

* refactor chart rendering into functions (#1577)

* little fixes from PR

* reintroduce fix to check for null values

* change getAllByClass to return element

* slice instead

* new stackedbar test

* feedback-1573-test (#1579)

* feedback-1573-test

* enable whole test set

* revert tests

Co-authored-by: Timmy Huang <tihuan@users.noreply.github.com>

* tweak test to actually render chart

* include snapshot

* remove async

* fix getAllHistograms

* properly grab id

Co-authored-by: Timmy Huang <tihuan@users.noreply.github.com>

Co-authored-by: Matt Weiden <538456+mweiden@users.noreply.github.com>
Co-authored-by: Severiano Badajoz <sbadajoz@chanzuckerberg.com>
This commit is contained in:
Timmy Huang
2020-06-24 11:45:39 -07:00
committed by GitHub
co-authored by Matt Weiden Severiano Badajoz
parent e22e671f10
commit 83376627e8
42 changed files with 1388 additions and 887 deletions
Download Patch File Download Diff File Expand all files Collapse all files
dev_docs/developer_guidelines.md
+64 -24
View File
@@ -1,6 +1,7 @@
# Developer guidelines
### Requirements
## Requirements
- npm
- Python 3.6+
- Chrome
@@ -11,20 +12,24 @@
### Environment
For all `make` commands, `common.mk` automatically checks whether required environment variables are set and, if they are not set, assigns them default values from `environment.default`.
For all `make` commands, `common.mk` automatically checks whether required environment variables are set and, if they are not set, assigns them default values from `environment.default.json`.
You can set these environment variables manually with the `export` shell command, as in `export JEST_ENV=debug`.
You can set these environment variables manually with the `export` shell command, as in `export JEST_ENV=debug`, or you can just pass the variables as part of the command. E.g., `HEADFUL=true make e2e` or `JEST_ENV=debug npm run e2e`
## Running test suite
Client and server tests run on Travis CI for every push, PR, and commit to master on github. End to end tests run nightly on master only.
Client and server tests run on Travis CI for every push, PR, and commit to master on github. End to end tests run nightly on master only.
### Unit tests
Steps to run the all unit tests:
1. Start in the project root directory
1. `make dev-env`
1. `make unit-test`
To run unit tests for the `client` code only:
1. Start in the project root directory
1. `cd client`
1. `make unit-test`
@@ -33,37 +38,58 @@ To run unit tests for the `client` code only:
To run E2E tests, run `cd client` and `make smoke-test`
The `JEST_ENV` environment variable enables the following E2E test options:
* `dev` - opens chromimum, runs tests with minimal slowdown, close on exit.
* `debug` - opens chromium, runs tests with 100ms slowdown, dev tools open, chrome stays open on exit.
* `prod` - run headless with no slowdown, chromium will not open.
#### Flags
Run end to end tests interactively during development
1. cellxgene should be installed as [specified in client dev](#install-1)
1. Follow [launch](#launch-1) instructions for client dev with dataset `example-dataset/pbmc3k`
1. Run `npm run e2e` or `make e2e` from the `client` directory
1. To debug a failing test `export JEST_ENV='debug'` and re-run.
1. `JEST_ENV`: This enables the following E2E test options. You can find their corresponding configs in [`jest-puppeteer.config.js`](../client/jest-puppeteer.config.js):
To run end to end tests _exactly_ as they will be run on CI use the following command:
```
- `dev` - opens window, runs tests with minimal slowdown, close on exit.
- `debug` - opens window, runs tests with 100ms slowdown, dev tools open, chrome stays open on exit.
- `prod`[default] - run headless with no slowdown, window will not open.
2. `HEADFUL`: Default is `false`. When set to `true`, it will launch the Chrome window for visual inspection. E.g., `HEADFUL=true npm run e2e`
3. `HEADLESS`: Default is `true`. When set to `false`, it will launch the Chrome window for visual inspection. E.g., `HEADLESS=false npm run e2e`
#### Run end to end tests interactively during development
1. cellxgene should be installed as [specified in client dev](#install)
1. Follow [launch](#launch) instructions for client dev with dataset `example-dataset/pbmc3k`
1. Run `npm run e2e` from the `client` directory
1. To debug a failing test, add `debugger` in any line of JS code as breakpoint, and launch the test again with [`ndb`](https://github.com/GoogleChromeLabs/ndb). E.g., `ndb make e2e` or `ndb npm run e2e`.
1. Please make sure to install `ndb` via `npm install -g ndb`
1. Check out [Debugging Tips](e2e_tests.md#debugging-tips) for more ideas!
#### To run end to end tests _exactly_ as they will be run on CI use the following command
```shell
JEST_ENV=prod make pydist install-dist dev-env smoke-test
```
## Server dev
### Install
To install from the source tree
* Build the client and put static files in place: `make build-for-server-dev`
* Install from local files: `make install-dev`
- Build the client and put static files in place: `make build-for-server-dev`
- Install from local files: `make install-dev`
To install from a candidate python distribution
* Make the distribution: `make pydist`
* Install it: `make install-dist`
- Make the distribution: `make pydist`
- Install it: `make install-dist`
### Launch
* `cellxgene launch [options] <datafile>` or `make start-server`
- `cellxgene launch [options] <datafile>` or `make start-server`
### Reloading
If you install cellxgene using `make install-dev` the server will be restarted every time you make changes on the server code. If changes affects the client, the browser must be reloaded.
### Linter
@@ -73,41 +99,55 @@ We use [`flake8`](https://github.com/PyCQA/flake8) to lint python and [`black`](
To auto-format code run `make fmt`. To run lint checks on the code run `make lint`.
### Test
If you would like to run the server tests individually, follow the steps below
1. Install development requirements `make dev-env`
1. Run `make unit-test` in the `server` directory or `make unit-test-server` in the root directory.
### Tips
* Install in a virtualenv
* May need to rebuild/reinstall when you make client changes
- Install in a virtualenv
- May need to rebuild/reinstall when you make client changes
## Client dev
### Install
1. Install prereqs for client: `make dev-env`
2. Install cellxgene server as described in the [server install](#install) instructions above.
### Launch
To launch with hot reloading, you need to launch the server and the client separately. Node's hot reloading starts the client on its own node server and auto-refreshes when changes are made to source files.
1. Launch server (the client relies on the REST API being available): `cellxgene launch --debug [other_options] <datafile>` or `make start-server`
2. Launch client: in `client/` directory run `make start-frontend`
3. Client will be served on `localhost:3000`
### Build
To build only the client: `make build-client`
### Linter
We use `eslint` to lint the code and `prettier` as our code formatter.
### Test
If you would like to run the client tests individually, follow the steps below in the `client` directory
1. For unit tests run `make unit-test`
1. For the smoke test run `make smoke-test` for the standard smoke test suite and `make smoke-test-annotations` for the annotations test suite.
If you would like to run the smoke tests against a hot-reloaded version of the client:
1. Start the hot-reloading servers as described in the [Client dev section](#client-dev). If you plan to run the standard test suite (without annotations), you'll have to start the backend server with annotations disabled (e.g. `CXG_OPTIONS='--debug --disable-annotations' make start-server`).
1. From the project root, `cd client`
1. Run either the standard E2E test suite with `CXG_CLIENT_PORT=3000 make e2e` or the annotations test suite with `CXG_CLIENT_PORT=3000 make e2e-annotations`
1. Run either the standard E2E test suite with `npm run e2e` or the annotations test suite with `npm run e2e-annotations`
### Tips
* You can also install/launch the server side code from npm scrips (requires python3.6 with virtualenv) with the `scripts/backend_dev` script.
- You can also install/launch the server side code from npm scrips (requires python3.6 with virtualenv) with the `scripts/backend_dev` script.
- Check out [e2e Tests](e2e_tests.md) for more details
dev_docs/developer_scripts.md
+2 -2
View File
@@ -109,8 +109,8 @@ Methods used to test the client javascript code
`backend-dev` targets. One starts the server, the other runs the tests. If
developing a front-end feature and just checking if tests pass, this is
probabaly the one you want to run.
* `make e2e` Runs backend tests without starting the server. You will need to
* `npm run e2e` Runs backend tests without starting the server. You will need to
start the rest api separately with the pbmc3k.h5ad file. Note you can use
the `JEST_ENV` environment variable to change how JEST runs in the browser.
The test runs against `localhost:5005` by default. You can use the
The test runs against `localhost:3000` by default. You can use the
`CXG_URL_BASE` env variable to test non-localhost deployments of cellxgene.
dev_docs/e2e_tests.md
+67 -26
View File
@@ -1,47 +1,88 @@
## End to end testing thesis
# End to end testing
## Thesis
End to end (e2e) tests, when done correctly, can surface errors and regressions in the application before it is released to production. The tests are run against a version of the web app that is running the full stack. By checking the fully functional app, you are able to surface issues with logic and data at every level of the stack (you still have to figure out which level that error is occurring at). You are also able to test how features that are added affect the whole application.
The major problem most developers have with e2e tests are that they the tests can be flaky and thus not trusted. Failing tests wind up being disabled instead of fixed. To guard against that we need to establish principals about what features to test and have guidelines for writing tests to ensure that they are as robust as possible. By ensuring each test has a high value and is unlikely to fail when it shouldn't it increases the motivation to fix the test as opposed to deleting it.
The major problem most developers have with e2e tests are that the tests can be flaky and thus not trusted. Failing tests wind up being disabled instead of fixed. To guard against that we need to establish principals about what features to test and have guidelines for writing tests to ensure that they are as robust as possible. By ensuring each test has a high value and is unlikely to fail when it shouldn't it increases the motivation to fix the test as opposed to deleting it.
### Choosing what to test:
Additionally, we try to combat flaky tests by re-running failed tests two more times before declaring a test is truly failing. Even though this mediation doesn't solve the root cause of the flakiness, it does help decrease the false positive cases caused by the flakiness.
* Common actions - Features and actions that are highly used will cause more impact if they break
* High value - Features that may be rarely used, but cause a major business impact if they break
* Depend on the whole stack and not just the FE - We want to test the backend and data as well as the FE. Features that only depend on the javascript code should be tested in unit tests instead.
* Feature is in a stable state - If the feature is in active development, it's likely that the test will have to be constantly updated. Wait until it is in a steady state before adding an e2e test.
* Feature is a likely target of regressions - If a feature has caused regressions in the past, it's good to add some monitoring to it.
### Choosing what to test
### Guidelines for writing tests:
- Common actions - Features and actions that are highly used will cause more impact if they break
- High value - Features that may be rarely used, but cause a major business impact if they break
- Depend on the whole stack and not just the FE - We want to test the backend and data as well as the FE. Features that only depend on the javascript code should be tested in unit tests instead.
- Feature is in a stable state - If the feature is in active development, it's likely that the test will have to be constantly updated. Wait until it is in a steady state before adding an e2e test.
- Feature is a likely target of regressions - If a feature has caused regressions in the past, it's good to add some monitoring to it.
* Use data attributes as selectors instead of positional, classname, or id selectors. Add `data-testid` or `data-testclass` attributes to components that are inputs and outputs. The advantages are 1) You can move the component around and even change it's html element type and it will not break the test and 2) It is self documenting what the attribute is used by as opposed to a class name or id that could be used by CSS or something else.
* Use the helper functions in `puppeteerUtils.js` and ` for performing common actions. Puppeteer operates at a low level and these abstract it to a level of user actions.
### Guidelines for writing tests
- Use data attributes as selectors instead of positional, classname, or id selectors. Add `data-testid` or `data-testclass` attributes to components that are inputs and outputs. The advantages are 1) You can move the component around and even change its html element type and it will not break the test and 2) It is self documenting what the attribute is used by as opposed to a class name or id that could be used by CSS or something else.
- Use the helper functions in `puppeteerUtils.js` and `cellxgeneActions.js` for performing common actions. Puppeteer operates at a low level and these abstract it to a level of user actions.
- We use [`expect-puppeteer`](https://github.com/smooth-code/jest-puppeteer/tree/master/packages/expect-puppeteer#api)'s API to avoid directly using Puppeteer's API where possible. So **please only resort to Puppeteer's API if you cannot achieve an interaction you need via `expect-puppeteer`.**
For actions that don't meet these guidelines, you can create a test in feature.test.js that does not run on PRs so if the feature breaks it doesn't block anyone.
## Where/How
Where: client/\_\_tests\_\_/e2e/e2e.test.js
How: `npm run e2e`
How: `npm run e2e` or `make e2e`
Tests are written using the Jest testing framework. Browser automation is handled by puppeteer. Puppeteer provides an API to control Chrome over the DevTools Protocol. This allows us to write code that mimics user actions and check page state.
Tests are written using the Jest testing framework. Browser automation is handled by Puppeteer. Puppeteer provides an API to control Chrome over the DevTools Protocol. This allows us to write code that mimics user actions and check page state.
Tests are running on CI with every PR, they can also be run locally (see [developer guidelines](developer_guidelines.md)).
Tests are running on CI with every PR, they can also be run locally (see [developer guidelines](developer_guidelines.md)).
The cellxgene instance is running with the pbmc3k dataset. Tests for additional datasets can be added, the data.js file contains the values for inputs and outputs expected for different datasets. Choose values that make sense for that dataset.
The cellxgene instance is running with the `pbmc3k` dataset. Tests for additional datasets can be added, the `data.js` file contains the values for inputs and outputs expected for different datasets. Choose values that make sense for that dataset.
## Setting up e2e tests locally
See [developer guidelines](developer_guidelines.md)
## How to write a new test
1. Ensure that your new test meets the guidelines above for high-value robust user actions to test.
2. Clearly define the series of user actions and outputs you want to test
3. Add testid (and/or testclass if element is not unique) data attributes to inputs and outputs you want to interact with.
4. Create a new test block possible nested in a describe block if you want to add more than one related test client/\_\_tests\_\_/e2e/e2e.test.js
5. Write the test code.
6. Run tests locally to ensure they pass.
**Debugging Tips**
* Mystery bugs are often caused by a missing `async` or `await`. Double check every function and every line (twice!).
* Switch from to debug mode by running `export JEST_ENV='debug'` on the CLI before running the tests
* Put a javascript breakpoint in by adding this line `await jestPuppeteer.debug()` to the test at the point which you want to pause execution.
* Limit the tests that are run by adding `.only` after `describe` or `test` to limit execution to just that block of tests. (Remember to take the `.only` out before you commit!)
1. Ensure that your new test meets the guidelines above for high-value robust user actions to test.
1. Clearly define the series of user actions and outputs you want to test
1. Add `testid` (and/or `testclass` if element is not unique) data attributes to inputs and outputs you want to interact with.
E.g., `<div data-testid="test-id" />` or `<div data-testclass="test-class" />`
1. Create a new test block possible nested in a describe block if you want to add more than one related test client/\_\_tests\_\_/e2e/e2e.test.js
1. Write the test code.
1. Run tests locally to ensure they pass.
## Debugging Tips
- Mystery bugs are often caused by a missing `async` or `await`. Double check every function and every line (twice!).
- Use `ndb` for debugging: Put a javascript breakpoint in by adding this line `debugger` to the test at the point which you want to pause execution. And then run `ndb npm run e2e` or `ndb make e2e`.
NOTE: You will need to global install [`ndb`](https://github.com/GoogleChromeLabs/ndb) via `npm install -g ndb`, which provides an improved debugging experience for Node.js, enabled by Chrome DevTools
- Limit the tests that are run by adding `.only` after `describe` or `test` to limit execution to just that block of tests. (Remember to take the `.only` out before you commit!)
- Avoid using `beforeEach() => {}` to set up test environment for each test, since Jest/Jasmine doesn't fail a test when a set up step fails, and we will end up getting obscure errors and don't know why a test fails. Another reason is for readability as explained [here](https://kentcdodds.com/blog/avoid-nesting-when-youre-testing/)
- Use [`takeScreenshot()`](../client/__tests__/e2e/takeScreenshot.js) to timestamp and store screenshots in the already git ignored folder `client/__tests__/screenshots/`, so you don't accidentally check in your screenshot in your commit
## Config Files
1. [jest-puppeteer.config.js](../client/jest-puppeteer.config.js) is for configuring Puppeteer's launch options as specified [here](https://github.com/puppeteer/puppeteer/blob/v3.1.0/docs/api.md#puppeteerlaunchoptions). Note: The link is to the doc for **v3.1.0**
1. [e2eJestConfig.json](../client/__tests__/e2e/e2eJestConfig.json) is for configuring `jest` when running e2e tests. For example, in `package.json` we have:
```ts
{
"e2e": "jest --config __tests__/e2e/e2eJestConfig.json e2e/e2e.test.js",
}
```
which tells `jest` to use `e2eJestConfig.json` as the config file to run e2e test file `e2e.test.js`
1. [puppeteer.setup.js](../client/__tests__/e2e/puppeteer.setup.js) is for configuring `jest`, `browser`, and `page` objects at runtime