bmccandless 5c0b8c6296 Improve diffexp for tiledb (#1388) ...

* Improve diffexp for tiledb

- The rows from the A and B sets are gathered and processed at the same time.  In this
  way the matrix is only accessed once instead of twice for each tile.
- There is now a single thread queue that gets shared between all callers of the diffexp.
  This will slow down work if diffexp gets too busy.
- There is a target_workunit amount of work given to each thread.  Previously the
  workunit was (rows selected * width of tile), which could be small.  Now multiple
  column tiles can be combined into one workunit.  If the target is too small then
  thread and other overheads may reduce performance.  If target_workunit is too large
  then the size of the gathered sub matrix may take up too much memory.
- add configuration parameters (max_workers, cpu_multiplier, and  target_workunit)

2020-04-13 18:53:13 -07:00

1.2 KiB

Raw Blame History

View Raw