Adds trough dedup performance rework#388

Draft

adam-miller wants to merge 18 commits into

master-ait-contribfrom

adds-trough-dedup-performance-rework

Commits on Jan 28, 2021

Fixing synchronization issues in TroughCrawlLogFeed.java to reduce blocking
adam-miller
committed

Commits on Feb 2, 2021

Trying to avoid hard locks on updating url caches
adam-miller
committed

Commits on Feb 18, 2021

Fix error log reporting of batch size for trough crawl logs
adam-miller
committed

Commits on Feb 23, 2021

Commits on Feb 25, 2021

Commits on Feb 26, 2021

Switching to in memory sqlite for dedup cache
adam-miller
committed

Commits on Mar 10, 2021

Commits on Apr 5, 2021

Commits on May 13, 2021

Refactor crawl log batch posting. Add configuration options for trough dedup batches. Fix posting of trough dedup at end of crawl.
adam-miller
committed

Commits on May 17, 2021

Ensure we set dedup schema when writing dedup shards
adam-miller
committed

Commits on May 20, 2021

Add dedup load from in-memory dedup cache db
adam-miller
committed

Commits on May 28, 2021

Refactor Trough client URL cache put. Limit TroughContentDigestHistory query.
adam-miller
committed