Refactor: thread pool using #1986

xDimon · 2024-02-14T13:21:07Z

Referenced issues

Description of the Change

Benefits

Possible Drawbacks

Usage Examples or Tests

Alternate Designs

core/application/app_state_manager.hpp

turuslan · 2024-02-14T13:45:11Z

core/network/warp/cache.hpp

    outcome::result<WarpSyncProof> getProof(
        const primitives::BlockHash &after_hash) const;

    void warp(const primitives::BlockInfo &block);

   private:
    outcome::result<void> cacheMore(primitives::BlockNumber finalized);
-    outcome::result<void> start();


Suggested change

outcome::result<void> start();

outcome::result<void> startOutcome();

core/utils/watchdog.hpp

turuslan · 2024-02-14T13:53:22Z

test/core/consensus/babe/babe_test.cpp

+                                  main_pool_handler,
+                                  worker_pool_handler);


Suggested change

main_pool_handler,

worker_pool_handler);

TestThreadPool{io_},

io_);

https://github.com/qdrvm/kagome/pull/1979/files#diff-9acad5d8bb06a51b2416a01eab196a89f36afb5b2c711a3936c24153ecc9693dR228-R229

It wont work 'cause arguments have a strict type, and TestThreadPool is not derived of them

It works in that branch (also strong type, with constructor(TestThreadPool))

turuslan · 2024-02-14T14:05:56Z

core/utils/pool_handler.hpp

+        : is_active_{false}, ioc_{std::move(io_context)} {}
+    ~PoolHandler() = default;


Must be constructed as "active",
or will lose callbacks before AppStateManager calls start()

Suggested change

: is_active_{false}, ioc_{std::move(io_context)} {}

~PoolHandler() = default;

: is_active_{true}, ioc_{std::move(io_context)} {}

Making it active is responsibility of using class. To avoid callback loosing, class-owner should start pool handler at 'prepare' phase.

Should we fix PoolHandler { is_active_ = true } constructor,
or each class constructor Foo(PoolHandler pool_handler) { pool_handler.start() }?

turuslan · 2024-02-14T14:45:08Z

core/network/impl/router_libp2p.cpp

-    app_state_manager_->takeControl(*statement_fetching_protocol_.get());
-
-    app_state_manager_->takeControl(*send_dispute_protocol_.get());
+    main_pool_handler_->execute([wp = weak_from_this()] {


missing setProtocolHandler

Suggested change

main_pool_handler_->execute([wp = weak_from_this()] {

REINVOKE(main_pool_handler_)

turuslan · 2024-02-14T14:47:21Z

core/telemetry/impl/service_impl.cpp

    if (enabled_) {
-      message_pool_ = std::make_shared<MessagePool>(
-          kTelemetryMessageMaxLengthBytes, kTelemetryMessagePoolSize);
      app_state_manager_->takeControl(*this);
    } else {
      SL_INFO(log_, "Telemetry disabled");
    }
  }

  bool TelemetryServiceImpl::prepare() {


Suggested change

bool TelemetryServiceImpl::prepare() {

bool TelemetryServiceImpl::prepare() {

REINVOKE(io_context_)

turuslan · 2024-02-14T14:48:11Z

core/telemetry/impl/service_impl.cpp

    return true;
  }

  bool TelemetryServiceImpl::start() {


Suggested change

return true;

}

bool TelemetryServiceImpl::start() {

Signed-off-by: Dmitriy Khaustov aka xDimon <[email protected]>

…hread pool) Signed-off-by: Dmitriy Khaustov aka xDimon <[email protected]>

Signed-off-by: Dmitriy Khaustov aka xDimon <[email protected]>

core/network/beefy/beefy_impl.cpp

Signed-off-by: Dmitriy Khaustov aka xDimon <[email protected]>

core/network/impl/router_libp2p.hpp

test/mock/core/network/beefy_protocol_mock.hpp

Signed-off-by: Dmitriy Khaustov aka xDimon <[email protected]>

xDimon requested review from turuslan and iceseer February 14, 2024 13:21

turuslan approved these changes Feb 14, 2024

View reviewed changes

turuslan reviewed Feb 14, 2024

View reviewed changes

iceseer approved these changes Feb 19, 2024

View reviewed changes

xDimon added 25 commits February 20, 2024 16:41

feature: main thread pool

f0be68b

Signed-off-by: Dmitriy Khaustov aka xDimon <[email protected]>

refactor: remove refactor main_runner thread (will replaced by main t…

039f6fa

…hread pool) Signed-off-by: Dmitriy Khaustov aka xDimon <[email protected]>

refactor: use main thread pool in babe

ccf6741

Signed-off-by: Dmitriy Khaustov aka xDimon <[email protected]>

refactor: use main thread pool in grandpa

370f7e5

Signed-off-by: Dmitriy Khaustov aka xDimon <[email protected]>

refactor: use main thread pool in dispute coordinator

f4a54cc

Signed-off-by: Dmitriy Khaustov aka xDimon <[email protected]>

refactor: use main thread pool in grandpa environment

13e7b19

Signed-off-by: Dmitriy Khaustov aka xDimon <[email protected]>

refactor: use main thread pool in approval distribution

7d92b96

Signed-off-by: Dmitriy Khaustov aka xDimon <[email protected]>

refactor: use main thread pool in verified justification queue

ba2e0eb

Signed-off-by: Dmitriy Khaustov aka xDimon <[email protected]>

refactor: use main thread pool in block tree

dc11788

Signed-off-by: Dmitriy Khaustov aka xDimon <[email protected]>

feature: warp for thread handler

06cea7e

Signed-off-by: Dmitriy Khaustov aka xDimon <[email protected]>

refactor: use main thread pool in parachain processor

cc815a3

Signed-off-by: Dmitriy Khaustov aka xDimon <[email protected]>

refactor: use main thread pool in tx protocol

2c59c31

Signed-off-by: Dmitriy Khaustov aka xDimon <[email protected]>

refactor: use main thread pool in synchronizer

da3fb1d

Signed-off-by: Dmitriy Khaustov aka xDimon <[email protected]>

refactor: use main thread pool in block executor

5649f46

Signed-off-by: Dmitriy Khaustov aka xDimon <[email protected]>

fix: start main thread handler there lost

3740b1b

Signed-off-by: Dmitriy Khaustov aka xDimon <[email protected]>

feature: worker thread pool handler

4f8dbda

Signed-off-by: Dmitriy Khaustov aka xDimon <[email protected]>

feature: main thread pool handler

64efa05

Signed-off-by: Dmitriy Khaustov aka xDimon <[email protected]>

refactor: improve app state manager

86e75bd

Signed-off-by: Dmitriy Khaustov aka xDimon <[email protected]>

refactor: use worker pool handler in babe

ae7d729

Signed-off-by: Dmitriy Khaustov aka xDimon <[email protected]>

refactor: use worker pool handler in approval

11f8f60

Signed-off-by: Dmitriy Khaustov aka xDimon <[email protected]>

refactor: use worker pool handler in block executor

713b4cc

Signed-off-by: Dmitriy Khaustov aka xDimon <[email protected]>

refactor: use worker pool handler in parachain processor

0e770a9

Signed-off-by: Dmitriy Khaustov aka xDimon <[email protected]>

refactor: use worker pool handler in warp sync

68e8591

Signed-off-by: Dmitriy Khaustov aka xDimon <[email protected]>

refactor: inject main pool handler instead of main thread pool

2f59c31

Signed-off-by: Dmitriy Khaustov aka xDimon <[email protected]>

refactor: thread management in approval

436d700

Signed-off-by: Dmitriy Khaustov aka xDimon <[email protected]>

xDimon added 7 commits February 20, 2024 17:20

+ beefy mock

b75ba06

Signed-off-by: Dmitriy Khaustov aka xDimon <[email protected]>

feature: setup number of workers for tests purposes

3b90748

Signed-off-by: Dmitriy Khaustov aka xDimon <[email protected]>

fix: wrong named test

ef7e5c3

Signed-off-by: Dmitriy Khaustov aka xDimon <[email protected]>

fix: tests

61a9a7f

Signed-off-by: Dmitriy Khaustov aka xDimon <[email protected]>

fix: start protocols in main thread pool

92c3ae8

Signed-off-by: Dmitriy Khaustov aka xDimon <[email protected]>

feature: special thread pool for telemetry

a27ee9a

Signed-off-by: Dmitriy Khaustov aka xDimon <[email protected]>

fix: review issues

629e7b3

Signed-off-by: Dmitriy Khaustov aka xDimon <[email protected]>

xDimon force-pushed the refactor/thread_pool_using branch from a1dd273 to 629e7b3 Compare February 20, 2024 14:50

turuslan reviewed Feb 20, 2024

View reviewed changes

core/network/beefy/beefy_impl.cpp Outdated Show resolved Hide resolved

xDimon added 11 commits February 21, 2024 00:20

feature: ctor of main and workers pools for purposes of test

a69fae5

Signed-off-by: Dmitriy Khaustov aka xDimon <[email protected]>

fix: babe test

9721796

Signed-off-by: Dmitriy Khaustov aka xDimon <[email protected]>

fix: block executor test

7e4a457

Signed-off-by: Dmitriy Khaustov aka xDimon <[email protected]>

feature: beefy thread pool

201163e

Signed-off-by: Dmitriy Khaustov aka xDimon <[email protected]>

feature: use beefy thread pool

1834bbe

Signed-off-by: Dmitriy Khaustov aka xDimon <[email protected]>

fix: beefy test

6a200c2

Signed-off-by: Dmitriy Khaustov aka xDimon <[email protected]>

fix: naming and source location of beefy parts

06a275c

Signed-off-by: Dmitriy Khaustov aka xDimon <[email protected]>

git: Merge branch 'master' into refactor/thread_pool_using

35bff35

Signed-off-by: Dmitriy Khaustov aka xDimon <[email protected]>

fix: remove duplicated files

321c114

Signed-off-by: Dmitriy Khaustov aka xDimon <[email protected]>

refactor: replace various includes by forward declarations

979782d

Signed-off-by: Dmitriy Khaustov aka xDimon <[email protected]>

fix: tests

180f72a

Signed-off-by: Dmitriy Khaustov aka xDimon <[email protected]>

turuslan reviewed Feb 22, 2024

View reviewed changes

core/network/impl/router_libp2p.hpp Outdated Show resolved Hide resolved

turuslan reviewed Feb 22, 2024

View reviewed changes

test/mock/core/network/beefy_protocol_mock.hpp Outdated Show resolved Hide resolved

xDimon added 5 commits February 22, 2024 16:54

git: Merge branch 'master' into refactor/thread_pool_using

2b5390b

Signed-off-by: Dmitriy Khaustov aka xDimon <[email protected]>

fix: review issue

4a81e6e

Signed-off-by: Dmitriy Khaustov aka xDimon <[email protected]>

fix: mismatched-tags warning

ed76871

Signed-off-by: Dmitriy Khaustov aka xDimon <[email protected]>

fix: block executor test

0ebb5a1

Signed-off-by: Dmitriy Khaustov aka xDimon <[email protected]>

git: Merge branch 'master' into refactor/thread_pool_using

b37b335

Signed-off-by: Dmitriy Khaustov aka xDimon <[email protected]>

xDimon enabled auto-merge (squash) February 28, 2024 12:55

xDimon merged commit 69e08b4 into master Feb 28, 2024
8 of 10 checks passed

xDimon deleted the refactor/thread_pool_using branch February 28, 2024 13:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor: thread pool using #1986

Refactor: thread pool using #1986

xDimon commented Feb 14, 2024

turuslan Feb 14, 2024

xDimon Feb 14, 2024

turuslan Feb 14, 2024

xDimon Feb 14, 2024

turuslan Feb 15, 2024

turuslan Feb 14, 2024

xDimon Feb 14, 2024

turuslan Feb 16, 2024

turuslan Feb 14, 2024

turuslan Feb 14, 2024

turuslan Feb 14, 2024

	outcome::result<void> start();
	outcome::result<void> startOutcome();

		: is_active_{false}, ioc_{std::move(io_context)} {}
		~PoolHandler() = default;

	: is_active_{false}, ioc_{std::move(io_context)} {}
	~PoolHandler() = default;
	: is_active_{true}, ioc_{std::move(io_context)} {}

	main_pool_handler_->execute([wp = weak_from_this()] {
	REINVOKE(main_pool_handler_)

	bool TelemetryServiceImpl::prepare() {
	bool TelemetryServiceImpl::prepare() {
	REINVOKE(io_context_)

Refactor: thread pool using #1986

Refactor: thread pool using #1986

Conversation

xDimon commented Feb 14, 2024

Referenced issues

Description of the Change

Benefits

Possible Drawbacks

Usage Examples or Tests

Alternate Designs

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment