Skip to content

stream pipelining - #3095

Merged
mangas merged 2 commits into
masterfrom
filipe/stream-pipeline
Jan 17, 2022
Merged

stream pipelining#3095
mangas merged 2 commits into
masterfrom
filipe/stream-pipeline

Conversation

@mangas

@mangas mangas commented Dec 22, 2021

Copy link
Copy Markdown
Contributor

No description provided.

@mangas
mangas force-pushed the filipe/stream-pipeline branch 3 times, most recently from 3d2a5ee to 0870ece Compare December 23, 2021 13:36

@leoyvens leoyvens left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A more encapsulated approach which I'd suggest is to introduce a BufferedBlockStream, which takes as input a block stream and a buffer size, and returns

Comment thread core/src/subgraph/instance_manager.rs
Comment thread graph/src/blockchain/block_stream.rs Outdated
@mangas
mangas force-pushed the filipe/stream-pipeline branch 2 times, most recently from 1bacbbe to 72d5aaf Compare December 23, 2021 17:02
@mangas
mangas marked this pull request as ready for review December 23, 2021 17:27
@mangas
mangas force-pushed the filipe/stream-pipeline branch 2 times, most recently from e518cf7 to abf15a3 Compare December 31, 2021 00:50
@mangas
mangas requested a review from leoyvens December 31, 2021 00:50
@mangas
mangas force-pushed the filipe/stream-pipeline branch 3 times, most recently from 3f6b299 to cc1e71a Compare January 5, 2022 16:52

@evaporei evaporei left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, still I think it's worth to have @leoyvens take a last look

@mangas
mangas force-pushed the filipe/stream-pipeline branch 10 times, most recently from 126af66 to 4e5652b Compare January 10, 2022 11:23
Comment thread core/src/subgraph/instance_manager.rs Outdated
Comment thread core/src/subgraph/instance_manager.rs Outdated
Comment thread core/src/subgraph/instance_manager.rs Outdated
Comment thread core/src/subgraph/instance_manager.rs Outdated
Comment thread graph/src/blockchain/block_stream.rs
Comment thread graph/src/blockchain/block_stream.rs
Comment thread graph/src/blockchain/block_stream.rs
Comment thread graph/src/blockchain/mock.rs Outdated
Comment thread runtime/wasm/src/host_exports.rs Outdated
Comment thread graph/src/blockchain/mod.rs Outdated
@mangas
mangas force-pushed the filipe/stream-pipeline branch 2 times, most recently from a6013a8 to ded67bf Compare January 13, 2022 16:28
use tokio::task;

const MINUTE: Duration = Duration::from_secs(60);
const BUFFERED_BLOCK_STREAM_SIZE: usize = 100;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BUFFERED_POLLING_BLOCK_STREAM_SIZE would be more accurate.

@mangas
mangas force-pushed the filipe/stream-pipeline branch from ded67bf to 448970a Compare January 14, 2022 15:15
@mangas
mangas force-pushed the filipe/stream-pipeline branch 3 times, most recently from 76f8966 to b0802c4 Compare January 17, 2022 14:40
@mangas
mangas force-pushed the filipe/stream-pipeline branch from b0802c4 to 74cd3b7 Compare January 17, 2022 14:47
@mangas
mangas merged commit 6f348ad into master Jan 17, 2022
@mangas
mangas deleted the filipe/stream-pipeline branch January 17, 2022 15:19
@leoyvens leoyvens mentioned this pull request Mar 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants