Build view combining raw data and incremental rollup

[cleaton]

Hi, thanks for the great extension.

is it possible to build pipelines in a way that enables “real-time” aggregates? For example by combining latest raw data and pre-aggregates:

‘’’
select * from view_counts where ts <= last_incremental_update
union all
select (aggregates that created view_counts) from raw_views where ts > last_incremental_update
‘’’

[marcoslot]

You could join with the incremental.sequence_pipelines or incremental.time_interval_pipelines tables to obtain the last_incremental_update value. You typically only need to filter the raw data, since the view_counts is not going to contain anything more recent than that.

[cleaton]

Is it ok to replace the view as part of the incremental transaction or is that not recommended / supported? I’m even thinking if it would be possible to have it as an optional feature to build realtime view that hides the internal incremental process latency.

[marcoslot]

I guess it's possible, might have some locking implications. Definitely an interesting feature to think about. Maybe a stable function that returns the last processed value already goes a long way, as long as we can use it with an index scan.