Datasource Paginated Query

Impure

Executes a SQL SELECT query in pages and loops until all results are fetched. The SQL must contain {{LIMIT}} and {{OFFSET}} placeholders where page size and offset should be applied.

Pure Function

No

Execute Inputs

Execution

INPUT

Execution Input.

Execute Outputs

Loop Body

OUTPUT

Executed for each page of results.

Loops

Complete

OUTPUT

Fires when the data is successfully inserted into the cube.

Data Inputs

Datasource

INPUT

Name of the datasource to use.

SQL Query

INPUT

The SQL SELECT query to execute. Must include {{LIMIT}} and {{OFFSET}} placeholders so pagination can be applied. Example (MySQL/Postgres): SELECT * FROM table ORDER BY id LIMIT {{LIMIT}} OFFSET {{OFFSET}}

Parameters

INPUT

Optional list of parameters to substitute for ? placeholders.

Data Outputs

Page Index

OUTPUTinteger

Current page index.

Multiple

Rows

OUTPUTarray

List of rows returned by the query.

Multiple

Settings

Throw on SQL Error

BOOLEAN

If true, will throw an error if a SQL error occurs during query.

Default: true

Page Size

INTEGER

Number of rows to fetch per page.

Default: 500

Debug

STRING

Break on node when executing.

Default: No

Options:

NoBeforeInputsBeforeProcessingAfterProcessing