Allow creation of new connections for each parallel execution in Bulk to Synapse type flows.

3 votes

Currently, in Nested flows, when executing consecutive SQL Script flows with "Parallel" nested flow option turned on, new connection is created for each SQL Script executed. I presume this is due to the connection being created in the context of the flow, and running them in parallel creates the need to have separate connections.
However, In Bulk To Synapse type flows, with Parallel enabled for each transformation, only one connection is used for all transformations. 1 flow - 1 connection, even though there is Parallel execution targeting the database, like consecutive SQL scripts.

This can create a "softlock" type of queueing in BEFORE/AFTER COPY scripts, namely procedure execution, as Procedure execution in Synapse Analytics gets queued, if the same procedure is already being executed under the same connection.

An option to enable creating a new connection for each transformation, would solve this "softlock" type situation.

This could also have specific uses in other flow types.

Done Suggested by: Heiko Upvoted: 05 Jan, '23 Comments: 1

Comments: 1