Suggestions for improving Etlworks Integrator.

Packages and versioning

It would be nice to create a "package" of flows and connections and store it as a certain version. It should then be easy to switch between versions just by toggling ...
Suggested by: Franck van der Sluis (19 Apr, '19) Upvoted: 12 Aug, '20 Comments: 3
Done roadmap

2FA Login

Our security policy will require all tools that have access to company data, to have 2FA required to login. Currently a user can login using only the username and ...
Suggested by: Franck (06 Jun, '19) Upvoted: 03 Apr, '20 Comments: 0

The ability to manage tags: edit, delete

Suggested by: Rupesh Sawant (23 Jun, '20) Upvoted: 12 Sep Comments: 0
Planned roadmap

SSO Authentication

Etlworks handles and processes a lot of sensitive data. It is important that this is adequately shielded and becomes part of our company-wide security umbrella. The ...
Suggested by: Joris Hooi (24 Oct, '19) Upvoted: 05 May, '22 Comments: 1
Done roadmap

New fields' mapping editor

Enhance field mapping editor: - add support for nested fields in the destination - add an ability to modify attributes, such as JSON array, XML CDATA, XML ...
Suggested by: Etlworks (06 Dec, '18) Upvoted: 14 Jul Comments: 0
Planned roadmap

Drag and drop process builder for nested flows

Replace existing nested flow editor with drag and drop process builder: - works similarly to the diagram editor - ability to select existing flows from the side-bar ...
Suggested by: Etlworks (06 Dec, '18) Upvoted: 14 Jul Comments: 0
Planned roadmap

Email on schedule failure only if last retry

I don't want emails for transient errors if a retry worked
Suggested by: Bryan (09 Aug, '22) Upvoted: 12 Oct, '22 Comments: 2

Ability to schedule and execute flows by data integration agents running behind the firewall

Basically, add support for hybrid data integration when flows can be executed on data integration agents running behind the firewall.
Suggested by: Olena Sherbinin (23 Mar, '19) Upvoted: 27 Dec, '21 Comments: 1
Done roadmap

Usability enhancement for Flows Builder

1. Ability to edit inner flow from the nested flow 2. Ability to stop API flow, also stop flow from scheduler 3. Add Find Usage for flows (within nested flows)
Suggested by: Maksym (03 Oct, '19) Upvoted: 13 Apr Comments: 0
Planned roadmap

Better ways to determine how flows, connections, etc. are interconnected

Right now when you have multiple flows that share connections or other flows, it's difficult to understand how making a change to one will possibly impact other ...
Suggested by: Shanon (07 Jul, '19) Upvoted: 28 Sep, '19 Comments: 5
Done roadmap

Implement priority queue and throttling for user-defined APIs (listeners)

Scenario 1: upstream system slammed the system with thousands of API calls in a short span of time, eventually freezing and requiring a reboot to stop all jobs ...
Suggested by: NBCUniversal (07 Nov, '20) Upvoted: 08 Mar, '21 Comments: 3
Done roadmap

IBM i Debezium Connector

Hello, would it be possible to add IBM i/AS400 Debezium Connector capabilities via this repo please?
Suggested by: Ryan Cronin (01 Feb) Upvoted: 22 Sep Comments: 1
Done roadmap

Provisioning from AWS and Azure Marketplaces

Add ability to provision Etlworks from AWS and Azure Marketplaces with Amazon and Microsoft billing
Suggested by: Etlworks (06 Dec, '18) Upvoted: 13 Jan Comments: 0
Planned roadmap

Connector for Google Analytics 4,

Since Google is discontinuing Universal Analytics and moving to only supporting Google Analytics 4,
Suggested by: Justin Viegelahn (18 Feb) Upvoted: 18 Feb Comments: 1
Done roadmap

AWS Kinesis Integration

AWS Kinesis Integration
Suggested by: Karthik (08 Jun, '20) Upvoted: 14 Jul, '20 Comments: 1
Done roadmap

Search functionality / better documentation of flows

We use a good deal of javascript and in complex flows it can be really difficult to debug, esp. global and flow variables because there's no way to search (that I ...
Suggested by: Shanon (31 Oct, '19) Upvoted: 01 Apr Comments: 1
Done roadmap

(re-)Run from error screen

If investigate an issue within ETLworks. You go to the error (via status) to the statistics section. When solving the issue, you have to search for the flow in order ...
Suggested by: Marvin (27 Feb) Upvoted: 14 Jun Comments: 0
Under consideration

Add an apply to all button when editing a transform

Apply the settings for that transform to all the other transforms in the current flow
Suggested by: Bryan (11 Mar, '20) Upvoted: 23 Apr, '21 Comments: 0
Planned roadmap

Drag&drop SQL query builder

Suggested by: Olena Sherbinin (13 Mar, '19) Upvoted: 12 May Comments: 0
Planned roadmap

Auto-completion in JavaScript code editors

Add auto-completion in JavaScript code editors: - {field names} - functions
Suggested by: Etlworks (06 Dec, '18) Upvoted: 24 May, '22 Comments: 0
Planned roadmap

Scheduler queue

Introduce queuing in scheduler so you can set how many scheduled tasks can execute simultaneously, and if one tries to run while another is running it will wait till ...
Suggested by: Bryan (18 Mar, '20) Upvoted: 12 Oct, '22 Comments: 2
Done roadmap

Add support for JMS, ActiveMQ and AWS SQS

Suggested by: Etlworks (03 Oct, '19) Upvoted: 14 Jul, '21 Comments: 2
Done roadmap

Allow creation of new connections for each parallel execution in Bulk to Synapse type flows.

Currently, in Nested flows, when executing consecutive SQL Script flows with "Parallel" nested flow option turned on, new connection is created for each SQL Script ...
Suggested by: Heiko (14 Sep, '22) Upvoted: 05 Jan Comments: 1

Bulk insert with db to db flows

Having inserts into postgres and sqlserver work like redshift would be a nice improvement. Right now it runs a bunch of insert statements. Instead, pull source data ...
Suggested by: Bryan Meyerovich (10 Oct, '20) Upvoted: 15 Nov, '20 Comments: 1
Done roadmap

Add ability to resume a sync from where it might have failed, or was stopped/canceled

Sometimes syncs fail for whatever reasons, but being able to easily resume from the flows that were not completed would be fantastic (assuming we separately addressed ...
Suggested by: Graham (09 Oct, '19) Upvoted: 11 Oct, '19 Comments: 1

Import entire schemes without specifying individual table

I really liked how aws dms operated, you pointed it at a schema on your source, told it the target schema in your destination, and had a wildcard supporting ...
Suggested by: Bryan (24 Jul, '19) Upvoted: 28 Sep, '19 Comments: 1
Done roadmap

UI Suggestion: Flow execution/failure duration metrics

Currently, some flows only have a console log of events that happen during flow execution. It would be useful to have a graphical representation, especially nested ...
Suggested by: Drew Thompson (17 Jan, '19) Upvoted: 08 Jan, '21 Comments: 1

FLOWs visual based on tree hierarchy

When having multiple flows, it would be nice if you can collapse actions into the nested flows (like a tree). Result then, only nested flows will be shown, until ...
Suggested by: Marvin (02 Jun, '22) Upvoted: 03 Jun, '22 Comments: 0
Under consideration

Add interactive Azure Active directory authentication for SQL Server connector

SQL Server JDBC driver supports accessToken authentication which requires a web flow for signing in with Microsoft. It would be great if Etlworks connector could ...
Suggested by: Olena Sherbinin (12 Oct, '21) Upvoted: 18 Oct, '21 Comments: 0

Support IAM For S3 Authentication

We run ETL Works on a self-hosted EC2 server. To be able to connect to AWS S3 from ETL Works, it currently requires an access key and secret key. Our organisation ...
Suggested by: Mark (13 Sep, '21) Upvoted: 14 Sep, '21 Comments: 0

Allow setting flow parameter in explorer

Made a api connector with a {param1} in the url, want to do this in explorer: select * from content where param1=123
Suggested by: Bryan (31 Aug, '21) Upvoted: 26 Sep, '21 Comments: 0
Planned roadmap

View of schedules across all tenants + Last execution duraton

A view were all schedules across all tenants are visible with additional column with last execution duration will help balance load across instance. It will help pick ...
Suggested by: Jaroslaw (20 Nov, '20) Upvoted: 07 Jul Comments: 0
Planned roadmap

Allow setting flow variables in scheduler

Suggested by: Bryan (27 Mar, '20) Upvoted: 05 Oct, '20 Comments: 0
Done roadmap

Add a next/previous transform button when editing transform details

Easily transition between transforms to more quickly apply a change to all transforms without losing your place in the list going back and fourth
Suggested by: Bryan (19 Mar, '20) Upvoted: 07 Jul Comments: 0
Planned roadmap

Allow deleting multiple transforms from a single flow

Allow deleting multiple transforms from a single flow to make it easier to duplicate a large flow and only keep a small subset of tables from it.
Suggested by: Bryan (09 Mar, '20) Upvoted: 19 Mar, '20 Comments: 0
Planned roadmap

Statistics Page Should include Agent Metrics too

Currently Statistics page does not include metrics for the flows which are executed by Agent... it would make dashboard much better if we can see the whole picture ...
Suggested by: Atul Kapoor (09 May, '22) Upvoted: 09 May, '22 Comments: 3
Done roadmap

Add run schedule api endpoint

I'd like to run schedule job via api instead of just flow. This way I can take advantage of the scheduler's extra options like expected flow execution time. Pass ...
Suggested by: Bryan Meyerovich (07 Aug, '20) Upvoted: 07 Aug, '20 Comments: 2

Recreating table for HWM flow after data is populated

On a non-HWM transform with new source columns, the destination table isn’t dropped until after the data is populated. On a HWM transform it drops it before data ...
Suggested by: Bryan Meyerovich (03 Aug, '20) Upvoted: 05 May, '22 Comments: 2
Under consideration

Allow for table list alphabetization in explorer

Expanded salesforce tables in explorer, table names are not alphabetized, makes it hard to find anything
Suggested by: Bryan Meyerovich (10 Jun, '20) Upvoted: 10 Jun, '20 Comments: 2
Done roadmap

Add button to restart service

Sometimes there is a need to restart ETL works service, it would be nice to have a button or something like that in the interface
Suggested by: Jarmo Liivak (09 Jun, '22) Upvoted: 20 Jul, '22 Comments: 1
Not planned

Add ability to generate new Google Sheets

I'm looking for a way to generate new Google Sheets from ETLworks. Or at minimum, use the Google Drive connector to create a new file, then the Google Sheets ...
Suggested by: Frank (30 Nov, '21) Upvoted: 30 Nov, '21 Comments: 1
Done roadmap

Parallel CDC snapshot

Currently, CDC flows run snapshots sequentially, which for large databases can be a bottleneck. It would be great to have an option to run snapshots in parallel, each ...
Suggested by: Olena Sherbinin (23 Sep, '21) Upvoted: 23 Sep, '21 Comments: 1
Not planned

support parameterized macros

allow passing params directly to macros when referencing them {{4:MacroName}('MySpecialParam')} or something like that
Suggested by: Bryan (11 Sep, '21) Upvoted: 13 Sep, '21 Comments: 1
Not planned

Allow setting connections per transform instead of per flow for bulk flows

In my use case I can put my transforms all in one flow, instead of one transform per flow and putting all of them into another nested flow
Suggested by: Bryan (25 Aug, '21) Upvoted: 26 Aug, '21 Comments: 1

Add Create table SQL field to bulk flows

Suggested by: Bryan (25 Aug, '21) Upvoted: 26 Aug, '21 Comments: 1

Support for RabbitMQ or AMQP protocol as source/destination

Suggested by: Ryan (29 Jul, '21) Upvoted: 29 Jul, '21 Comments: 1

Change View Schedules Icon if all schedules are disabled

I disabled some schedules and kept some active and realized when i sorted by "Schedules" in the flow list screen they all look the same
Suggested by: Bryan (23 Jun, '21) Upvoted: 24 Jun, '21 Comments: 1

Feature request: smb storage connector through ssh bastion

Suggested by: Bryan (27 Apr, '21) Upvoted: 27 Apr, '21 Comments: 1
Done roadmap

Cdata smartsheet data type conversion

I’m doing the text replace in create table sql and it’s working fine, I’ve had to replace datetime and bigint for now. I don’t expect many random new tables to port ...
Suggested by: Bryan Meyerovich (12 Feb, '21) Upvoted: 22 Feb, '21 Comments: 1

Add {SCHEMA} to copy sql

I need a parameter I can use in the COPY SQL to inject the target schema. {TABLE} has targetschema.tablename, I just need targetschema
Suggested by: Bryan Meyerovich (11 Jan, '21) Upvoted: 11 Jan, '21 Comments: 1