Suggestions for improving Etlworks Integrator.

43 votes

Packages and versioning

It would be nice to create a "package" of flows and connections and store it as a certain version. It should then be easy to switch between versions just by toggling ...
Suggested by: Franck van der Sluis (19 Apr, '19) Upvoted: 15 Feb Comments: 3
Done roadmap
32 votes

2FA Login

Our security policy will require all tools that have access to company data, to have 2FA required to login. Currently a user can login using only the username and ...
Suggested by: Franck (06 Jun, '19) Upvoted: 03 Apr, '20 Comments: 0
Done
13 votes

The ability to manage tags: edit, delete

Suggested by: Rupesh Sawant (23 Jun, '20) Upvoted: 11 Dec, '23 Comments: 0
Planned roadmap
10 votes

SSO Authentication

Etlworks handles and processes a lot of sensitive data. It is important that this is adequately shielded and becomes part of our company-wide security umbrella. The ...
Suggested by: Joris Hooi (24 Oct, '19) Upvoted: 05 May, '22 Comments: 1
Done roadmap
9 votes

Drag and drop process builder for nested flows

Replace existing nested flow editor with drag and drop process builder: - works similarly to the diagram editor - ability to select existing flows from the side-bar ...
Suggested by: Olena Sherbinin (06 Dec, '18) Upvoted: 06 Jan Comments: 0
Planned roadmap
8 votes

Usability enhancement for Flows Builder

1. Ability to edit inner flow from the nested flow 2. Ability to stop API flow, also stop flow from scheduler 3. Add Find Usage for flows (within nested flows)
Suggested by: Maksym (03 Oct, '19) Upvoted: 14 Feb Comments: 0
Planned roadmap
8 votes

New fields' mapping editor

Enhance field mapping editor: - add support for nested fields in the destination - add an ability to modify attributes, such as JSON array, XML CDATA, XML ...
Suggested by: Olena Sherbinin (06 Dec, '18) Upvoted: 26 Jan Comments: 0
Done roadmap
7 votes

Bulk edit configuration mapping

Instead of updating source/destination field names or data types individually, allow selection of desired fields for bulk modification(s).
Suggested by: Jonathon (31 Oct, '23) Upvoted: 06 Jan Comments: 2
Under consideration
6 votes

Email on schedule failure only if last retry

I don't want emails for transient errors if a retry worked
Suggested by: Bryan (09 Aug, '22) Upvoted: 12 Oct, '22 Comments: 2
Done
6 votes

IBM i Debezium Connector

Hello, would it be possible to add IBM i/AS400 Debezium Connector capabilities via this repo please? https://github.com/jhc-systems/debezium-connector-ibmi
Suggested by: Ryan Cronin (01 Feb, '23) Upvoted: 16 Jan Comments: 1
Done roadmap
6 votes

Ability to schedule and execute flows by data integration agents running behind the firewall

Basically, add support for hybrid data integration when flows can be executed on data integration agents running behind the firewall.
Suggested by: Olena Sherbinin (23 Mar, '19) Upvoted: 27 Dec, '21 Comments: 1
Done roadmap
6 votes

(re-)Run from error screen

If investigate an issue within ETLworks. You go to the error (via status) to the statistics section. When solving the issue, you have to search for the flow in order ...
Suggested by: Marvin (27 Feb, '23) Upvoted: 14 Feb Comments: 0
Planned
6 votes

Provisioning from AWS and Azure Marketplaces

Add ability to provision Etlworks from AWS and Azure Marketplaces with Amazon and Microsoft billing
Suggested by: Olena Sherbinin (06 Dec, '18) Upvoted: 06 Jan Comments: 0
Planned roadmap
5 votes

Better ways to determine how flows, connections, etc. are interconnected

Right now when you have multiple flows that share connections or other flows, it's difficult to understand how making a change to one will possibly impact other ...
Suggested by: Shanon (07 Jul, '19) Upvoted: 28 Sep, '19 Comments: 5
Done roadmap
5 votes

Implement priority queue and throttling for user-defined APIs (listeners)

Scenario 1: upstream system slammed the system with thousands of API calls in a short span of time, eventually freezing and requiring a reboot to stop all jobs ...
Suggested by: Olena Sherbinin (07 Nov, '20) Upvoted: 08 Mar, '21 Comments: 3
Done roadmap
5 votes

Drag&drop SQL query builder

Suggested by: Olena Sherbinin (13 Mar, '19) Upvoted: 06 Jan Comments: 0
Planned roadmap
4 votes

Connector for Google Analytics 4,

Since Google is discontinuing Universal Analytics and moving to only supporting Google Analytics 4,
Suggested by: Olena Sherbinin (18 Feb, '23) Upvoted: 18 Feb, '23 Comments: 1
Done roadmap
4 votes

AWS Kinesis Integration

AWS Kinesis Integration
Suggested by: Karthik (08 Jun, '20) Upvoted: 14 Jul, '20 Comments: 1
Done roadmap
4 votes

Search functionality / better documentation of flows

We use a good deal of javascript and in complex flows it can be really difficult to debug, esp. global and flow variables because there's no way to search (that I ...
Suggested by: Shanon (31 Oct, '19) Upvoted: 01 Apr, '23 Comments: 1
Done roadmap
4 votes

Test Script button

Testing the „Execute JavaScript or Python code“ type flows is very troublesome, as you have to save (which closes the modal), press run, open the flow execution view, ...
Suggested by: Heiko (10 Aug, '23) Upvoted: 13 Feb Comments: 0
Under consideration
4 votes

Add exceptions/exclusions rules to schedules

Often times we need to apply schedules to not run at certain times for various reasons. Right now, we have to create multiple schedules. It would be nice to ...
Suggested by: Ryan Cronin (08 Jul, '23) Upvoted: 13 Feb Comments: 0
Planned roadmap
4 votes

View of schedules across all tenants + Last execution duraton

A view were all schedules across all tenants are visible with additional column with last execution duration will help balance load across instance. It will help pick ...
Suggested by: Jaroslaw (20 Nov, '20) Upvoted: 28 Jan Comments: 0
Done roadmap
4 votes

Auto-completion in JavaScript code editors

Add auto-completion in JavaScript code editors: - {field names} - functions
Suggested by: Olena Sherbinin (06 Dec, '18) Upvoted: 24 May, '22 Comments: 0
Planned roadmap
3 votes

Scheduler queue

Introduce queuing in scheduler so you can set how many scheduled tasks can execute simultaneously, and if one tries to run while another is running it will wait till ...
Suggested by: Bryan (18 Mar, '20) Upvoted: 12 Oct, '22 Comments: 2
Done roadmap
3 votes

Add support for JMS, ActiveMQ and AWS SQS

Suggested by: Maksym Sherbinin (03 Oct, '19) Upvoted: 14 Jul, '21 Comments: 2
Done roadmap
3 votes

Allow creation of new connections for each parallel execution in Bulk to Synapse type flows.

Currently, in Nested flows, when executing consecutive SQL Script flows with "Parallel" nested flow option turned on, new connection is created for each SQL Script ...
Suggested by: Heiko (14 Sep, '22) Upvoted: 05 Jan, '23 Comments: 1
Done
3 votes

Bulk insert with db to db flows

Having inserts into postgres and sqlserver work like redshift would be a nice improvement. Right now it runs a bunch of insert statements. Instead, pull source data ...
Suggested by: Bryan (10 Oct, '20) Upvoted: 15 Nov, '20 Comments: 1
Done roadmap
3 votes

Add ability to resume a sync from where it might have failed, or was stopped/canceled

Sometimes syncs fail for whatever reasons, but being able to easily resume from the flows that were not completed would be fantastic (assuming we separately addressed ...
Suggested by: Graham (09 Oct, '19) Upvoted: 11 Oct, '19 Comments: 1
Done
3 votes

Import entire schemes without specifying individual table

I really liked how aws dms operated, you pointed it at a schema on your source, told it the target schema in your destination, and had a wildcard supporting ...
Suggested by: Bryan (24 Jul, '19) Upvoted: 28 Sep, '19 Comments: 1
Done roadmap
3 votes

UI Suggestion: Flow execution/failure duration metrics

Currently, some flows only have a console log of events that happen during flow execution. It would be useful to have a graphical representation, especially nested ...
Suggested by: Drew Thompson (17 Jan, '19) Upvoted: 08 Jan, '21 Comments: 1
Done
3 votes

Table view in Test Transformation

1) If the destination connection is a database display output in a table view 2) Use code editor for a preview
Suggested by: Maksym Sherbinin (10 Jan, '19) Upvoted: 03 Nov, '23 Comments: 1
Done roadmap
3 votes

Ability to create and share templates

Templates are similar to export/import. Additionally, users should be able to: - Create a template from any Flow/Connection/Format. - Share templates with everyone ...
Suggested by: Olena Sherbinin (06 Dec, '18) Upvoted: 11 Dec, '23 Comments: 1
Done roadmap
3 votes

Repeat long running schedule notifications

When telling a schedule to alert you if it’s been running for more than X minutes, allow it to alert you every X minutes. Right now I can’t tell the difference ...
Suggested by: Bryan (23 Aug, '23) Upvoted: 14 Feb Comments: 0
Under consideration
3 votes

FLOWs visual based on tree hierarchy

When having multiple flows, it would be nice if you can collapse actions into the nested flows (like a tree). Result then, only nested flows will be shown, until ...
Suggested by: Marvin (02 Jun, '22) Upvoted: 03 Jun, '22 Comments: 0
Under consideration
3 votes

Add interactive Azure Active directory authentication for SQL Server connector

SQL Server JDBC driver supports accessToken authentication which requires a web flow for signing in with Microsoft. It would be great if Etlworks connector could ...
Suggested by: Olena Sherbinin (12 Oct, '21) Upvoted: 18 Oct, '21 Comments: 0
Done
3 votes

Support IAM For S3 Authentication

We run ETL Works on a self-hosted EC2 server. To be able to connect to AWS S3 from ETL Works, it currently requires an access key and secret key. Our organisation ...
Suggested by: Mark (13 Sep, '21) Upvoted: 14 Sep, '21 Comments: 0
Done
3 votes

Allow setting flow parameter in explorer

Made a api connector with a {param1} in the url, want to do this in explorer: select * from content where param1=123
Suggested by: Bryan (31 Aug, '21) Upvoted: 26 Sep, '21 Comments: 0
Planned roadmap
3 votes

Allow setting flow variables in scheduler

Suggested by: Bryan (27 Mar, '20) Upvoted: 05 Oct, '20 Comments: 0
Done roadmap
3 votes

Add a next/previous transform button when editing transform details

Easily transition between transforms to more quickly apply a change to all transforms without losing your place in the list going back and fourth
Suggested by: Bryan (19 Mar, '20) Upvoted: 07 Jul, '23 Comments: 0
Planned roadmap
2 votes

Statistics Page Should include Agent Metrics too

Currently Statistics page does not include metrics for the flows which are executed by Agent... it would make dashboard much better if we can see the whole picture ...
Suggested by: Atul Kapoor (09 May, '22) Upvoted: 09 May, '22 Comments: 3
Done roadmap
2 votes

Add run schedule api endpoint

I'd like to run schedule job via api instead of just flow. This way I can take advantage of the scheduler's extra options like expected flow execution time. Pass ...
Suggested by: Bryan Meyerovich (07 Aug, '20) Upvoted: 07 Aug, '20 Comments: 2
Done
2 votes

Recreating table for HWM flow after data is populated

On a non-HWM transform with new source columns, the destination table isn’t dropped until after the data is populated. On a HWM transform it drops it before data ...
Suggested by: Bryan Meyerovich (03 Aug, '20) Upvoted: 05 May, '22 Comments: 2
Under consideration
2 votes

Allow for table list alphabetization in explorer

Expanded salesforce tables in explorer, table names are not alphabetized, makes it hard to find anything
Suggested by: Bryan Meyerovich (10 Jun, '20) Upvoted: 10 Jun, '20 Comments: 2
Done roadmap
2 votes

Searching flows should also search table names

I want to find a flow that syncs a specific table, the search only includes the flow names. If it's easy include the table names or other sub-fields or scripts
Suggested by: Bryan (12 Feb, '20) Upvoted: 18 Mar, '20 Comments: 2
Done roadmap
2 votes

Monospace default font in editors

Set default font in all editors to monospace, or allow customization option somewhere (either at bottom of editor or at profile level, preferably both).
Suggested by: Jonathon (31 Oct, '23) Upvoted: 02 Nov, '23 Comments: 1
Done
2 votes

dashboard of tenants overview and alerts

We would like to have a dashboard with statistics on all the tenants on one screen for managing and keeping the resources allocated inline 1. how many failures have ...
Suggested by: itamar (16 Feb, '23) Upvoted: 16 Feb, '23 Comments: 1
Done
2 votes

Add button to restart service

Sometimes there is a need to restart ETL works service, it would be nice to have a button or something like that in the interface
Suggested by: Jarmo Liivak (09 Jun, '22) Upvoted: 20 Jul, '22 Comments: 1
Not planned
2 votes

Add ability to generate new Google Sheets

I'm looking for a way to generate new Google Sheets from ETLworks. Or at minimum, use the Google Drive connector to create a new file, then the Google Sheets ...
Suggested by: Olena Sherbinin (30 Nov, '21) Upvoted: 30 Nov, '21 Comments: 1
Done roadmap
2 votes

Parallel CDC snapshot

Currently, CDC flows run snapshots sequentially, which for large databases can be a bottleneck. It would be great to have an option to run snapshots in parallel, each ...
Suggested by: Olena Sherbinin (23 Sep, '21) Upvoted: 23 Sep, '21 Comments: 1
Done
2 votes

support parameterized macros

allow passing params directly to macros when referencing them {{4:MacroName}('MySpecialParam')} or something like that
Suggested by: Bryan (11 Sep, '21) Upvoted: 13 Sep, '21 Comments: 1
Not planned